findOneAndUpdate causing duplication problem - arrays

I am having a problem in findOneAndUpdate in mongoose.
The case is that i am updating a document by finding it.
The query is as follows:
UserModel.findOneAndUpdate({
individualId: 'some id'
}, {
$push: {
supporterOf: 'some string'
}
})
The 'supporterOf' is the ref of UserModel and its type is 'ObjectId'.
The issue i am facing here is that, 'some string' is being pushed twice under 'supporterOf' in the document.
Can anyone tell me that how to push an array element inside the document?

I was having same problem, solution is.
I was keeping await like below.
**await** schema.findOneAndUpdate(queryParms, {
"$push": {
"array1": arrayDetails,
"array2": array2Details
}
}, {
"upsert": true,
"new": true
},
function (error, updateResponse) {
if (error) {
throw new Error (error);
} else {
// do something with updateResponse;
}
});
simply removing await helped me resolving this problem.
Need to find the root cause.
any pointer for references are welcome.

I have recently encountered the same problem. However, I managed to overcome this issue by some other logics (details given below) but couldn't understand the reason behind that why findOneAndUpdate inserting duplicate entries in mongodb.
You can overcome this problem by following logic.
Use findOne or findById instead of findOneAndUpdate to search the document in your collection and then manually update your document and run save().
You can have better idea with this code snippet
return new Promise(function (resolve, reject) {
Model.findOne({
someCondition...
}, function (err, item) {
if (err) {
reject(err);
} else {
item.someArray.push({
someKeyValue...
});
item.save().then((result) => {
resolve(result)
}).catch((err) => {
reject(err)
});
}
}).catch((err) => {
reject(err)
});
});
This will not insert duplicate item. However, if you come to know the reasoning behind duplication, must do update this thread.

The issue seems to stem from combining an await and a callback. I had the same issue until I realised I was using an (err, resp) callback and a .catch(...).
models[auxType].findOneAndUpdate(
filter,
updateObject,
options,
(err, resp)=>{
if (err) {
console.log("Update failed:",err)
res.json(err)
} else if (resp) {
console.log("Update succeeded:",resp)
res.json(resp)
} else {
console.log("No error or response returned by server")
}
})
.catch((e)=>{console.log("Error saving Aux Edit:",e)}); // << THE PROBLEM WAS HERE!!
The problem resolved as soon as I removed the .catch(...) line.
From the mongoose documentation:
"Mongoose queries are not promises. They have a .then() function for co and async/await as a convenience. However, unlike promises, calling a query's .then() can execute the query multiple times."
(https://mongoosejs.com/docs/queries.html#queries-are-not-promises)

Use $addToSet instead of $push, it should solve the problem. I believe there is an issue with the data structure used in the creation of a mongoose 'Model'. As we know push is an array (which allows duplication) operation while addToSet may be a Set operation (Sets do not allow duplication).

The problem with the accepted answer is that it only solves the problem by wrapping it in an unnecessary additional promise, when the findOneAndUpdate() method already returns a promise. Additionally, it uses both promises AND callbacks, which is something you should almost never do.
Instead, I would take the following approach:
I generally like to keep my update query logic separate from other concerns for both readability and re-usability. so I would make a wrapper function kind of like:
const update = (id, updateObj) => {
const options = {
new: true,
upsert: true
}
return model.findOneAndUpdate({_id: id}, {...updateObj}, options).exec()
}
This function could then be reused throughout my application, saving me from having to rewrite repetitive options setup or exec calls.
Then I would have some other function that is responsible for calling my query, passing values to it, and handling what comes back from it.
Something kind of like:
const makePush = async () => {
try {
const result = await update('someObjectId', {$push: {someField: value}});
// do whatever you want to do with the updated document
catch (e) {
handleError(e)
}
}
No need to create unnecessary promises, no callback hell, no duplicate requests, and stronger adherence to single responsibility principles.

I was having the same problem. My code was:
const doc = await model.findOneAndUpdate(
{filter}, {update},
{new: true}, (err, item) => if(err) console.log(err) }
)
res.locals.doc = doc
next();
The thing is, for some reason this callback after the "new" option was creating a double entry. I removed the callback and it worked.

I had the same problem.
I found a solution for me:
I used a callback and a promise (so using keyword "await") simultaneously.
Using a callback and a promise simultaneously will result in the query being executed twice. You should be using one or the other, but not both.
options = {
upsert: true // creates the object if it doesn't exist. defaults to false.
};
await Company.findByIdAndUpdate(company._id,
{ $push: { employees: savedEmployees } },
options,
(err) => {
if (err) {
debug(err);
}
}
).exec();
to
options = {
upsert: true // creates the object if it doesn't exist. defaults to false.
};
await Company.findByIdAndUpdate(company._id,
{ $push: { employees: savedEmployees } },
options
).exec();

UserModel.findOneAndUpdate(
{ _id: id },
{ object }
)
Even if you use _id as a parameter don't forget to make the filter explicit by id

In my case, changing the async callback solved the problem.
changing this:
await schema.findOneAndUpdate(
{ queryData },
{ updateData },
{ upsert: true },
(err) => {
if (err) console.log(err);
else await asyncFunction();
}
);
To this:
await schema.findOneAndUpdate(
{ queryData },
{ updateData },
{ upsert: true },
(err) => {
if (err) console.log(err);
}
);
if (success) await asyncFunction();

The $addToSet instead of $push allowed me to prevent duplicate entry in my mongoDb array field of User document like this.
const blockUserServiceFunc = async(req, res) => {
let filter = {
_id : req.body.userId
}
let update = { $addToSet: { blockedUserIds: req.body.blockUserId } };
await User.findOneAndUpdate(filter, update, (err, user) => {
if (err) {
res.json({
status: 501,
success: false,
message: messages.FAILURE.SWW
});
} else {
res.json({
status: 200,
success: true,
message: messages.SUCCESS.USER.BLOCKED,
data: {
'id': user._id,
'firstName': user.firstName,
'lastName': user.lastName,
'email': user.email,
'isActive': user.isActive,
'isDeleted': user.isDeleted,
'deletedAt': user.deletedAt,
'mobileNo': user.mobileNo,
'userName': user.userName,
'dob': user.dob,
'role': user.role,
'reasonForDeleting': user.reasonForDeleting,
'blockedUserIds': user.blockedUserIds,
'accountType': user.accountType
}
});
}
}
).catch(err => {
res.json({
status: 500,
success: false,
message: err
});
});
}

Related

NextJS Global Variable with Assignment

I'm new to NextJS, and trying to figure out, how to create a global variable that I could assign a different value anytime. Could someone give a simple example? (I know global might not be the best approach, but still I would like to know how to set up a global variable).
Let's say:
_app.js
NAME = "Ana" // GLOBAL VARIABLE
page_A.js
console.log(NAME) // "Ana"
NAME = "Ben"
page_B.js
console.log(NAME) // "Ben"
try using Environment Variables
/next.config.js
module.exports = {
env: {
customKey: 'my-value',
},
}
/pages/page_A.js
function Page() {
return <h1>The value of customKey is: {process.env.customKey}</h1>
}
export default Page
but you can not change its contents, except by changing it directly in next.config.js
Nextjs no special ways to provide global variables you want. You can achieve by:
Stateful management tool, like redux-react
Using Context
It's not like it's impossible,
I created a file called _customGlobals.jsx and put this as content
String.prototype.title = function () {
const sliced = this.slice(1);
return (
this.charAt(0).toUpperCase() +
(sliced.toUpperCase() === sliced ? sliced.toLowerCase() : sliced)
);
};
and imported it in _app.jsx like this:
import "./_customGlobals";
So now I can call this function on any string anywhere in my project like this:
"this is a title".title()
Database designed for this purpose. But for one var it's not wise to install whole db!
So, you can do it in a JSON file.
Add a var to a JSON file and use a function to update it.
this is a simple function for this usage:
const fs = require('fs');
function updateJSONFile(filePath, updates) {
fs.readFile(filePath, 'utf8', function (err, data) {
if (err) {
console.error(err);
return;
}
let json = JSON.parse(data);
for (let key in updates) {
json[key] = updates[key];
}
fs.writeFile(filePath, JSON.stringify(json, null, 2), 'utf8', function (err) {
if (err) {
console.error(err);
}
});
});
}
So, use it like this:
updateJSONFile('file.json', { name: 'John Doe', age: 30 });
You can create another function to read it dynamicly:
function readJSONFile(filePath) {
fs.readFile(filePath, 'utf8', function (err, data) {
if (err) {
return callback(err);
}
let json;
try {
json = JSON.parse(data);
} catch (error) {
return callback(error);
}
return callback(null, json);
});
}
and you can use it like this:
const readedFile = readJSONFile('file.json')
I deleted the callback function to have a simple code but you can add callback function to log error messages.

Return specific array from object collection

So I get some data into my socket
The code in Client is :
useEffect(() => {
const socket = io("http://localhost:5000/api/socket");
socket.on("newThought", (thought) => {
console.log(thought);
});
}, []);
And then the code in my server is
connection.once("open", () => {
console.log("MongoDB database connected");
console.log("Setting change streams");
const thoughtChangeStream = connection.collection("phonenumbers").watch();
thoughtChangeStream.on("change", (change) => {
io.of("/api/socket").emit("newThought", change);
});
});
When something in my "phonenumbers" collection gets changed I get in return the whole collection . How would I be able to only get the array that got changed from the object in collection?
So for example if in the collection the only service that changed is the one with id "607deefd13c4ebcbcfa0900a" that should be the only one returned and not the whole collection object.
The fullDocument parameter to the options (second) argument to the watch method can be used to get a delta describing the changes to the document for update operations:
const thoughtChangeStream = connection.collection("phonenumbers").watch([], {
fullDocument: 'updateLookup'
});
thoughtChangeStream.on("change", (change) => {
io.of("/api/socket").emit("newThought", change);
});
This will then return a response document like this where updateDescription contains the fields that were modified by the update:
{
_id: {
_data: '8260931772000000012B022C0100296E5A1004ABFC09CB5798444C8126B1DBABB9859946645F696400646082EA7F05B619F0D586DA440004'
},
operationType: 'update',
clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1620252530 },
ns: { db: 'yourDatabase', coll: 'yourCollection' },
documentKey: { _id: 6082ea7f05b619f0d586da44 },
updateDescription: {
updatedFields: { updatedField: 'newValue' },
removedFields: []
}
}
Note: This will only work for update operations and will not work for replace, delete, insert, etc.
See also:
http://mongodb.github.io/node-mongodb-native/3.0/api/Collection.html.
https://docs.mongodb.com/manual/reference/change-events/

Retrieve collection + subcollection give me an empty render

The problem
At web app startup, the main task is to retrieve all user informations that are stored in various Firestore's collections and sub-collections.
The problem is that, even if I uses promises and a Loading State to prevent an empty app rendering, the app is rendered with all collections data, except for collections that have sub-collections.
The process
- If user is logged
-- Set Loading Status Active
-- Load Collection A
-- Load Collection B and forEach, load all sub-collection
-- Load Collection C
-- Set Loading Status Inactive
At this point, the app is rendered, but only with Collection A and Collection C. The collection B is loaded (I can see it by Redux Logs) but can't be seen in the app.
Those datas appears only if I change the component status (open/close a menu, for example).
Some code
Here's how I retrieve a collection with sub-collections:
export function setCompanyJobs(user) {
return {
type: "SET_COMPANY_JOBS",
payload: loadCompanyJobs(user),
};
}
Function that retrieve the main collection
export function loadCompanyJobs(user) {
return new Promise((resolve, reject) => {
let companyJobs = [];
db.collection("company").doc(user.selectedCompany).collection("jobs").get().then((jobs) => {
jobs.forEach((job) => {
loadJobLinkedServices(user, job).then((jobLinkedServices) => {
companyJobs.push({
id: job.id,
...
});
});
});
resolve(companyJobs);
}).catch(function (error) {
...
});
});
}
Function that retrieve all collection's sub-collections
export function loadJobLinkedServices(user, job){
return new Promise((resolve, reject) => {
let jobLinkedServices = [];
db.collection("company").doc(user.selectedCompany).collection("jobs").doc(job.id).collection("linkedServices").get().then((linkedServices) => {
linkedServices.forEach((linkedService) => {
jobLinkedServices.push({
id: linkedService.id,
...
});
});
resolve(jobLinkedServices)
}).catch(function (error) {
...
});
})
When you do
return new Promise((resolve, reject) => {
let companyJobs = [];
db.collection("company").doc(user.selectedCompany).collection("jobs").get().then((jobs) => {
jobs.forEach((job) => {
loadJobLinkedServices(user, job).then((jobLinkedServices) => {
companyJobs.push({
id: job.id,
...
});
});
});
resolve(companyJobs);
}).catch(function (error) {
...
});
});
nothing ensure that your Promise resolves only after ALL the queries triggered in the jobs.forEach() loop are done (i.e. the promises returned by the calls to the loadJobLinkedServices function have resolved).
I don't know reactjs but I think you can use the JavaScript Promise.all() method along the following lines:
return new Promise((resolve, reject) => {
let promises = [];
let companyJobs = [];
db.collection("company").doc(user.selectedCompany).collection("jobs").get().then((jobs) => {
jobs.forEach((job) => {
promises.push(loadJobLinkedServices(user, job));
});
Promise.all(promises).
then(results => {
//Loop over the results array to populate the companyJobs array
resolve(companyJobs);
})
}).catch(function (error) {
...
});
});
Also, don't forget to correctly chain your calls to the different asynchronous functions, i.e. something like:
query Collection A
THEN query Collection B
THEN query all sub-collections (with Promise.all())
THEN query Collection C
THEN set Loading Status Inactive
Finally, a last remark: Note that the get() method returns a promise, so I am not sure that you need to wrap the calls to the get() method into some new Promises (Again, I am not versed in reactjs, so this remark may be wrong).
In other words, I think you could do something like the following (for example for the loadJobLinkedServices function):
export function loadJobLinkedServices(user, job){
let jobLinkedServices = [];
return db.collection("company").doc(user.selectedCompany).collection("jobs").doc(job.id).collection("linkedServices").get()
.then((linkedServices) => {
linkedServices.forEach((linkedService) => {
jobLinkedServices.push({
id: linkedService.id,
...
});
});
return jobLinkedServices;
}).catch(function (error) {
...
});
})

MongoDB: how can I find and merge array

I'm trying to find a specific ID in my document and then merge an array to the existing one, for example if I have this array stored in db.friends:
["12","13","14"]
and I send this array: ["12","16","18"], db.friends should contain: ["12","13","14","16","18"]
I'm using underscore library, but I'm not sure I have to (maybe "aggregate" in mongoose?)
Here is what I did, can you tell me where am I wrong?
function saveFollowers(req, res) {
var friends = req.body.friends; // the new array to merge ["54aafe9df4ee360300fc94c7"];
User.findOne({_id: req.user._id}).exec(function (err, user) {
if (err) {
res.jsonp({error: "Error fetching user info"})
} else {
friends = _.extend(friends, user.friends); //user.friends=existing friends we have in db
user.save(function (err) {
if (err) { res.jsonp({error: "Cant save"}); }
console.log("Friends NOW:"+JSON.stringify(friends)); //Here I don't see the merge, also, I can't see it in mongo db.
res.jsonp("success");
});
}
});
Thank you!
With your current implementation, you haven't actually modified the friends key in the returned user object. So rather you can use the union method as
user.friends = _.union(friends, user.friends); //user.friends=existing friends
user.save(function (err) { .. }
Or with ES6 using the spread operator for concatenating the array and Set for creating a distinct set of elements:
user.friends = [...new Set([...friends ,...user.friends])];
user.save(function (err) { .. }
Another alternative is using the aggregation framework, you could utilize the $setUnion operator:
function saveFollowers(req, res) {
var friends = req.body.friends; // the new array to merge ["54aafe9df4ee360300fc94c7"];
User.aggregate([
{ "$match": { _id: req.user._id } },
{
"$project": {
"friends": { "$setUnion": [ "$friends", friends ] }
}
}
]).exec(function (err, results){
if (err) {
res.jsonp({error: "Error fetching user info"})
} else {
User.findByIdAndUpdate(req.user._id,
{ "$set": { "friends": results[0].friends } },
{ "new": true },
function (err, user) {
if (err) { res.jsonp({error: "Cant save"}); }
console.log("Friends NOW: "+ JSON.stringify(user.friends));
res.jsonp("success");
}
);
}
});
}

Synchronous database queries with Node.js

I have a Node.js/Express app that queries a MySQL db within the route and displays the result to the user. My problem is how do I run the queries and block until both queries are done before redirecting the user to the page they requested?
In my example I have 2 queries that need to finish before I render the page. I can get the queries to run synchronously if i nest query 2 inside the 'result' callback of query 1. This however will become very convoluted when the number of queries increase.
How do I go about running multiple (in this case 2) database queries synchronously without nesting the subsequent query in the prior query's 'result' callback?
I've looked at the 'Flow control / Async goodies' in the Node modules and tried flow-js but I can't get it to work with the async queries.
Listed below are the 2 queries that I'm attempting to execute from the '/home' route. Can the Node experts explain the 'right' way to do this.
app.get('/home', function (req,res) {
var user_array = [];
var title_array = [];
// first query
var sql = 'select user_name from users';
db.execute(sql)
.addListener('row', function(r) {
user_array.push( { user_name: r.user_name } );
})
.addListener('result', function(r) {
req.session.user_array = user_array;
});
// second query
var sql = 'select title from code_samples';
db.execute(sql)
.addListener('row', function(r) {
title_array.push( { title: r.title } );
})
.addListener('result', function(r) {
req.session.title_array = title_array;
});
// because the queries are async no data is returned to the user
res.render('home.ejs', {layout: false, locals: { user_name: user_array, title: title_array }});
});
The goal with node is not to care what order things happen in. This can complicate some scenarios. There is no shame in nesting callbacks. Once you are used to how it looks, you may find that you actually prefer that style. I do; it is very clear what order callbacks will fire. You can forgo the anonymous functions to make it less verbose if you have to.
If you are willing to restructure your code a bit, you can use the "typical" nested callback method. If you want to avoid callbacks, there are numerous async frameworks that will try and help you do this. One that you might want to check out is async.js (https://github.com/fjakobs/async.js). Example of each:
app.get('/home', function (req,res) {
var lock = 2;
var result = {};
result.user_array = [];
result.title_array = [];
var finishRequest = function(result) {
req.session.title_array = result.title_array;
req.session.user_array = result.user_array;
res.render('home.ejs', {layout: false, locals: { user_name: result.user_array, title: result.title_array }});
};
// first query
var q1 = function(fn) {
var sql = 'select user_name from users';
db.execute(sql)
.addListener('row', function(r) {
result.user_array.push( { user_name: r.user_name } );
})
.addListener('result', function(r) {
return fn && fn(null, result);
});
};
// second query
var q2 = function(fn) {
var sql = 'select title from code_samples';
db.execute(sql)
.addListener('row', function(r) {
result.title_array.push( { title: r.title } );
})
.addListener('result', function(r) {
return fn && fn(null, result);
});
}
//Standard nested callbacks
q1(function (err, result) {
if (err) { return; //do something}
q2(function (err, result) {
if (err) { return; //do something}
finishRequest(result);
});
});
//Using async.js
async.list([
q1,
q2,
]).call().end(function(err, result) {
finishRequest(result);
});
});
For a one-off, I would probably just use a reference counting type approach. Simply keep track of how many queries you want to execute and render the response when they have all finished.
app.get('/home', function (req,res) {
var lock = 2;
var user_array = [];
var title_array = [];
var finishRequest = function() {
res.render('home.ejs', {layout: false, locals: { user_name: user_array, title: title_array }});
}
// first query
var sql = 'select user_name from users';
db.execute(sql)
.addListener('row', function(r) {
user_array.push( { user_name: r.user_name } );
})
.addListener('result', function(r) {
req.session.user_array = user_array;
lock -= 1;
if (lock === 0) {
finishRequest();
}
});
// second query
var sql = 'select title from code_samples';
db.execute(sql)
.addListener('row', function(r) {
title_array.push( { title: r.title } );
})
.addListener('result', function(r) {
req.session.title_array = title_array;
lock -= 1;
if (lock === 0) {
finishRequest();
}
});
});
An even nicer approach would be to simply call finishRequest() in each 'result' callback an check for non-empty arrays before you render the response. Whether that will work in your case depends on your requirements.
Here's a really easy trick to handle multiple callbacks.
var after = function _after(count, f) {
var c = 0, results = [];
return function _callback() {
switch (arguments.length) {
case 0: results.push(null); break;
case 1: results.push(arguments[0]); break;
default: results.push(Array.prototype.slice.call(arguments)); break;
}
if (++c === count) {
f.apply(this, results);
}
};
};
Example
Usage:
var handleDatabase = after(2, function (res1, res2) {
res.render('home.ejs', { locals: { r1: res1, r2: res2 }):
})
db.execute(sql1).on('result', handleDatabase);
db.execute(sql2).on('result', handleDatabase);
So basically you need reference counting. This is the standard approach in these situations. I actually use this small utility function instead of flow control.
If you want a full blown flow control solution I would recommend futuresJS
I find that the async library is the best for things like this. https://github.com/caolan/async#parallel
I can't test this or anything, so forgive me if there are some typos. I refactored your query function to be reusable. So, calling queryRows will return a function that matches the format of the async module's parallel callback functions. After both queries are complete, it will call the last function and pass the result of the two queries as an argument, which you can read to pass to your template.
function queryRows(col, table) {
return function(cb) {
var rows = [];
db.execute('SELECT ' + col + ' FROM ' + table)
.on('row', function(r) {
rows.push(r)
})
.on('result', function() {
cb(rows);
});
}
}
app.get('/home', function(req, res) {
async.parallel({
users: queryRow('user_name', 'users'),
titles: queryRow('title', 'code_samples')
},
function(result) {
res.render('home.ejs', {
layout: false,
locals: {user_name: result.users, title: result.titles}
});
});
});
There are some solutions here, but in my opinion the best solution is to make the code synchronously in a very easy way.
You could use the "synchonize" package.
Just
npm install synchronize
Then var sync = require(synchronize);
Put logic which should be synchronous into a fiber by using
sync.fiber(function() {
//put your logic here
}
An example for two mysql queries:
var express = require('express');
var bodyParser = require('body-parser');
var mysql = require('mysql');
var sync = require('synchronize');
var db = mysql.createConnection({
host : 'localhost',
user : 'user',
password : 'password',
database : 'database'
});
db.connect(function(err) {
if (err) {
console.error('error connecting: ' + err.stack);
return;
}
});
function saveSomething() {
var post = {id: newId};
//no callback here; the result is in "query"
var query = sync.await(db.query('INSERT INTO mainTable SET ?', post, sync.defer()));
var newId = query.insertId;
post = {foreignKey: newId};
//this query can be async, because it doesn't matter in this case
db.query('INSERT INTO subTable SET ?', post, function(err, result) {
if (err) throw err;
});
}
When "saveSomething()" is called, it inserts a row in a main table and receives the last inserted id. After that the code below will be executed. No need for nesting promises or stuff like that.
option one: if all your queries related to each other, create stored procedure, put all your data logic into it and have a single db.execute
option two: if your db uses one connection then commands a guaranteed to be executed serially and you can use this as async helper
db.execute(sql1).on('row', function(r) {
req.session.user_array.push(r.user);
});
db.execute(sql2)
.on('row', function(r) {
req.session.title_array.push(r.title);
})
.on('end'), function() {
// render data from req.session
});
You can use fibers to write pseudo-synchronous code with Node.JS take a look at these tests for DB https://github.com/alexeypetrushin/mongo-lite/blob/master/test/collection.coffee
they are asynchronous but looks like synchronous, more details http://alexeypetrushin.github.com/synchronize

Resources