Pouchdb view returning empty result - database

Good morning,
I'm currently working with Couchdb and Pouchdb and I'm having a problem with one query on Pouchdb side.
I have a database with different documents setup like this:
{
"_id": "fd87b66087503d760fa501fa49029f94",
"_rev": "1-e2be19d447c98d624c2c8492eaf0a3f4",
"type": "product",
"name": "Blanc de Morgex et de la Salle Brut Extreme 2014",
"category": "Wine",
"subcategory": null,
"zone": "Italy",
"nation": "Valle d'Aosta",
"province": "Morgex, AO",
"cellar": "Cave Mont Blanc",
"price": 30,
"structure": null,
"year": 2014,
"mescita": null,
"tag": null
}
The query I wrote should return the available years of products that match some filters. This is the query, with reduce : _count:
function (doc) {
if(doc.category && doc.type == 'product' && doc.year != null) {
emit(doc.year , 1);
}
}
If I try it with Postman adding the group = true parameter everything works and the result is something like:
{
"rows": [
{
"key": 2004,
"value": 2
},
{
"key": 2006,
"value": 2
},
{
"key": 2008,
"value": 2
}
]
}
The problem is when i run this view with Pouchdb with the following code which return a JSON with an empty array:
wine_db.query('wine_list/years', {reduce: '_count', key : "Bollicine", group : true, group_level: 2}).then(function(doc) {
years_list = doc;
console.log('getting year list');
console.log(doc);
}).catch(function(err){
console.log(err);
});
I've tried to play a little with the parameters of the function and even changing the function to return just a list of all the years, but nope.
I can't find the problem neither a different solution so I'm open to every suggestion you can have.
Another solution (group result)
Working on the indications and on the solution suggested by #user3405291 I finally found a way to group the results by year.
Since the emit function return a complex key ['CATEGORY', YEAR] I can use the startkey and endkey parameters to query the result just for a section of the index returned keeping this way the reduce function enable to group the result.
In the end the view function is:
function (doc) {
if(doc.category && doc.type == 'product' && doc.year) {
emit([doc.category, doc.year], doc.year );
}
}
And the Pouchdb query:
wine_db.query('wine_list/years',
{
startkey : ['CATEGORY'],
endkey : ['CATEGORY', {}],
group: true
}
).then(function (doc) {
years_list = doc;
console.log(years_list);
}).catch(function (err) {
console.log(err);
});
The result, where value is the total number of elements with that index:
{
"rows": [
{
"key": [
"Bollicine",
2004
],
"value": 2
},
{
"key": [
"Bollicine",
2006
],
"value": 2
},
{
"key": [
"Bollicine",
2008
],
"value": 2
}
]
}

In your view map function you emit the year as the key/index:
emit(doc.year , 1);
Now, I'm not sure why your are doing your query with a key like {key : "Bollicine"}:
wine_db.query('wine_list/years', {key : "Bollicine"})
.then(res=>{console.log(res)})
Of course you would get an empty response, because your view is actually indexing your docs according to year. I think you might want to do a query with a key like: {key : "2014"}
UPDATE
Based on your comments, I feel like you need to find docs based on both year and category. I'm not sure if I understand what you want, but this may help you: change your view map function like this:
function (doc) {
if(doc.category && doc.type == 'product' && doc.year) {
emit([doc.year, doc.category] , 1);
}
}
The above view will index your docs according to both year and category. You then query your view like this:
wine_db.query('wine_list/years', {key : ['2014', 'Bollicine']})
.then(res=>{console.log(res)})
The above query will give you all the docs with year field equal to 2014 and category field equal to Bollicine.
Second Update
your code works, but I just get the result for the year 2014. What I'm trying to accomplish is to get all the available years given a specific category
One solution is this:
function (doc) {
if(doc.category && doc.type == 'product' && doc.year) {
emit(doc.category, doc.year);
}
}
The above view will index your docs according to category as key and will return the year as value. Therefore you can query like this:
wine_db.query('wine_list/years', {key : 'Bollicine'})
.then(res=>{console.log(res)})
You should get a response like this, by which you have all the available years for Bollicine category:
{
"total_rows": 400,
"offset": 0,
"rows": [
{
"key": "Bollicine",
"value": "2014"
},
{
"key": "Bollicine",
"value": "2015"
},
{
"key": "Bollicine",
"value": "2018"
}
]
}

Related

How to update array elements in MongoDB instead of creating new one

I have problem with my code. I want to update document, which looks like this:
{
"_id": {
"$oid": "5bf2ad5a0d46b81798232cf9"
},
"manufacturer": "VW",
"model": "Golf ",
"VIN": "WVWZZZ1J212566691",
"doors": 3,
"class": "compact",
"reservations": [
{
"_id": {
"$oid": "5bf2ad5a0d46b81798232cfa"
},
"pick_up_date": {
"$date": "2014-10-13T09:13:00.000Z"
},
"drop_off_date": {
"$date": "2014-10-14T09:13:00.000Z"
},
"user_id": {
"$oid": "5bec00bdfb6fc005dcd5423b"
}
}
],
"createdAt": {
"$date": "2018-11-19T12:32:26.665Z"
},
"updatedAt": {
"$date": "2018-11-19T12:32:26.665Z"
},
"__v": 0
}
I want to add new reservation to array Reservations. My function:
const createReservationForCarId = ({ body , params }, res, next) =>
Carmodel.findById(params.id)
.then(notFound(res))
.then((carmodel) => carmodel ? Object.assign(carmodel, body).save() : null)
.then((carmodel) => carmodel ? carmodel.view(true) : null)
.then(success(res))
.catch(next)
But when I'm trying to update through:
router.put('/:id/reservation',
createReservationForCarId)
Body:
{
"reservations":[
{
"pick_up_date" : "October 13, 2017 11:13:00",
"drop_off_date": "October 14, 2017 11:13:00",
"user_id":"5bec00bdfb6fc005dcd5423b"
}
]
}
Mongo instead of update my old document is creating new one with only one reservation gave in above body.
What should I do to only update existing document, not creating new one?
I suggest you use $push operator of Mongoose which will append your object to the existing array and it will not create a new one.
Check out this link:-
https://github.com/Automattic/mongoose/issues/4338
You can try the findAndModify instead of findById in your code. Check out the syntax here
I haven't used mongoose but they have documentation for array manipulation.
See here:
https://mongoosejs.com/docs/api.html#mongoosearray_MongooseArray-push
Also here is the related mongo doc for array manipulation:
https://docs.mongodb.com/manual/reference/operator/update/push/

MongoDB - NodeJS - Mongoose Slow Query Array

I have a question about Mongo and NodeJS currently in one project that i am working on have some issues with the performance. It does work but has a mix of ideas as far as i see. I would like to see some ideas from a more experience person.
I will share the flow with you and the used libs and tools on the way i try to understand it there days and as far as i see might be a better way to do it but still will be slow and un-efficient.
Used Tools and Lib: lodash, mongoose, async, and more.
This is what is the results from NewRelic:
Database MongoDB preferences toArray 19.8 13.6 423ms
Database MongoDB spotifies toArray 19.4 12.6 415ms
Database MongoDB mifi toArray 19.2 12.6 409ms
Database MongoDB fifi toArray 18.9 12.6 404ms
Database MongoDB locations toArray 14.3 18.5 305ms
A sample of what is used to add to the model all the time:
const subObjectsOutCards = [{
name : 'preferencesModel',
model : 'Preferences'}, {
name : 'spotifyModel',
model : 'Spotify'} ,{
name : 'MifiModel',
model : 'mifi'}, {
name : 'fiModel',
model : 'Fifi'}];
Next One:
exports.fetchUsers = (callback) => {
Account.find().populate('userTrips').exec((err, docs) => {
if (_.isEmpty(docs)){
callback(null, null)
} else {
async.map(docs, (doc, callback) => {
var object = doc.toObject();
exports.fetchSubObjectsToUserForCards(object, doc._id, (result) => {
callback(null, result);
})
}, (err, results1) => {
var updatedAccounts = results1;
async.map(updatedAccounts, (doc, callback) => {
locationController.getMostRecentLocationById(doc._id, (result) => {
var updatedDoc = doc;
updatedDoc.locationModel = result;
callback(null, updatedDoc);
})
}, (err, results2) => {
callback(results2);
})
})
}
})}
This is where it goes after:
exports.fetchObjectSub = (account, userId, callback) => {
var newAccount = account;
var itemsProcessed = 0;
subObjectsOut.forEach(function(object){
mongoose.model(object.model).find({'accountId' : userId}, function(err, doc){
if (err){
callback(false);
} else {
if (!_.isNil(doc[0]) && newAccount !== false){
doc = doc[0].toObject();
newAccount[object.name] = doc;
itemsProcessed++;
if (itemsProcessed === subObjectsOut.length){
callback(newAccount);
}
} else {
itemsProcessed++;
}
}
});
});}
Account Example:
{
"_id": "59f372389f89d1cb0dbabdbad",
"residence": "Katowice, Poland",
"orientation": "Bisexual",
"lastName": "Kowalczyk",
"job": "Web Developer # Freelance",
"gender": "Female",
"firstName": "Marina",
"dob": "7/10/1998",
"about": "Computer science student",
"lastConnection": "2017-11-24T19:16:28.780Z",
"created_at": "2017-10-27T20:55:06.070Z",
"phone": {
"number": "7183173136"
},
"profilePicture": {
"url": "https://.JPG",
"pictureType": ".JPG"
},
"interests": [
"Sports",
"Food",
"Cycling",
"Running",
"Cooking",
"Movies",
"Fashion",
"Business",
"Travel",
"Music",
"Theatre",
"Yoga",
"Party",
"Dancing",
"Reading"
],
"__v": 38}
Preferences Example:
{
"_id": "59a83258c7fd5b4ae586c53b",
"visibilityLocation": true,
"visibilityGenderPreferences": true,
"visibilityFb": false,
"visibilityDistance": false,
"visibilityAge": false,
"showMyProfileAs": "Male",
"showMe": "Males",
"locationAccuracy": 0,
"accountId": "59f372389f89d1cb0dbabdbad",
"created_at": "2017-08-31T13:29:42.462Z",
"distance": {
"max": 30,
"metrics": "K",
"min": 0
},
"ageRange": {
"max": 32,
"min": 18
},
"__v": 0}
EDIT: As requested
Indexes - automatic from the system on the _id
Index on accountId in each table from the main one
Operation Performed is on 1000 elements and its really slow: Speeds that i get are 3000ms to 9000ms and all the slow down on average says 5M ms for these documents which is insane....
Example Account and Preferences can be found above before edit.
When i started with this i thought straight that are the filters that we have but seems its not the case. Since thats pretty fast. Problems comes from here and the tools confirm it.
The idea about the structure of the Schema as far as i see is to be a flat one and after to be used Collections all the time instead of references of the other tables. And for each one since there are a lot filters there need to be a search for each person on each one of their five tables and getting their account on the way and adding for each table a field with it and creating a documents. And all of them wait for each other all the time. So if you have 5 users = 5 new documents and after for each there was a loop and a new documents. Please if anybody can help a bit with this would be great. Thanks you

Update array of subdocuments in MongoDB

I have a collection of students that have a name and an array of email addresses. A student document looks something like this:
{
"_id": {"$oid": "56d06bb6d9f75035956fa7ba"},
"name": "John Doe",
"emails": [
{
"label": "private",
"value": "private#johndoe.com"
},
{
"label": "work",
"value": "work#johndoe.com"
}
]
}
The label in the email subdocument is set to be unique per document, so there can't be two entries with the same label.
My problems is, that when updating a student document, I want to achieve the following:
adding an email with a new label should simply add a new subdocument with the given label and value to the array
if adding an email with a label that already exists, the value of the existing should be set to the data of the update
For example when updating with the following data:
{
"_id": {"$oid": "56d06bb6d9f75035956fa7ba"},
"emails": [
{
"label": "private",
"value": "me#johndoe.com"
},
{
"label": "school",
"value": "school#johndoe.com"
}
]
}
I would like the result of the emails array to be:
"emails": [
{
"label": "private",
"value": "me#johndoe.com"
},
{
"label": "work",
"value": "work#johndoe.com"
},
{
"label": "school",
"value": "school#johndoe.com"
}
]
How can I achieve this in MongoDB (optionally using mongoose)? Is this at all possible or do I have to check the array myself in the application code?
You could try this update but only efficient for small datasets:
mongo shell:
var data = {
"_id": ObjectId("56d06bb6d9f75035956fa7ba"),
"emails": [
{
"label": "private",
"value": "me#johndoe.com"
},
{
"label": "school",
"value": "school#johndoe.com"
}
]
};
data.emails.forEach(function(email) {
var emails = db.students.findOne({_id: data._id}).emails,
query = { "_id": data._id },
update = {};
emails.forEach(function(e) {
if (e.label === email.label) {
query["emails.label"] = email.label;
update["$set"] = { "emails.$.value": email.value };
} else {
update["$addToSet"] = { "emails": email };
}
db.students.update(query, update)
});
});
Suggestion: refactor your data to use the "label" as an actual field name.
There is one straightforward way in which MongoDB can guarantee unique values for a given email label - by making the label a single separate field in itself, in an email sub-document. Your data needs to exist in this structure:
{
"_id": ObjectId("56d06bb6d9f75035956fa7ba"),
"name": "John Doe",
"emails": {
"private": "private#johndoe.com",
"work" : "work#johndoe.com"
}
}
Now, when you want to update a student's emails you can do an update like this:
db.students.update(
{"_id": ObjectId("56d06bb6d9f75035956fa7ba")},
{$set: {
"emails.private" : "me#johndoe.com",
"emails.school" : "school#johndoe.com"
}}
);
And that will change the data to this:
{
"_id": ObjectId("56d06bb6d9f75035956fa7ba"),
"name": "John Doe",
"emails": {
"private": "me#johndoe.com",
"work" : "work#johndoe.com",
"school" : "school#johndoe.com"
}
}
Admittedly there is a disadvantage to this approach: you will need to change the structure of the input data, from the emails being in an array of sub-documents to the emails being a single sub-document of single fields. But the advantage is that your data requirements are automatically met by the way that JSON objects work.
After investigating the different options posted, I decided to go with my own approach of doing the update manually in the code using lodash's unionBy() function. Using express and mongoose's findById() that basically looks like this:
Student.findById(req.params.id, function(err, student) {
if(req.body.name) student.name = req.body.name;
if(req.body.emails && req.body.emails.length > 0) {
student.emails = _.unionBy(req.body.emails, student.emails, 'label');
}
student.save(function(err, result) {
if(err) return next(err);
res.status(200).json(result);
});
});
This way I get the full flexibility of partial updates for all fields. Of course you could also use findByIdAndUpdate() or other options.
Alternate approach:
However the way of changing the schema like Vince Bowdren suggested, making label a single separate field in a email subdocument, is also a viable option. In the end it just depends on your personal preferences and if you need strict validation on your data or not.
If you are using mongoose like I do, you would have to define a separate schema like so:
var EmailSchema = new mongoose.Schema({
work: { type: String, validate: validateEmail },
private: { type: String, validate: validateEmail }
}, {
strict: false,
_id: false
});
In the schema you can define properties for the labels you already want to support and add validation. By setting the strict: false option, you would allow the user to also post emails with custom labels. Note however, that these would not be validated. You would have to apply the validation manually in your application similar to the way I did it in my approach above for the merging.

"There is no index available for this selector" despite the fact I made one

In my data, I have two fields that I want to use as an index together. They are sensorid (any string) and timestamp (yyyy-mm-dd hh:mm:ss).
So I made an index for these two using the Cloudant index generator. This was created successfully and it appears as a design document.
{
"index": {
"fields": [
{
"name": "sensorid",
"type": "string"
},
{
"name": "timestamp",
"type": "string"
}
]
},
"type": "text"
}
However, when I try to make the following query to find all documents with a timestamp newer than some value, I am told there is no index available for the selector:
{
"selector": {
"timestamp": {
"$gt": "2015-10-13 16:00:00"
}
},
"fields": [
"_id",
"_rev"
],
"sort": [
{
"_id": "asc"
}
]
}
What have I done wrong?
It seems to me like cloudant query only allows sorting on fields that are part of the selector.
Therefore your selector should include the _id field and look like:
"selector":{
"_id":{
"$gt":0
},
"timestamp":{
"$gt":"2015-10-13 16:00:00"
}
}
I hope this works for you!

How to get a count of a items within a collection that match a specific criteria?

I have a collection like so:
{ "items": [
{
"id": "123",
"meta": {
"activity": 2
}
},
{
"id": "13423",
"meta": {
"activity": 4
}
}
]}
Given the collection, how can I get the collection's total activity? In the case above the result would be 6.
I'm using backbone and underscore.
You're using underscore.js, so you have some good tools at your disposal. Look at _.map() to get started.
var flatArray = _.map(collection.items, function(x){return x.meta.activity;});
// should return: [2,4]
Then you can use _.reduce() to turn this into a single value.
var total = _.reduce(flatArray, function(memo, num) {return memo + num;}, 0);
// should return: 6
There's lots of other great tools in underscore.js, it's worth a look to see if anything else would work for you too.
In underscore you can use the reduce function which will reduce a list of values into a single value using the given iterator function.
var myCollection = { "items": [
{
"id": "123",
"meta": {
"activity": 2
}
},
{
"id": "13423",
"meta": {
"activity": 4
}
}
]};
var totalActivity = _.reduce(myCollection.items, function(memo, item){ return memo + item.meta.activity; }, 0);

Resources