I have a Collection looks like :
[{
"company": "A",
"name": "N1",
"age": "C1"
},{
"company": "A",
"name": "N2",
"age": "C2"
},{
"company": "B",
"name": "N3",
"age": "C3"
}]
I have 2 optimization strategy：
1. sharding
sharding key: company.
Then company A / company B should divide to two shards.
And I will store them in different mongod server.
2. divide them by collection name
company A 's collection is
col_A = [{
"name": "N1",
"age": "C1"
},{
"name": "N2",
"age": "C2"
}]
company B 's collection is
col_B = [{
"name": "N3",
"age": "C3"
}]
Which plan is better?
I think the second solution reduces the number of primary keys and the amount of data. But it will cause the number of collections to increase.
Which should I choose? Thanks a lot...
It depends on your data size and how your scaling strategy.
Sharding is the better approach when you are talking about the future data size as it will better handle horizontal scaling. with your second option, it won't be easy to scale horizontally.
If you're taking the first option, you probably want to consider other potential sharding keys that could have the best performance result for your query. For example, if one company was making the most queries through the entire application, the sharding base on the company might not be the best.
One more advantage of sharding is you don't need to refractory your model/repository that much compared to using multiple collections.
Related
I have records with a "title" and a "brand" fields and i query both fields.
Sometimes a record has the brand in the title, which will result in higher scores, but I want to score them the same.
How can i rate records lower were both fields match?
Your solution is not ideal.
In Solr, there is the Dismax query parser that allows you to search for individual terms across several fields, using some other parameters to influence the final score.
The q parameter defines the main query while the qf parameter can be used to specify a list of fields with which to search.
In addition, the tie parameter lets you control how much the final score of the query will be influenced by the scores of the lower-scoring fields compared to the highest-scoring field.
Let's make a simple example.
Using the standard query parser this is what you will obtain running this query (q=adidas):
http://localhost:8983/solr/indexName/select?q=title:adidas%20OR%20brand:adidas&fl=id,title,brand,score
"docs": [
{
"id": "2",
"title": "Shoes Adidas",
"brand": "Adidas",
"score": 0.9623127
},
{
"id": "1",
"title": "Shoes",
"brand": "Adidas",
"score": 0.31506687
},
{
"id": "6",
"title": "Shirt",
"brand": "Adidas",
"score": 0.31506687
}
]
The doc with id 2 has a higher score than the others because the score is the sum of two clauses ('adidas' in title + 'adidas' in brand).
If you perform a Dismax query with tie=0 (a pure "disjunction max query"):
http://localhost:8983/solr/indexName/select?defType=dismax&q=adidas&qf=brand%20title&fl=id,title,brand,score&tie=0
You will obtain:
"docs": [
{
"id": "2",
"title": "Shoes Adidas",
"brand": "Adidas",
"score": 0.6472458
},
{
"id": "1",
"title": "Shoes",
"brand": "Adidas",
"score": 0.31506687
},
{
"id": "6",
"title": "Shirt",
"brand": "Adidas",
"score": 0.31506687
}
]
The doc with id 2 has a lower score than before because only the maximum scoring subquery contributes to the final score, i.e. it takes the max score between 0.6472458 and 0.31506687 without summing them (0.9623127).
With the qf parameter, it is also possible to assign a boost factor to increase or decrease the importance of a particular field in the query, for example:
&qf=brand^3 title
It makes matches in brand much more significant than matches in title.
In any case, boosting should be used with caution because it may lead to unexpected results. Every decision with boosting should be supported by an online and offline search relevance evaluation.
Can this help you?
I solved it by removing all occurrences of the brand in the title (and other fields) when writing the index.
im working on an api and need fast queries for my MongoDB. I have objects like:
{
_id: "123456",
values: [
{
"value": "A",
"begin":0,
"end":1,
},
{
"value": "B",
"begin":1,
"end":2,
},
{
"value": "C",
"begin":3,
"end":7,
}
],
"name": "test"
}
I have about 6k documents in my Database and i want to unwind and group the values so i can count how many different values i have. After the unwind operation i have about 32 million documents.
I tried some variations of the aggregation but everytime its slow. I always need over 20 seconds.
I want to try the indexes but i cant creat and index that makes it any faster. I tried it with the MongoDB Compass app but i think im doing somethink wrong while i create the index.
Hope someone can help me.
Greetings
I am building a site using Couchdb and ReactJS.
One of my pages displays a list of up to 10,000 financial transactions, each txn consisting of:
date
in amount
out amount
payee
category item
notes
I have a pagination strategy and only load and display 100 transactions at a time.
At any one time, I want to be able to search a single column - I use a drop down to tell the search functionality which index to use for searching.
I also want to be able to sort each column.
So far I have used multiple views and I have all of the above functionality working.
During development I used a string for the category item. Now that I have worked out how to get all of the above to work, I need to properly tackle the category item column entry.
A category item belongs to a category, so a category can have one or more category items so there is a one to many relationship between the category and the items.
Each txn can have one and only one category item.
A category is made up of a small number of fields.
A category item is made up of a small number of fields.
I am struggling to find the best approach to doing this.
I have considered each of the approaches described in https://docs.couchbase.com/server/5.0/data-modeling/modeling-relationships.html.
At this point, I am considering one of the following approaches and I was wondering if anyone had any advice - I have include examples of the txns, cats and cat items at the end of this post?
Embed the cat item in the txn and hopefully suss how to both search and sort on the cat item.name
Abandon pagination and load all the txns into the virtual dom, and sort and search the dom directly
Currently each distinct item is a separate document and I use referencing to maintain the relationship. I have considered using the id to store searching and sorting data but I don't see how this would work to give me all that I need.
Txn
{
"_id": "1",
"type": "txn"
"date": "2020-01-20",
"cat": "3",
"notes": "xxxx",
"out": 10,
"in": 0
}
Category
{
"_id": "2",
"type": "cat",
"name": "Everyday Expenses",
"weight": 2
}
Category Item
{
"_id": "3",
"type": "catitem",
"cat": "2",
"name": "Groceries (£850)",
"weight": 0,
"notes": "blah, blah, blah"
}
I am running ReactJS on node.js and I am using pouchdb.
I currently have 2 documents in my mongodb collection. They are:
"_id": "5b6c7109c21dfe2a4b557b1e",
"title": "Childish Gambino",
"datetime": "2018-09-16T00:00:00.000Z",
"venue": {
"name": "Madison Square Garden",
"city": "New York",
"state": "NY"
"_id": "5b6c7109c21dfe2a4b557b1d",
}
and
"_id": "5b6c71133acdbe2a4e51615d",
"title": "The Eagles",
"datetime": "2018-09-12T00:00:00.000Z",
"venue": {
"name": "Madison Square Garden",
"city": "New York",
"state": "NY"
"_id": "5b6c71133acdbe2a4e51615c",
}
Although these are two different concerts, both take place at Madison Square Garden. However, in the 2 documents, Madison Square Garden has 2 different "_id" attributes. Is there a way I can make them have the same _id or would I have to manually input the same _id every time I add a document that takes place at Madison Square Garden? Is manually inputting _id generally considered an okay practice? New to mongodb so any help would be appreciated. Thanks!
If you use MongoDB version 3.4 and up you could use $lookup which means that in your schema you could add a reference to the actual Madison Square Garden (which I assume resides in another collection). This way your data could look something like this:
"_id": ObjectId('5b6c71133acdbe2a4e51615d'),
"title": "The Eagles",
"datetime": "2018-09-12T00:00:00.000Z",
"venueId": ObjectId('5b6c71133acdbe2a4e51615c')
And you would be able to cross-reference that venue using the $lookup feature.
Note: This is by no means more efficient than having the data embedded but it gives you one reference _id.
Sometimes when using Azure Search's paging there may be duplicate documents in the results. Here is an example of a paging request:
GET /indexes/myindex/docs?search=*$top=15&$skip=15&$orderby=rating desc
Why is this possible? How can it happen? Are there any consistency guarantees when paging?
The results of paginated queries are not guaranteed to be stable if the underlying index is changing, or if you are relying on sorting by relevance score. Paging simply changes the value of $skip for each page, but each query is independent and operates on the current view of the data (i.e. – there is no snapshotting or other consistency mechanism like you’d find in a general-purpose database).
Here is an example of how you might get duplicates. Assume an index with four documents:
{ "id": "1", "rating": 5 }
{ "id": "2", "rating": 3 }
{ "id": "3", "rating": 2 }
{ "id": "4", "rating": 1 }
Now assume you want to page through the results with a page size of two, ordered by rating. You’d execute this query to get the first page:
$top=2&$skip=0&$orderby=rating desc
And get these results:
{ "id": "1", "rating": 5 }
{ "id": "2", "rating": 3 }
Now you insert a fifth document into the index:
{ "id": "5", "rating": 4 }
Shortly thereafter, you execute a query to fetch the second page of results:
$top=2&$skip=2&$orderby=rating desc
And get these results:
{ "id": "2", "rating": 3 }
{ "id": "3", "rating": 2 }
Notice that you’ve fetched document 2 twice. This is because the new document 5 has a greater value for rating, so it sorts before document 2 and lands on the first page.
In situations where you're relying on document score (either you don't use $orderby or you're using $orderby=search.score()), paging can return duplicate results because each query might be handled by a different replica, and that replica may have different term and document frequency statistics -- enough to change the relative ordering of documents at page boundaries.
For these reasons, it’s important to think of Azure Search as a search engine (because it is), and not a general-purpose database.