I use Solr to index and search on a system with about 100,000 products and 300,000 users. The field to index is "price". But for each user, the price may be different.
For example:
- For Product1 and User1, User2.
- User1 sees the Product1's price 100$.
- But User2 cannot not see the price (User2 has to fulfill some conditions to see the price) although User2 still sees Product1 when searching.
At the time of indexing, we cannot determine to set the price for a specific user or not. The product has a flag called "Required Contract". And when a user log in, we will check if the user has applied the "contract" for that product to show or hide the price.
The straight forward solution for this problem is to create different "price" fields for each user. So when indexing, we loop through the list of users and index the "price" field for that user. And when searching, we use the correct "price" field for the login user. Obviously, this is not a practical solution.
My question is how to index "price" field in this case or is there any other approaches on the solution?
It seems that you don't have specific prices for each user, but for groups of users. Presumably a not-very-high number of unique groups. If so, a simple solution is to have multiple price fields (price_group_1, price_group_2, ...), one for each user group, and let your display-code show the one matching the users group.
After researching for a while I came up with 2 solutions:
1/ The first one is to duplicate a document that requires the condition. The duplicated document contains no price (or any value assigned to it). Because the numbers of documents requiring conditions are not too much so duplicating is not a waste of resources. So when doing a searching, by combining some other flags the result is the duplicated documents not the original one that fulfill my requirements.
2/ Another approach is using Function Queries. I figured this out by searching on Google for my problem as I didn't know about that feature at the beginning.
Hope that can help someone's problems!
Related
I am currently exploring MongoDB.
I built a notes web app and for now the DB has 2 collections: notes and users.
The user can create, read and update his notes.
I want to create a page called /my-notes that will display all the notes that belong to the connected user.
My question is:
Should the notes model has an ownerId field or the opposite - the user model will have a field of noteIds of type list.
Points I found relevant for the decision making:
noteIds approach:
There is no need to query the notes that hold the desired ownerId (say we have a lot of notes then we will need indexes and search accross the whole notes collection). We just need to find the user by user ID and then get all the notes by their IDs.
In this case there are 2 calls to DB.
The data is ordered by the order of insertion to the notesIds field in the document.
ownerId approach:
We do need to find the notes by their ownerId field across the notes collection which might be more computer "intensive".
We can paginate / sort the data as we want - more control over the data.
Are there any more points you can think of?
As I can conclude this is a question of whether you want less computer intensive DB calls vs more control over the data.
What are the "best practices"?
Thanks,
A similar use case is explained in the documentation. If there is no limit on number of notes a user can have, it might be better to store a userId reference field in notes document.
As you've figured out already, pagination would be easier in the second approach. Also when updating notes, you can simply updateOne({ _id: "note_id", userId: 1 }) instead of checking user's document if the note actually belong to the user.
For a project, I have a database with some tables.
All of them are related between them. The table organization has a relationship with offer and user, etc.
However, I have some columns that only serve to link two tables together.
Take a look at the diagram. I use users_interests to link the user to his interests. Same thing for badges and group.
It doesn't really feel efficient. When I try to get the interests of a users, I must first go through requesting the user interests and from that, require the interest details.
Is there a way that I can request the interests of an user without having to go through a second table ?
https://dbdiagram.io/d/5da3f119ff5115114db53551
There's a couple things you should consider when looking at this question/functionality and what you're going to be doing with the data.
Do you have a unique set of interests that will remain the same or be added to, without duplicates?
Do you need to have multiple interests for a single user?
If you answered "yes" to both of these questions, then you should leave things the way they are. You'll be able to reuse the same interests entry for multiple users as well as each user being able to have multiple interests.
If you only want one interest per user, but want to keep a "master list" (aka: lookup table) of interests, then you can move the interest_id to the users table.
If you don't care about duplicates, but you still want each user to have multiple interests, move the name column to the users_interests table.
Finally, if you don't care about duplicates and you want each user to have a single interest, move the name column to the users table.
FYI, how you currently have the tables structured will likely take up less disk space for a large amount of users.
It's unclear what you're trying to do, so I included all options. Each option has it's own set of pros and cons, and each will be required of some other project or another at some point. You might want to learn the difference and reasoning behind why you would use one option over another now, so you don't have to think too hard about it later.
You can do it if you will make interests like this
id
user_id
name
but then you will lose uniqness of your interests, you will have many rows with the same name only because they connect to different users. Now db is perfectly ok according my understanding...It is as it should be
I am making a system similar to our Play Store's star rating system, where a product or entity is given ratings and reviews by multiple users and for each entity, the average rating is displayed.
But the problem is, whether i should store the ratings in database of each entity with a list of users who rated it and rating given, but it will make it hard for a user to check which entities he has rated, as we need to check every entity for user's existence,
Or, should i store each entity with rating in user database but it will make rendering of entity harder
So, is there a simple and efficient way in which it can be done
Or is storing same data in both databases efficient, also i found one example of this system in stackoverflow, when the store up and down votes of a question, and give +5 for up vote while - for down vote to the asking user, which means they definitely need to store each up and down vote in question database, but when user opens the question, he can see his vote, therefore it is stored in user's database
Thanx for help
I would indeed store the 'raw' version at least, so have a big table that stores the productid/entityid, userid and rating. You can query from that table directly to get any kind of result you want. Based on that you can also calculate (or re-calculate) projections if you want, so its a safe bet to store this as the source of truth.
You can start out with a simple aggregate query, as long as that is fast enough, but to optimize it, you can make projections of the data in a different format, for instance the average review score per product. This van be achieved using (materialized) views, or you can just store the aggregated rating separately whenever a vote is cast.
Updating that projected aggregate can be very lightweight as well, because you can store the average rating for an entity, together with the number of votes. So when you update the rating, you can do:
NewAverage = (AverageRating * NumberOfRatings + NewRating) / (NumberOfRatings + 1)
After that, you store the new average and increment number of ratings. So there is no need to do a full aggregation again whenever somebody casts a vote, and you got the additional benefit of tracking the number of votes too, which is often displayed as well on websites.
The easiest way to achieve this is by creating a review table that holds the user and product. so your database should look like this.
product
--id
--name
--price
user
--id
-- firstname
--lastname
review
--id
--userId
--productId
--vote
then if you want to get all review for a product by a user then you can just query
the review table. hope this solves your problem?
I'm in the process of writing a SuiteTalk integration, and I've hit an interesting data transformation issue. In the target system, we have a sort of notes table which has a category column and then the notes column. Data going into that table from NetSuite could be several different fields on a single entity in NetSuite terms, but several records of different categories in our terms.
If you take the example of a Sales Order, you might have two text fields that we need to bring across as notes. For each of those fields I need to create a row, with both the notes field in the same column but separate rows. This would allow me to add a dynamic column that give the category for each of those fields.
So instead of
SO number notes 1 notes 2
SO1234567 some text1 some text2
You’d get
SO Number Category Text
SO1234567 category 1 some text1
SO1234567 category 2 some text2
The two problems I’m really trying to solve here are:
Where can I store the category name? It can’t be the field name in NetSuite. It needs to be configurable per customer as the number of notes fields in each record type might vary across implementations. This is currently my main blocker.
Performance – I could create a saved search for each type of note, and bring one row across each time, but that’s not really an acceptable performance hit if I can do it all in one call.
I use Saved Searches in NetSuite to provide a configurable way of filtering the data to import into the target system.
If I were writing a SQL query, i would use the UNION clause, with the first column being a dynamic column denoting the category and the second column being the actual data field from NetSuite. My ideal would be if I could somehow do a similar thing either as a single saved search, or as one saved search per entity, without having to create any additional fields within NetSuite itself, so that from the SuiteTalk side I can just query the search and pull in the data.
As a temporary kludge, I now have multiple saved searches in NetSuite, one per category, and within the ID of the saved search I expect the category name and an indicator of the record type. I then have a parent search which gives me the searches for that record type - it's very clunky, and ultimately results in far too many round trips for me to be satisfied.
Any idea if something like this is at all possible?? Or if not, is there a way of solving this without hard-coding the category values in the front end? Even if I can bring back multiple recordsets in one call, that would be a performance enhancement.
I've asked the same question on the NetSuite forums but to no avail.
Thanks
At first read it sounds like you are trying to query a set of fields from entities. The fields may be custom fields or built in fields. Can you not just query the entities where your saved search has all the potential category columns and then transform the received data into categories?
Otherwise please provide more specifics in Netsuite terms about what you are trying to do.
I need to create a new collection on my Solr 6.1.0 cluster where every row is a content and every content can belong to one or many categories, which are specified in a multivalued field categories.
In my web app the user can search by categories, and if wanted it can even group results by category. If it wants to order by category, what about the contents which belong to more than one category?
In this case, the search results page should show the same content more times in different categories. I don't want the web application to filter and order results because in this case, it should ask Solr for every row (I know this is not advised for bad performance), so is there a way to let Solr make this? For example, repeating the same content in two categories if a flag is enabled or if I am asking Solr to sort by category?
Until now I bypassed the problem cloning one record for every category and specifying the category ID in a single int field. But this is not optimized, because in this case my index is much bigger than it could be, and every content metadata a part of category is just the same for every content, and because of this I would like to have 1 content = 1 Solr record.