With things like SO, Digg, Reddit, etc...
Should one keep track of downvotes in the database independent of upvotes? Or should they simply have a "votes" field that is decremented/incremented based off what the user does with no persisting of that?
How should votes be handled?
On SO, up votes earn +10, down votes -2. For this to work they need to be tracked separately. It's quite possible for a controversial answer to generate a few of each, and just showing an aggregate number won't mean much. So I'd say keep them separate.
I would keep them separate. Some questions have a lot of activity (up and down) and you really like to identify those.
Even if you are not interested in the difference right now, an extra field in the table is not that expensive, so it does not hurt to separate it. Because if you want to add it later, there is no way you can retrieve the data if it is not stored separately.
I also assume SO keeps separate votes for CW and non CW entries. Because if the question changes to CW later on, even with a recalc, the original gained/lost rep is kept.
Depends what you wish to do with your data.
If you only want to display votes than I say you only use one field. It's like number of views of a thread on forum. You want to see what gets most clicks, but not how many times someone viewed it.
Voting system on SO is a bit more complex. Since they can cancel all votes from particular user they have to keep track of who voted for/against what. This, I think, is written in another table, but because it is expensive to recalculate all votes every time someone views a question, they keep calculated value in a field, changing it whenever someone votes.
I can advise you to store them separately maybe even with extra data who authored a particular up or downvote. Who knows, you may come up with a nice idea tomorrow and you will need this additional data to implement it.
But it would also be good to have a sort of pre-calculated field (let's call it cache) which is updated whenever an up- or a downvote is submitted. The pages will then be rendered with this precalculated field. This will increase the response time and lessen load on DB.
If it is too costy to recalculate values immediately you may consider runnning some scheduler tasks (once per hour?) which will process up-to-date votes and recalculate the cached values.
Considering the amount of data you'll have in the database for social voting website the additional space for an extra int column to store down-votes is going to negligible so you'd be crazy not to.
Well, given that SO has +10 for an upvote and -2 for a downvote, and there's recalculation occasionally going on, it would need to store them independently.
Otherwise an answer with 10 upvotes and 5 downvotes which originally gave you 90 points, this would recalc to 50 if they weren't stored separately.
I'd keep them separate so that I could review them separately. Socially, upvoting is very different to downvoting, and I'd want to be able to look at them independently if it were me.
Definitely separately for one very simple reason. Tomorrow you'll want to do something extra that needs that information (some sort of report or graph for example). Besides keeping them separately costs you nothing.
since people can usually vote once, and (in SO for example) can cancel their votes, you need to know who voted, at what time, what vote, and on which item.
I am certain that the downvotes and upvotes are kept separately, though there could be an aggregate field that keeps the count. SO lets you change a vote later (make the downvote an upvote), and that's why I believe the votes are logged for each user too.
Related
been trying to solve this problem for a week and couldn't come up with any solutions in all my research so I thought I'd ask you all.
I have a "Product" table and a "productSent" table, here's a quick scheme to help explain:
class Product(ndb.Model):
name = ndb.StringProperty();
rating = ndb.IntegerProperty
class productSent(ndb.Model): <--- the key name here is md5(Product Key+UUID)
pId = ndb.KeyProperty(kind=Product)
uuId = ndb.KeyProperty(kind=userData)
action = ndb.StringProperty()
date = ndb.DateTimeProperty(auto_now_add=True)
My goal is to show users the highest rated product that they've never seen before--fast. So to keep track of the products users have seen, I use the productSent table. I created this table instead of using Cursors because every time the rating order changes, there's a possibility that the cursor skips the new higher ranking product. An example: assume the user has seen products 1-24 in the db. Next, 5 users liked product #25, making it the #10 product in the database--I'm worried that the product will never be shown again to the user (and possibly mess things up on a higher scale).
The problem with the way I'm doing it right now is that, once the user has blown past the first 1,000 products, it really starts slowing down the query performance. Because I'm literally pulling 1,000+ results, checking if they've been sent by querying against the productSent table (doing a keyName lookup to speed things up) and going through the loop until 15 new ones have been detected.
One solution I thought of was to add a repeated property (listProperty) to the Product table of all the users who have seen a product. Or if I don't want to have inequality filters I could put a repeated property of all the users who haven't seen a product. That way when I query I can dynamically take those out. But I'm afraid of what happens when I have 1,000+ users:
a) I'll go through the roof on the limit of repeated properties in one entity.
b) The index size will increase size costs
Has anyone dealt with this problem before (I'm sure someone has!) Any tips on the best way to structure it?
update
Okay, so had another idea. In order to minimize the changes that take place when a rating (number of likes) changes, I could have a secondary column that only has 3 possible values: positive, neutral, negative. And sort by that? Ofcourse for items that have a rating of 0 and get a 'like' (making them a positive) would still have a chance of being out of order or skipped by the cursor--but it'd be less likely. What do y'all think?
Sounds like the inverse, productNotSent would work well here. Every time you add a new product, you would add a new productNotSent entity for each user. When the user wants to see the highest rated product they have not seen, you will only have to query over the productNotSent entities that match that user. If you put the rating directly on the productNotSent you could speed the query up even more, since you will only have to query against one Model.
Another idea would be to limit the number of productNotSent entities per user. So each user only has ~100 of these entities at a time. This would mean your query would be constant for each user, regardless of the number of products or users you have. The creation of new productNotSent entities would become more complex, though. You'd have to have a cron job or something that "tops up" a user's collection of productNotSent entities when they use some up. You also may want to double-check that products rated higher than those already within the user's set of productNotSent entities get pushed in there. These are a little more difficult and well require some design trade-offs.
Hope this helps!
I do not know your expected volumes and exact issues (only did a quick perusal of your question), but you may consider using Json TextProperty storage as part of your plan. Create dictionaries/lists and store them in records by json.dump()ing them to a TextProperty. When the client calls, simply send the TextProperties to the client, and figure everything out on the client side once you JSON.parse() them. We have done some very large array/object processing in JS this way, and it is very fast (particularly indexed arrays). When the user clicks on something, send a transaction back to update their record. Set up some pull or push queue processes to handle your overall product listing updates, major customer rec updates, etc.
One downside is higher bandwidth going out of you app, but I think this cost will be minimal given potential processing savings on GAE. If you structure this right, you may be able to use get_by_id() to replace all or most of your planned indices and queries. We have found json.loads() and json.dumps() to be very fast inside the app, but we only use simple dictionary/list structures.This approach will be, though, a big, big quantum measure lower than your planned use of queries. The other potential issue is that very large objects may run into soft memory limits. Be sure that your Json objects are fairly simple+lightweight to avoid this (e.g. do no include product description, sub-objects, etc. in the Json item, just the basics such as product number). HTH, -stevep
I am working on a voting system for my company, in this system, each user can vote in one or more topics. Voting in a topic will cause this topic to get +1 in reputation while all other topics will lose -1 in reputation. (These numbers are rather arbitrary, the real values are a complex formula).
I was thinking to structure my database creating a view where I will store the ID for each topics, the total of votes for this topic, the total of votes for other topics, and the calculated reputation.
My problem is regarding performance/scalability. I think this solution is not very effective, because if I have 100 topics and 2 votes per second, I will have to refresh all the values for all the topics two times a second, that would be a total of 200 records per second.
Is this analysis correct? Is mssql engine going to handle it easily? Or should I look for some other approach? What do you suggest?
thanks,
Oscar
This is going to be very hard on the database.
What variables does your formula depend on? If the amounts you're adding and subtracting are somewhat global, you can keep track of them and only calculate the score of each topic when you need to display it.
For example, you could increment the score of one topic only, but also increment a global variable that keeps track of how many "upvotes" there are in total. that way when you want to display the score of any topic, you would be able to calculate it on the fly without bombarding your database.
More information about the formula can get you better advice.
EDIT:
You have this formula:
origninalRep + (upvotes*10 - otherVotes)
You can get originalRep straightforward with one query.
Same for upvotes.
The "tricky" part would be otherVotes. But that's simply globalVotesCount - upvotes, and you have those if you keep track of the global upvotes count.
I hope this helps.
I'd love some opinions on whether this database design I'm currently pursuing is sound or not.
Lets assume I'm building a table called "Home", this table has a text field called "rooms". In this field is the serialized data for a set of rooms that this house has. My first instinct was to, of course, normalize this data into a separate "Rooms" table. However, due to some frustrating experiences with overly normalized databases in the past, I stopped to ask myself a few questions:
Will I ever need to find a specific room?
Will I ever need to update an individual room?
Will any Home records ever share Room records?
The answer to each of these questions is "no". Room records are all unique to each Home. Queries will never need to be performed to find out how many Homes in the database have bathrooms, for instance. Data will always be pulled from the perspective of the Home. The number of bedrooms and bathrooms will be explicitly stored on the Home record for searching.
So instead of having to constantly join Rooms, I wondered what would be the harm in serializing this data and just popping it into a text field.
This makes a lot of sense to me, but I'm hoping for a sanity check. Thanks for any input!
A pragmatic answer...
a) probability that you might want to decompose it in the future
b) benefit of not doing so now
c) cost of changing the schema later on.
If a * c > b then you should decompose now.
Well, you might not have a need TODAY to query to find out things like:
What is the average number of bathrooms in a home in Ohio?
Where do homes have more bedrooms? The East Coast or the West Coast?
How does house price correlate with the size of the master bedroom? What would be the average dollar value return of increasing the master bedroom size by 30%?
etc, etc.
You will be in a much better position in the future if you design your foundation correctly to begin with... no matter how enticing the short-cut may seem right now.
Plus, with a separate ROOMS table, you will be able to add additional room fields that make sense later (like width/height, color, floor level, etc.) which would all be very hard if the data were just globbed into a single field.
People will want to query in unexpected ways, like:
I have bad knees. Can you list houses with the master bedroom and master bathroom on the first floor?
In general, having a ROOMS table will just make your application more powerful, and easier to use.
Hey, I get what you're saying about "overly normalized data". We've all been there, and it DOES bite. However, having a ROOMS table in a database with housing info isn't being "overly normalized". It's just building the app the right way.
In addition to what others have said about doing the right thing, I would like to add a comment about performance.
Since you will be storing the serialized room data as a column in table Home, the row size will increase significantly. This will result in worse performance for all other queries.
Well, you say that room records are unique, but you can't enforce that. So you have no way to know this for sure in your current design: all your code should be perfect in representing this.
"constantly joining" isn't that hard to do, but if it is, you can always make a View for that, and you're done.
What are the advantages and disadvantages?
That depends on what your site is, and how you use that information. On StackOverflow, you are awarded a "yearling" badge once a full year elapses from the time you registered. Clearly here that information is necessary.
If I were you, I'd save it. It's a small piece of information that may become useful eventually. It's better to have it and not need it than to need it and not have it. It would be rather difficult to extrapolate an accurate registration date retrospectively if you don't store it to begin with.
Advantage:
You don't get in a migration horror when needing it at some point. For a lot of data you cannot find out this data afterwards. You could fake around with MODIFICATION_DATE but often this is not accurate and sits in the future (e.g. when profile can be edited by user).
Disadvantage:
In case you never need this information, you wasted space (though another small data payload column shouldn't make a problem). Further more you have an 'all-time' deprecated field, which can be confusing to new developers ("what is this column for, cannot see where it is used...?")
As mentioned the registration-date is most likely a valuable information I would add it from start on. When thinking of persistent data and its model you sometimes have to think "more" for the future.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
This is a question not really about "programming" (is not specific to any language or database), but more of design and architecture. It's also a question of the type "What the best way to do X". I hope does no cause to much "religious" controversy.
In the past I have developed systems that in one way or another, keep some form of inventory of items (not relevant what items). Some using languages/DB's that do not support transactions. In those cases I opted not to save item quantity on hand in a field in the item record. Instead the quantity on hand is calculated totaling inventory received - total of inventory sold. This has resulted in almost no discrepancies in inventory because of software. The tables are properly indexed and the performance is good. There is a archiving process in case the amount of record start to affect performance.
Now, few years ago I started working in this company, and I inherited a system that tracks inventory. But the quantity is saved in a field. When an entry is registered, the quantity received is added to the quantity field for the item. When an item is sold, the quantity is subtracted. This has resulted in discrepancies. In my opinion this is not the right approach, but the previous programmers here swear by it.
I would like to know if there is a consensus on what's the right way is to design such system. Also what resources are available, printed or online, to seek guidance on this.
Thanks
I have seen both approaches at my current company and would definitely lean towards the first (calculating totals based on stock transactions).
If you are only storing a total quantity in a field somewhere, you have no idea how you arrived at that number. There is no transactional history and you can end up with problems.
The last system I wrote tracks stock by storing each transaction as a record with a positive or negative quantity. I have found it works very well.
The Data Model Resource Book, Vol. 1: A Library of Universal Data Models for All Enterprises
The Data Model Resource Book, Vol. 2: A Library of Data Models for Specific Industries
The Data Model Resource Book: Universal Patterns for Data Modeling
I have Vol 1 and Vol 2 and these have been pretty helpful in the past.
It depends, inventory systems are about far more than just counting items. For example, for accounting purposes, you might need to know accounting value of inventory based on FIFO (First-in-First-out) model. That can't be calculated by simple "totaling inventory received - total of inventory sold" formula. But their model might calculate this easily, because they modify accounting value as they go. I don't want to go into details because this is not programming issue but if they swear by it, maybe you didn't understand fully all their requirements they have to accommodate.
both are valid, depending on the circumstances. The former is best when the following conditions hold:
the number of items to sum is relatively small
there are few or no exceptional cases to consider (returns, adjustments, et al)
the inventory item quantity is not needed very often
on the other hand, if you have a large number of items, several exceptional cases, and frequent access, it will be more efficient to maintain the item quantity
also note that if your system has discrepancies then it has bugs which should be tracked down and eliminated
i have done systems both ways, and both ways can work just fine - as long as you don't ignore the bugs!
It's important to consider the existing system and the cost and risk of changing it. I work with a database that stores inventory kind of like yours does, but it includes audit cycles and stores adjustments just like receipts. It seems to work well, but everyone involved is well trained, and the warehouse staff aren't exactly quick to learn new procedures.
In your case, if you're looking for a little more tracking without changing the whole db structure then I'd suggest adding a tracking table (kind of like from your 'transaction' solution) and then log changes to the inventory level. It shouldn't be too hard to update most changes to the inventory level so that they also leave a transaction record. You could also add a periodic task to backup the inventory level to the transaction table every couple hours or so so that even if you miss a transaction you can discover when the change happened or roll back to a previous state.
If you want to see how a large application does it take a look at SugarCRM, they have and inventory management module though I'm not sure how it stores the data.
I think this is actually a general best-practices question about doing a (relatively) expensive count every time you need a total vs. doing that count every time something changes, then storing the count in a field and reading that field whenever you need a total.
If I couldn't use transactions, I would go with the live count every time I needed a total. If transactions are available, it would be safe to perform the inventory update operations and the saving of the re-counted total within the same transaction, which would ensure the accuracy of the count (although I'm not sure this would work with multiple users hitting the database).
But if performance is not really a huge problem (and modern databases are good enough at counting rows that I would rarely even worry about this) I'd just stick with the live count each time.
I would opt for the first way, where
the quantity on hand is calculated
totaling inventory received - total of
inventory sold
The Right Way, IMO.
EDIT: I would also want to factor in any stock losses/damages into the system, but I'm sure you have that covered.
I've worked on systems that solve this problem before. I think the ideal solution is a precomputed column, which gets you the best of both worlds. Your total would be a field somewhere, thus no expensive lookups, but it can't get out of sync with the rest of your data (the database maintains the integrity). I don't remember which RDMSs support precomputed columns, but if you don't have transactions, that might not be available either.
You could potentially fake precomputed columns (very effectively... I see no downside) using triggers. You'd probably need transactions though. IMHO, keeping data integrity when you're doing this sort of controlled denormalization is the only legitimate use for a trigger.
Django-inventory geared more to fixed assets, but might give you some ideas.
IE: ItemTemplate (class) -> ItemsOnHand (instance)
ItemsOnHand can be linked to more ItemTemplates; Example Printer & the ink cartridges is requires. This also allows to set Reorder points for each ItemOnHand.
Each ItemsOnHand is linked to InventoryTransactions, this allows for easy auditing.
To avoid calculating actual on hand items from thousand of invetory transactions, checkpoints are used which are just a balance + a date. To calculate items on hand query to find the most recent checkpoint and start adding or substracting items to find the current balance of items. Define new checkpoints periodically.
I can see some benefit to having the two columns, but I'm not following the part about discrepancies - you seem to be implying that having the two columns (in and out) is less prone to discrepancy than a single column (current). Why is that?
Is not having one or two columns, what I meant with "totaling inventory received - total of inventory sold" is something like this:
Select sum(quantity) as inventory_received from Inventory_entry
Select sum(quantity) as inventory_sold from Sales_items
then
Qunatity_on_hand = inventory_received - inventory_sold
Please keep in mind that I oversimplified this and my initial explanation. I know there is much more to inventory that just keeping track of quantities, but in this case that's were the problem lies and what we want to fix. At this point the reason to change it is preciselly the cost of supporting the problems caused by the current design.
Also I wanted to mention that although this is not a "coding" question is related to algoritms and design which IMHO are very important topics.
Thanks everybody for your answers so far.
Nelson Marmol
We solve different problems, but our approach to some of them might be interesting to you.
We allow the system to make a "best guess", and give the users regular feedback about any of those guesses that look wrong.
To apply this to inventory, you could have 3 fields:
inventory_received
inventory_sold
estimated_on_hand
Then, you could run a process (daily?) along the lines of:
SELECT *
FROM Inventory
WHERE estimated_on_hand != inventory_received - inventory_sold
Of course, this relies on users looking at this alert, and doing something about it.
Also, you could have a function to reset inventory some how, either by updating inventory_sold/received, or perhaps adding another field "inventory_adjustment", which could be positive or negative.
... just some thoughts. Hope it's helpful.