I have an app (vb.net) which collects data from users and stores the data locally on their laptop until they sync it up with a central SQLServer 2008 database. The sync needs to be in both directions. So right now, I have a timestamp on each record that gets set when that record gets updated. Then I compare times on the records to see which is more recent. If a record on the laptop is more recent than the one on the central DB, the laptop record gets sent up. And if the record on the central DB is more recent than the laptop, that record gets sent down to the laptop.
I have several hundred thousand records spread over about 15 tables. It is taking 3 to 4 minutes to run through all of them if you are local on the network. The problem really gets worse for remote users. It takes them 20 to 30 minutes to sync. via VPN.
I have about 5 users doing this and they all need to maintain the same information with each other by way of the central database. They all sync to the central DB, not with each other.
Is there a better way to check every record other than comparing timestamps?
Note that only a handful of records (5%) change each time they sync, but I don't know which ones it may be. It could be any of them. So I have to check all of them.
Thanks.
In my opinion timestamps are not the way to go for determining which records to send to the other party.
Although they might be "ok" for conflict resolution, time differences on synchronization parties (computers), might cause records to be skipped from sending out, causing real problems.
Myself I use an identity column (on the server side) on one specific table to generate sequence nr's, and in every transaction, I get a new sequence number, and assign this to all updated/inserted rows of the other tables that need synchronization.
Now when a client requests synchronization, it provides the server with the latest 'sequence' it received during last synchronization or 0 if it is the first time.
The server would send only those records that have a greater sequence number, and then determines what the highest sequence number was on those records it actually sent to the client, and give this number to the client for next synchronization requests.
In my scenario, conflict resolution is done on the client, because all business logic is their anyway, and this means, that the client always receives updates first, before it start to send theirs.
Because you use one newly generated sequence number for every transaction, you maintain referential integrity during each synchronization, but to make sure that's actually true,
you need to determine the currently highest sequence number before you start to send synchronization data, and never retrieve any records higher then this number, because otherwise you could break referential integrity.
This because, some other thread might have committed inserts of Orders and OrderItems after you already looked up the Orders but not the OrderItems, by which you have OrderItems in your outwards synchronization package without the Order.
For deletions, I use a IsDeleted column, and the server holds records for some period before they really get deleted.
When clients insert data, I give them feedback of what (primary) keys that records where given, etc.. etc..
Well, there is so much more to this then I can mention here, but here are some key thoughts for you that you should watch carefully:
How to prevent:
Missing records
Missing deletes
Double inserts
Unnecessary sending of records (I use a nullable field LastModifierId)
Input validation
Referential integrity
Conflict resolution
Performance costs (choose the right indexes, filtered unique indexes are great for keeping track of temporary client insert identities of records, so they might also be null, you need these to prevent double inserts)
Well good luck, hope this gives food for thoughts..
Related
How to handle concurrency-related issues on a DB table if multiple applications are reading and writing on it? This case may not be specific to microservices.
OPERATION
STATUS
GET_ORDER
COMPLETE
CALCULATE_PRICE
RUNNING
A very basic use-case: multiple applications are writing in the above table. Before writing, they check if same operation is already present in RUNNING status. If not present, they insert the entry. Otherwise they just skip. Both read and write operations are simple SQL queries.
Problem is - 2 different applications can read at the same time and find that there is no 'CREATE_INVOICE' operation RUNNING, so they both will insert it in the table which will now look like:
OPERATION
STATUS
GET_ORDER
COMPLETE
CALCULATE_PRICE
RUNNING
CREATE_INVOICE
RUNNING
CREATE_INVOICE
RUNNING
As a result the table has two duplicate CREATE_INVOICE records. Besides applying unique constraint on the table, what are the ways to resolve this?
By "2 different applications" do you mean that there are two completely separate applications which create invoices, or just 2 instances of the same application?
If the former, I'd be curious why there are two applications doing the same thing writing to the same DB.
If the latter, those instances will need to coordinate in some way (a uniqueness constraint on the table is an example of such coordination), and it's important to note that this coordination makes the application a little more stateful.
My preferred way of dealing with this would be to be event driven (e.g. by tapping into database change data capture) and sharding: for instance, when a GET_ORDER record is marked COMPLETE in the DB (resulting in a CDC record being published), based on the order ID, that CDC record is always routed to the same shard in the invoice creation application (or the price calculation application for that matter; your second table seems to imply that invoice creation can be simultaneous with price calculation), thus avoiding the conflict.
I currently ETL 3.2 billion rows of data daily and then distribute it. 75% of this data is static in that it does not change more than once in a 48 hour period but some of it changes more frequently. My issue is that almost all of this data does not contain a primary key from the source (174 distinct unique sources) So we mostly just dump and load nightly. Now we are getting requests to update the datasets more often to account for the changes that do occur more often. On some of the smaller tables we have created a round robin procedure that loads one table while the other is being used and then when it finishes it swaps the table pointer so I never effectively querying and loading the same table at the same time. This works but some of the new tables have many API's with direct queries pointing at them. This makes the round robin proc solution less desirable. Any one have any thoughts on how I might address this.. just to be clear and remind everyone I do not effectively have a primary key so I cannot just update the data that changed and some of the tables contain 50+ rows so calculating a change at the row level is not really possible either (unless I am just missing something)
Background
I am designing a Data Warehouse with SQL Server 2012 and SSIS. The source system handles hotel reservations. The reservations are split between two tables, header and header line item. The Fact table would be at the line item level with some data from the header.
The issue
The challenge I have is that the reservation (and its line items) can change over time.
An example would be:
The booking is created.
A room is added to the booking (as a header line item).
The customer arrives and adds food/drinks to their reservation (more line items).
A payment is added to the reservation (as a line item).
A room could be subsequently cancelled and removed from the booking (a line item is deleted).
The number of people in a room can change, affecting that line item.
The booking status changes from "Provisional" to "Confirmed" at a point in its life cycle.
Those last three points are key, I'm not sure how I would keep that line updated without looking up the record and updating it. The business would like to keep track of the updates and deletions.
I'm resisting updating because:
From what I've read about Fact tables, its not good practice to revisit rows once they've been written into the table.
I could do this with a look-up component but with upward of 45 million rows, is that the best approach?
The questions
What type of Fact table or loading solution should I go for?
Should I be updating the records, if so how can I best do that?
I'm open to any suggestions!
Additional Questions (following answer from ElectricLlama):
The fact does have a 1:1 relationship with the source. You talk about possible constraints on future development. Would you be able to elaborate on the type of constraints I would face?
Each line item will have a modified (and created date). Are you saying that I should delete all records from the fact table which have been modified since the last import and add them again (sounds logical)?
If the answer to 2 is "yes" then for auditing purposes would I write the current fact records to a separate table before deleting them?
In point one, you mention deleting/inserting the last x days bookings based on reservation date. I can understand inserting new bookings. I'm just trying to understand why I would delete?
If you effectively have a 1:1 between the source line and the fact, and you store a source system booking code in the fact (no dimensional modelling rules against that) then I suggest you have a two step load process.
delete/insert the last x days bookings based on reservation date (or whatever you consider to be the primary fact date),
delete/insert based on all source booking codes that have changed (you will of course have to know this beforehand)
You just need to consider what constraints this puts on future development, i.e. when you get additional source systems to add, you'll need to maintain the 1:1 fact to source line relationship to keep your load process consistent.
I've never updated a fact record in a dataload process, but always delete/insert a certain data domain (i.e. that domain might be trailing 20 days or source system booking code). This is effectively the same as an update but also takes cares of deletes.
With regards to auditing changes in the source, I suggest you write that to a different table altogether, not the main fact, as it's purpose will be audit, not analysis.
The requirement to identify changed records in the source (for data loads and auditing) implies you will need to create triggers and log tables in the source or enable native SQL Server CDC if possible.
At all costs avoid using the SSIS lookup component as it is totally ineffective and would certainly be unable to operate on 45 million rows.
Stick with the 'insert/delete a data portion' approach as it lends itself to SSIS ability to insert/delete (and its inability to efficiently update or lookup)
In answer to the follow up questions:
1:1 relationship in fact
What I'm getting at is you have no visibility on any future systems that need to be integrated, or any visibility on what future upgrades to your existing source system might do. This 1:1 mapping introduces a design constraint (its not really a constraint, more a framework). Thinking about it, any new system does not need to follow this particular load design, as long as it's data arrive in the fact consistently. I think implementing this 1:1 design is a good idea, just trying to consider any downside.
If your source has a reliable modified date then you're in luck as you can do a differential load - only load changed records. I suggest you:
Load all recently modified records (last 5 days?) into a staging table
Do a DELETE/INSERT based on the record key. Do the delete inside SSIS in an execute SQL task, don't mess about with feeding data flows into row-by-row delete statements.
Audit table:
The simplest and most accurate way to do this is simply implement triggers and logs in the source system and keep it totally separate to your star schema.
If you do want this captured as part of your load process, I suggest you do a comparison between your staging table and the existing audit table and only write new audit rows, i.e. reservation X last modified date in the audit table is 2 Apr, but reservation X last modified date in the staging table is 4 Apr, so write this change as a new record to the audit table. Note that if you do a daily load, any changes in between won't be recorded, that's why I suggest triggers and logs in the source.
DELETE/INSERT records in Fact
This is more about ensuring you have an overlapping window in your load process, so that if the process fails for a couple of days (as they always do), you have some contingency there, and it will seamlessly pick the process back up once it's working again. This is not so important in your case as you have a modified date to identify differential changes, but normally for example I would pick a transaction date and delete, say 7 trailing days. This means that my load process can be borken for 6 days, and if I fix it by the seventh day everything will reload properly without needing extra intervention to load the intermediate days.
I would suggest having a deleted flag and update that instead of deleting. Your performance will also be better.
This will enable you to perform an analysis on how the reservations are changing over a period of time. You will need to ensure that this flag is used in all the analysis to ensure that there is no confusion.
We are building a large stock and forex trading platform using a relational database. At any point during the day there will be thousands, if not millions, of records in our Orders table. Some orders, if not fulfilled immediately, expire and must be removed from this table, otherwise, the table grows very quickly. Each order has an expiration time. Once an order expires it must be deleted. Attempting to do this manually using a scheduled job that scans and deletes records is very slow and hinders the performance of the system. We need to force the record to basically delete itself.
Is there way to configure any RDBMS database to automatically remove a record based on a date/time field if the time occurs in the past?
Since you most likely will have to implement complex order handling, e.g. limit orders, stop-limit orders etc. you need a robust mechanism for monitoring and executing orders in real time. This process is not only limited to expired orders. This is a core mechanism in a trading platform and you will have to design a robust solution that fulfill your needs.
To answer your question: Delete expired orders as part of your normal order handling.
Why must the row be deleted?
I think you are putting the cart before the horse here. If a row is expired, it can be made "invisible" to other parts of the system in many ways, including views which only show orders meeting certain criteria. Having extra deleted rows around should not hamper performance if your database is appropriately indexed.
What level of auditing and tracking is necessary? Is no analysis ever done on expired orders?
Do fulfilled orders become some other kind of document/entity?
There are techniques in many databases which allow you to partition tables. Using the partition function, it is possible to regularly purge partitions (of like rows) much more easily.
You have not specified what DB you are using but lets assume you use MSSQL you could create a agent job that runs periodicly, but you are saying that that might not be a solution for you.
So what t about having an Insert Trigger that when new record is inserted you delete all the record that are expired? This will keep number of record all relatively small.
I'm storing a set of messages in a SQL table. Each message has a size and there's a column in the table which contains the size of the message. These messages are connected to accounts. When a new message arrives, I need to check that the current account size + the new message size is less than the quota for the account (which is just a "maxaccountsize" column in a row in the accounts table). If not, I need to report back to the sender that the message does not fit in the account.
To simplify:
Table messages:
ID int
AccountID int
Size int
Table accounts:
ID int
MaxSize int
To calculate the total size of each account, I execute statements similar to SELECT SUM(Size) from messages WHERE AccountID = 12345.
In a large user databases where there's hundreds of thousands of messages in accounts, this operation is heavy and becomes a big bottleneck when receiving a message. My software use both Microsoft SQL Server, MySQL and PostgreSQL as backend.
To solve this, I've added some in-memory caching of the value. This is cumbersome to me since I need to implement thread-safe updates of the cache, and I need to make sure that the cache is always up to date. Also, it doesn't work if someone manually edits the database.
An alternative solution would be to store the current account size in the accounts table. However, this would mean I have somewhat redundant data (of course, one can say that this is already the case today with my in-memory cache). If I choose this solution, I need to make sure I always update the account size when creating or deleting messages. This is also a bit cumbersome and I can bet that there will be times when the sum(size) does not equal to the CurrentAccountSize value in Accounts row. With the in-memory cache, at least it will be reset to its correct value when the server is restarted.
Does anyone have an opinion on what should be done in situations like these?
In this use case I would definitely store redundant data in your database.
Do you think your bank account calculates the sum of all transactions in its history when calculating your balance?