It is necessary to insert some data in DB once each web service method is called: in the beginning of the request processing and in the end.
My intention is to insert record that will contains all income information in the beginning of request processing and after that update the same record once request is processed and data are ready to be send back (or error is occurred and I is need to store error message).
The problem is that income data can be pretty long and LINQ To SQL before update need to fetch object data from DB and then "store" it again. In this case "income data" is going 3 times:
1st time when inserting - it goes into DB;
2nd time before object update - it is fetched from DB;
3rd time on update - it is going to DB again.
Is there any possibility to optimize such process if I already have object fetched from DB?
Is the same applied to Entity Framework? Does it allow to update only the part of object?
An ORM is geared towards converting complete rows to complete objects, and back again - so updates are always to the full object.
However, both Linq-to-SQL as well as Entity Framework are definitely smart enough to find out what properties have changed on an entity, so if you only update some fields, the generated SQL command using UPDATE will only update those changed fields.
So basically: you just try it! Fire up SQL profiler and see what SQL goes to the database; in Entity Framework, I'm positive that if you only change some fields, only those changed fields will be updated in an UPDATE statement and nothing else.
Related
So here is the issue
We are loading data into a CustomObject__c using DataLoader.
Usually the no of records that are passed are 3.
Also, if there is any issue with the data passed, they run the dataloader again and pass the corrected data. Now, the older data has to be deleted.
So, I am handling it in before insert code and calling a batch in after insert code.
Here is the code for my trigger:
trigger TriggerCustom on CustomObject__c (before insert, after insert) {
List<CustomObject__c> customobjectlist = [Select Id from CustomObject__c WHERE CreatedDate = TODAY ];
if (Trigger.isBefore) {
delete exchlisttoday;
}
if(Trigger.isAfter)
{
BatchApex b = BatchApex();
Database.executebatch(b);
}
}
This was designed keeping in mind they pass only 3 records at a time.
However, now they want to pass more than 200 records using data loader.
How can I modify my trigger so that it fires only after one single dataload is completed (for e.g. if they pass 1000 records at once, the trigger has to fire only after the 1000 records are completely inserted
Trigger will not know when you are done, after 3, 203 or 10000 records (you can use bulk api to load large volumes, they'll be chunked into 10K packets but still - triggers will work 200 at a time).
If you have scripted data load - maybe you can update something else as next step. Another object (something dummy that has just 1 record) and have trigger on this?
If you have scripted data load - maybe you can query the Ids and then pass them to delete operation which would run before the upload task. This becomes bit too much for poor little data loader but Talend, Informatica, Azure Data Factory, Jitterbit etc proper ETL tools could do it. (although deleting before is bit brave... what if the load fails? You're screwed... Maybe delete should be after successful update)
Maybe you can guarantee that last record in your daily load will have some flag set and in the trigger - look for that flag?
Maybe you can schedule the batch to run every hour. You can't do it easily from UI but you can write the cron expression and schedule as 1-liner in dev console. In the Schedulable's execute() make it check if there was anything loaded today and if there was even single record - trigger the batch?
I have a Logic App that uses the "SQL Server - When an item is modified (V2)" trigger, monitoring an Azure SQL DB for updated rows. When running this LA, I noticed that the modified row that came as output for this trigger did NOT contain the updated data.
I thought this might be by design (don't really see why, but ok...) so I added a "Get Row" action directly after the trigger, to go fetch the most recent data for the row that triggered the LA. But even this step still returned the old, not-updated data for that row.
However, when I resubmit the run some seconds later, the "Get Row" action does get the updated data from the database.
Is this normal behavior? Is the SQL DB row version already updated even though the data update isn't committed yet, triggering the Logic App but not returning the updated data yet?
Thanks for pointing me to add a timestamp to my table, I add the timestamp and then I can find the table in the selection. I test it in my side but the trigger works fine, it output the updated data. I provide my logic below for your reference:
My table show as:
My logic app:
Please note I disable the "Split On" in "Settings" of the trigger.
After running the update sql:
update Table3 set name = 'hury1' where id = 1;
update Table3 set name = 'jim1' where id = 2;
I got the result (the variable updateItems in screenshot contains both updated items):
I have a MariaDB and a Windows Service accessing this DB. For maintenance, I use HeidiSQL.
I now want to update a big table (8.000.000 entries) in HeidiSQL by inserting a new foreign key column and then filling the column with values using UPDATE. I suppose it may take about 30 minutes.
During this time, if a user wants to insert/read/delete values out of this table via the service, what will happen? Will the DB block the request? Should I stop the service to avoid corruption of data?
I made an example myself. The database seems to respond with the old values or status as long as the operations changing the data are still running in HeidiSQL.
What I tried:
I added a new column. While the adding process was still running I
sent a read message to my service. It responded without sending the
new column. As soon as the operation was finished, the new column has been sent, too.
I filled the new FK column with values. While the updating was
running I sent a read message. The service gave back the initial
values of the FK column (0) for all the rows. After the operation was
finished, the service would sent all the new values.
I'm trying to create SSIS package which will periodically send data to other database. I want to send only new records(I need to keep sent records) so I created status column in my source table.
I want my package to update this column after successfuly sending data, but I can't update all rows wih "unsent" status because during package execution some rows may have been added, and I also can't use transactions(I mean on isolation levels that would solve my problem: I can't use Serializable beacause i musn't prevent users from adding new rows, and Sequence Container doesn't support Snapshot).
My next idea was to use recordset and after sending data to other db use it to get ids of sent rows, but I couldn't find a way to use it as datasource.
I don't think I should set status "to send" and then update it to "sent", I believe it would be to costly.
Now I'm thinking about using temporary table, but I'm not convinced that this is the right way to do it, am I missing something?
Record Set is a destination. You cannot use it in Data Flow task.
But since the data is saved to a variable, it is available in the Control flow.
After completing the DataFlow, come to the control flow and create a foreach component that can run on the ResultSet varialbe.
Read each Record Set value into a variable and use it to run an update query.
Also, see if "Lookup Transform" can be useful to you. You can generate rows that match or doesn't match.
I will improve the answer based on discussions
What you have here is a very typical data mirroring problem. To start with, I would not simply have a boolean that signifies that a record was "sent" to the destination (mirror) database. At the very least, I would put a LastUpdated datetime column in the source table, and have triggers on that table, on insert and update, that put the system date into that column. Then, every day I would execute an SSIS package that reads the records updated in the last week, checks to see if those records exist in the destination, splitting the datastream into records already existing and records that do not exist in the destination. For those that do exist, if the LastUpdated in the destination is less than the LastUpdated in the source, then update them with the values from the source. For those that do not exist in the destination, insert the record from the source.
It gets a little more interesting if you also have to deal with record deletions.
I know it may seem wasteful to read and check a week's worth, every day, but your database should hardly feel it, it provides a lot of good double checking, and saves you a lot of headaches by providing a simple, error tolerant algorithm. Some record does not get transferred because of some hiccup on the network, no worries, it gets picked up the next day.
I would still set up the SSIS package as a server task that sends me an email with any errors, so that I can keep track. Most days, you get no errors, and when there are errors, you can wait a day or resolve the cause and let the next days run pick up the problems.
I am doing a similar thing, in my case, I have a status on the source record.
I read in all records with a status of new.
Then use a OLE DB Command to execute SQL on each row, changing
the status to "In progress"(in you where, enter a ? as the value in
the Component Property tab, and you can configure it as a parameter
from the table row like an ID or some pk in the Column Mappings
tab).
Once the records are processed, you can change all "In Progress"
records to "Success" or something similar using another OLE DB
Command.
Depending on what you are doing, you can use the status to mark records that errored at some point, and require further attention.
We have a SQL Server database table that consists of user id, some numeric value, e.g. balance, and a version column.
We have multiple threads updating this table's value column in parallel, each in its own transaction and session (we're using a session-per-thread model). Since we want all logical transaction to occur, each thread does the following:
load the current row (mapped to a type).
make the change to the value, based on old value. (e.g. add 50).
session.update(obj)
session.flush() (since we're optimistic, we want to make sure we had the correct version value prior to the update)
if step 4 (flush) threw StaleStateException, refresh the object (with lockmode.read) and goto step 1
we only do this a certain number of times per logical transaction, if we can't commit it after X attempts, we reject the logical transaction.
each such thread commits periodically, e.g. after 100 successful logical transactions, to keep commit-induced I/O to manageable levels. meaning - we have a single database transaction (per transaction) with multiple flushes, at least once per logical change.
what's the problem here, you ask? well, on commits we see changes to failed logical objects.
specifically, if the value was 50 when we went through step 1 (for the first time), and we tried to update it to 100 (but we failed since e.g. another thread changed it to 70), then the value of 50 is committed for this row. obviously this is incorrect.
What are we missing here?
Well, I do not have a ton of experience here, but one thing I remember reading in the documentation is that if an exception occurs, you are supposed to immediately rollback the transaction and dispose of the session. Perhaps your issue is related to the session being in an inconsistent state?
Also, calling update in your code here is not necessary. Since you loaded the object in that session, it is already being tracked by nhibernate.
If you want to make your changes anyway, why do you bother with row versioning? It sounds like you should get the same result if you simply always update the data and let the last transaction win.
As to why the update becomes permanent, it depends on what the SQL statements for the version check/update look like and on your transaction control, which you left out of the code example. If you turn on the Hibernate SQL logging it will probably become obvious how this is happening.
I'm not a nhibernate guru, but answer seems simple.
When nhibernate loads an object, it expects it not to change in db as long as it's in nhibernate session cache.
As you mentioned - you got multi thread app.
This is what happens=>
1st thread loads an entity
2nd thread loads an entity
1st thread changes entity
2nd thread changes entity and => finds out that loaded entity has changed by something else and being afraid that it has screwed up changes 1st thread made - throws an exception to let programmer be aware about that.
You are missing locking mechanism. Can't tell much about how to apply that properly and elegantly. Maybe Transaction would help.
We had similar problems when we used nhibernate and raw ado.net concurrently (luckily - just for querying - at least for production code). All we had to do - force updating db on insert/update so we could actually query something through full-text search for some specific entities.
Had StaleStateException in integration tests when we used raw ado.net to reset db. NHibernate session was alive through bunch of tests, but every test tried to cleanup db without awareness of NHibernate.
Here is the documention for exception in the session
http://nhibernate.info/doc/nhibernate-reference/best-practices.html