I am having to create a custom tool in .NET to data migrate and cleanse an existing system. And it also does some other things.
I have literally a table of around 2000 users and then for each of those users, they can have anything from 0 to 9,000 "customer accounts"
then for each of these users and their accounts, they will need to insert a menu system into another table for that user and account.
So I am having to go through all these objects created within the app, and execute an insert statement. The problem is, the performance is horrible. It took 3.5 hours to complete the inserts and each insert took around 3 seconds to complete.... pretty slow!
With the inserts, I am making sure that I dont insert duplicate data/make sure there isnt any existing duplicate data thus my insert statment:
IF NOT EXISTS (SELECT ID FROM UserWebAccessLevel WHERE UserID = '{0}' AND CustomerID = '{1}' AND MenuID = {2}) BEGIN INSERT INTO [WebAccessLevel] (UserID, CustomerID, MenuID) VALUES ('{0}', '{1}', {2}) END
doing this for each user, for their customers and for each menu item found within the app logic.... yeh, takes a long time.
im wondering how I can better enhance the performance of the insert statement?
Thank you
Without an index on your table, the SELECT ID FROM... could be slowing things down, as SQL Server has to look at each record one by one and compare the IDs in the WHERE clause. By adding an index containing the three IDs, SQL Server will be able to (more-or-less) instantly locate an existing record when a WHERE clause specifies the three IDs.
Assuming you are using SQL Server Management Studio:-
Right-click the table then click "Design".
Right-click where the list of columns has appeared, then click "Indexes/Keys".
in the next dialog that appears, click the Add button
In the list of properties on the r.h. side, find the one called "Columns" and click the box to the right of it. When you see the ellipses button ("...") click that.
In the next dialog that appears, add your three ID columns.
An index doesn't have to be unique. In the "Indexes/Keys" dialog you get the option to specify whether it should be unique or not.
The only other thing I can suggest checking is in your C#/VB code that executes the IF NOT EXISTS... SQL - make sure you aren't opening and closing the connection each time. That will really slow things down!
Related
This is similar to another question and I have given it the same name. But my situation is a bit different.
The first question for reference: Access Linked to SQL: Wrong data shown for a newly created record
I have an Access front end linked to tables in SQL Server. For all relevant tables, there is an autonumber (int with Identity Specification) as Primary Key. About half of the linked tables have the following issue, the others do not, despite being set up similarly:
When adding a new record to the table, the record is inserted in the SQL database, but then in the access front end view, be it a table or form, the added record is filled up with data of another record.
In the other question, it was explained that Access is querying SQL Server with ##IDENTITY. I saw the same thing in a trace. In my case it tries SELECT ##IDENTITY twice, then attempts to pull the new record with a sp_prepexec generated SQL that I can't read, and consistently gets the wrong one, in certain tables, not in others, which are set up basically the same.
The wrong record being returned seems to be an earlier autonumber in the table, and if I do it several times in a row, it returns a series of autonumbers in sequence, for instance, 18347, 18348, 18349. (These are the incorrect autonumbers being displayed, along with all data from their records, instead of the newly created record.) But if I wait a few minutes, there will be a gap, it might return 18456 next, for instance.
Refreshing does bring the correct record into view.
The autonumber fields do show up in Access design view as Primary Keys.
The Access front end is an .mdb file. We are using Access for Microsoft 365 MSO 64 bit.
As a general rule, this issue should not show up.
However, there are two cases to keep in mind.
First case:
Access when you START typing in a record, with a Access back end (BE), then the auto number is generated, and displayed instant, and this occurs EVEN before the record save.
And in fact if the record is not saved (user hits Esc key, or un-do from menu, or even ctrl-z). At that point, the record is not dirty and will not be saved. And of course this means gaps will and can appear in the autonumber.
WHEN using a linked table to sql server? You can start typing, and the record becomes dirty, but the AUTONUMBER will NOT display, and has NOT yet been generated. And thus your code cannot use the autonumber quite yet. The record has to be saved first before you can get/grab/use the autonumber.
Now for a form + sub form? Well, they work because access (for sql or access tables) ALWAYS does a record save of the main form when focus moves to the child form. So these setups should continue to work.
I note, and mention the above since SOME code that uses or requires use of the autonumber during a record add process MIGHT exist in your application. That code will have to be changed. Now to be fair, even in a fair large application, I tend to find few places where this occurs.
Often the simple solution is to modify the code, and simply force the record to be written, and then you have use of the autonumber.
You can do this:
if me.IsNewReocrd = True then
if me.dirty = true then me.Dirty = false
end if
' code here that needs the PK autonumber
lngNewID = me!id ' the autonumber is now generated and available for use.
The next common issue (and likely YOUR issue).
The table(s) in question have triggers. You have to modify the store procedures to re-select the PK id, and if you don't, then you see/find the symptoms you see. If the store procedure updates other tables, then it can work, but the last line of the store procedure will need to re-select the PK id.
So, in the last line of your store procedure that is attached to the table? you need to re-select the existing PK value.
eg:
SELECT #MyPK as ID
We have a system that stores tens of thousands of databases, each holding the data of a customer subscribing to our services. Each of those databases has their own user that is known and used by the custom thorough the distributed, thick, application front-ends we provide.
Now I would like to add a trigger to one of the tables in each one of those dbs, that should update one common db with some of the data from the inserted rows. Sort of a many-to-one db scenario if you will...
This new common db can be set up pretty much how we like - as far as users/permissions and so on, and the trigger we insert in the old dbs can be written pretty freely. But we can not change those customer dbs as far as users/permissions.
Currently I'm experimenting with using the guest user, it has been given writing access, on the common db (named "foo" in example below) like this, but it is not working (i think due to the guest user of the common db not being allowed to access the customer db table - named Bf in example below - it is triggered in?). It may also be that I'm using "execute as" where I should be using execute as login="? I'm having a hard time finding a comprehensible place that describes the difference.
This is the trigger we would like to get working, and inserted in every customer db:
ALTER TRIGGER [dbo].[trgAfterInsert] ON [dbo].[Bf]
WITH EXECUTE AS 'guest'
FOR INSERT
AS
begin
insert into [foo].[dbo].[baar]
(publicKey, fileId)
SELECT b.publicKey, a.autoId
FROM client_db_1.[dbo].[Integrationer] as b, inserted as a where
a.autoId in (select top 1 autoId from inserted order by autoId)
end
As you may guess, I'm not an experienced user of triggers or sql permissions/access work. But the info we want to collect is harmless, and it should take close to 0 time to execute, and in a secured, non-exposed, environment, so I'm very willing to read/learn if anyone has advice?
/Henry
Okay, just to clarify: I have a SQL Table (contains ID, School, Student ID, Name, Fee $, Fee Type, and Paid (as the columns)) that needs to be posted on a Grid that will uploaded on a website. The Grid shows everything correctly and shows what Fees need to be Paid. The Paid column has a bit data type for 1 or 0 (basically a checklist.) I am being asked to add two more columns: User and DateChanged. The reason why is to log which staff changed the "Paid" column. It would only capture the Username of the staff who changed it in the SQL Table and also the time. So to clarify even more, I need to create 2 columns: "User, DateChanged" and the columns would log when someone changed the "Paid" column.
For example: User:Bob checks the Paid column for X student on 5/2/17 at 10pm.
In the same row of X student's info, under User column Tom would appear there. Under DateChanged it would show 2017-05-02 10pm.
What steps would I take to make this possible.
I'm currently IT Intern and all this SQL stuff is new to me. Let me know if you need more clarification. FYI The two new columns: User, DateChanged will not be on the grid.
The way to do this as you've described is to use a trigger. I have an example of some code below but be warned as triggers can have unexpected side-effects, depending on how the database and app interface are set up.
If it is possible for you to change the application code that sends SQL queries to the database instead, that would be much safer than using a trigger. You can still add the new fields, you would just be relying on the app to keep them updated instead of doing it all in SQL.
Things to keep in mind about this code:
If any background processes or procedures make updates to the table, it will overwrite the timestamp and username automatically, because this is triggered on any update to the row(s) in question.
If the users don't have any direct access to SQL Server (in other words, the app is the only thing connecting to the database), then it is possible that the app will only be using one database login username for everyone, and in that case you will not be able to figure out which user made the update unless you can change the application code.
If anyone changes something by accident and then changes it back, it will overwrite your timestamp and make it look like the wrong person made the update.
Triggers can potentially bog down the database system if there are a very large number of rows and/or a high number of updates being made to the table constantly, because the trigger code will be executed every time an update is made to a row in the table.
But if you don't have access to change the application code, and you want to give triggers a try, here's some example code that should do what you are needing:
create trigger TG_Payments_Update on Payments
after update
as
begin
update Payments
set DateChanged = GetDate(), UserChanged = USER_NAME()
from Payments, inserted
where Payments.ID = inserted.ID
end
The web app already knows the current user working on the system, so your update would just include that user's ID and the current system time for when the action took place.
I would not rely on SQL Server triggers since that hides what's going on within the system. Plus, as others have said, they have side effects to deal with too.
I am working with code-igniter framework.How can i check a row deleted from a table in sql server.$this->db->affected->rows() always returning null, but the row is deleting successfully.
Delete/Drop is the most powerful command in programming. Once its executes all the database or table or row will be delete withing a very very short time.
Best practice is, add new column in your every table call online. To show your data filter with WHERE online=1. So once user press delete update it to online=0. So in view user understand that product has been removed from the site. But Admin knows what happen actually.
So finally you can sort out deleted product and showing products.
This what most of developers should do.
I want to comment this on your Question. Due to high text I posted this as answer
Basically what I'm trying to do is create a dynamic trigger where if a table from database 1 has a new record inputed. if it falls in the category of data that I need for database 2, it automatically populates the table in database 2 without me needed to manually update.
Right now I am going into the table in database 1 sorting for the category I need and copying the data I need into the table in database 2.
I tried to make this process easier by doing a select query for the columns that I need from database 1 to database 2, which works fine however it overwrites what I have already and I have to basically recreate everytime.
So after all that rambling I guess exactly what I need to know. Is there a way to create a trigger that if a new line item is inputed in database 1 with the tag matching the type of material I need to transfer to database 2. Also on top of that I only need to transfer 2 columns from database 1 to database 2.
I would try to post a sample code, however I have no idea where to start on this.
I suggest you look into Service Broker messaging. We use it quite a bit and it works quite well. You can send messages to the other database with the data that needs to be inserted and allow the second database to do all the work. This will alleviate the worries about the second database being offline or causing an error which rolls back into your trigger. If the second database is unavailable the messages will queue up in your database until it can send them. This isn't the easiest thing to set up but is a way to keep the two databases from being so closely tied together.
Service Broker
I am unclear about the logic in your selection but if you want to save a copy of what was just inserted into table1 into a table (table2) on another database, using a trigger, you can try this:
create trigger trig1 on dbo.table1
after insert as
insert into database2.dbo.table2 (col1,col2,col3) values (inserted.col1, inserted.col2)`
You could use an AFTER INSERT Trigger like this:
CREATE TRIGGER [FirstDB].[dbo].[YourTrigger]
ON [FirstDB].[dbo].[Table]
AFTER INSERT
AS
BEGIN
INSERT INTO [OtherDB].[dbo].[Table] SELECT (values...)
END
I recommend you consider non-trigger alternatives as well though. Cross-DB triggers could be risky (what if the other db is offline, etc.)