I have a stored procedure that relies on a query to a linked server.
This stored procedure is roughly structured as follows:
-- Create local table var to stop query from needing round trips to linked server
DECLARE #duplicates TABLE (eid NVARCHAR(6))
INSERT INTO #duplicates(eid)
SELECT eid FROM [linked_server].[linked_database].[dbo].[linked_table]
WHERE es = 'String'
-- Update on my server using data from linked server
UPDATE [my_server].[my_database].[dbo].[my_table]
-- Many things, including
[status] = CASE
WHEN
eid IN (
SELECT eid FROM #duplicates
)
THEN 'String'
ELSE es
END
FROM [my_server].[another_database].[dbo].[view]
-- This view obscures sensitive information and shows only the data that I have permission to see
-- Many other things
The query itself is much more complex, but the key idea is building this temporary table from a linked server (because it takes the query 5 minutes to run if I don't, versus 3 seconds if I do).
I've recently had an issue where I ended up with updates to my table that failed to get checked against the linked server for duplicate information.
The logical chain of events is this:
Get all of the data from the original view
The original view contains maybe 3000 records, of which maybe 30 are
duplicates of the entity in question, but with 1 field having a
different value.
I then have to grab data from a different server to know which of
the duplicates is the correct one.
When the stored procedure runs, it updates each record.
ERROR STEP - when the stored procedure hits a duplicate record, it
updates my_table again - so es gets changed multiple times in a row.
The temp table was added after the fact when we realized incorrect es values were being introduced to my_table.
'my_database` does not contain the data needed to determine which is the correct tuple, hence the requirement for the linked server.
As far as I can tell, we had a temporary network interruption or a connection timeout that stopped my_server from getting the response back from linked_server, and it just passed an empty table to the rest of the procedure.
So, my question is - how can I guard against this happening?
I can't just check if the table is empty, because it could legitimately be empty. I need to definitively know if that initial SELECT from linked_server failed, if it timed out, or if it intentionally returned nothing.
without knowing the definition of the table you're querying you could get into an issue where your data is to long and you get a truncation error on your table.
Better make sure and substring it...
DECLARE #duplicates TABLE (eid NVARCHAR(6))
INSERT INTO #duplicates(eid)
SELECT SUBSTRING(eid,1,6) FROM [linked_server].[linked_database].[dbo].[linked_table]
WHERE es = 'String'
-- Update on my server using data from linked server
UPDATE [my_server].[my_database].[dbo].[my_table]
-- Many things, including
[status] = CASE
WHEN
eid IN (
SELECT eid FROM #duplicates
)
THEN 'String'
ELSE es
END
FROM [my_server].[another_database].[dbo].[view]
I had a similar problem where I needed to move data between servers, could not use a network connection so I ended up doing BCP out and BCP in. This is fast, clean and takes away the complexity of user authentication, drivers, trust domains. also it's repeatable and can be used for incremental loading.
I recently had a horrible blunder.
While attempting to fix an issue we were having with our Exact Synergy system I was attempting to replace the data in two columns for one account with NULL, instead I replaced those two columns in ALL accounts with NULL. Completely restoring from a backup is not an option so now I am left trying to figure out how to replace the missing data.
I have made a full restore of a recent backup for this database to a test database and have confirmed that the data I need is there. I am trying to figure out how to properly write a query that will replace the data in the two columns.
Since this is a backup of the same database, the tables and columns are all identically named.
The databases are Synergy and Synergy_TESTDB
The owner of the tables is dbo
The table is called Addresses
The columns are called textfield1 and textfield2
What I would like to do is take the data in textfield1 and textfield2 from the backup database and use it to populate the empty, or NULL, columns in the live database.
I am extremely new to SQL, and would appreciate any help.
This is obviously untested. I take no responsibility for you using this code.
That said I'd like to try and help you.
The main point is the 3 part database.table naming: I'm assuming you restored backup to same server. I'm also assuming you have a primary key on the table? And that Synergy_TESTDB is the restored database:
update target
set target.textfield1 = source.textfield1
from Synergy.dbo.Addresses target
join Synergy_TESTDB.dbo.Addresses source on target.PrimaryKeyCol = source.PrimaryKeyCol
where target.textfield1 IS NULL
update target
set target.textfield2 = source.textfield2
from Synergy.dbo.Addresses target
join Synergy_TESTDB.dbo.Addresses source on target.PrimaryKeyCol = source.PrimaryKeyCol
where target.textfield2 IS NULL
(Sure it could be done in a single update, but I'm trying to keep it simple.)
I strongly suggest you try in another test database first.
A good habit to get in to is to use a pattern like this:
BEGIN TRANSACTION
-- Perform updates
-- Examine the results: select * from dbo.Blah ...
-- If results are wrong, we just rollback anyway
ROLLBACK
-- If results are what you want, uncomment the COMMIT and comment out the ROLLBACK
-- COMMIT TRANS
I have 200K+ rows data in xls and as per requirement i need to update database tables (2 tables) using xls data.
I know the process to copy data from xls to SQL server table however i am struggling with approach to update database tables.
I could not think of any other approach than writing a cursor and i dont want to go with cursor approach as updating
200k+ data using cursor may eat up transaction log and will take lot of time to finish the update.
Can someone help me with what else could be done to accomplish this.
Use the following techniques.
1 - Import the data into a staging table. Use the import / export tool is one way to do the task The target table should be in a throw away or staging database.
http://technet.microsoft.com/en-us/library/ms141209.aspx
2 - Make sure that the data types between the EXCEL data and TABLE data are the same.
3 - Make sure the existing target [TRG_TBL] TABLE has a primary key. Make sure the EXCEL data loaded into a [SRC_TBL] table has the same key. You can add a non-clustered index to speed up the JOIN in the UPDATE statement.
4 - Add a [FLAG] column as INT NULL to the [TRG_TABLE] with an ALTER TABLE command.
5 - Make sure a full backup is done before and after the large UPDATE. You can also use a DATABASE SNAPSHOT. The key point is to have a roll back plan in place if needed.
-- Select correct db
USE [TRG_DB]
GO
-- Set to simple mode
ALTER DATABASE [TRG_DB] SET RECOVERY SIMPLE;
GO
-- Update in batches
DECLARE #VAR_ROWS INT = 1;
WHILE (#VAR_ROWS > 0)
BEGIN
-- Update fields and flag on join
UPDATE TOP (10000) T
SET
T.FLD1 = S.FLD1,
-- ... Etc
T.FLAG = 1
FROM [TRG_TABLE] T JOIN [SRC_TABLE] S ON T.ID = S.ID
WHERE T.[FLAG] IS NULL
-- How many rows updated
SET #VAR_ROWS = ##ROWCOUNT;
-- WAL -> flush log entries to data file
CHECKPOINT;
END
-- Set to full mode
ALTER DATABASE [MATH] SET RECOVERY FULL;
GO
In summary, I gave you all the tools to do the job. Just modify them for your particular occurrence.
PS: Here is working code from my blog on large deletes. Same logic applies.
http://craftydba.com/?p=3079
PPS: I did not check the sample code for syntax. That is left up for you.
I've been asked to create a simple DataGrid-style application to edit a single table of a database, and that's easy enough. But part of the request is to create an audit trail of changes made, who made them, and the date/time.
How might you solve this kind of thing?
(I'll be using C# in VS2008, ADO.NET connected to SQL Server 2005, WPF and Xceed's DataGrid, if it makes any difference.)
There are two common ways of creating audit trails.
Code your data access layer.
In the database itself using triggers.
There are advantages and disadvantages to both. Some people prefer one over the other. It's often down to the type of app and the type of database use you can expect.
If you do it in your DA layer it's pretty much up to you. You just need to add code to every method that saves to the database to also save a log of the changes. This auditing code could be in your DA layer code, or even in your stored procs in your database if you are using stored procs for everything. Essentially the premise is the same, any time you make a change to the database, log that change.
If you want to go down the triggers route, you can write custom triggers for each table, or fashion a more generic trigger that works the same on lots of tables. Check out this article on audit triggers. This works by firing of triggers whenever a change is made, and the triggers log the changes. Remember that if you want to audit SELECT statements, you can't use triggers, you'll have to do that with in code/stored proc auditing. It's also worth remember that depending on your database, triggers may not fire in all circumstances. For example, most databases don't fire triggers during TRUNCATE statements. Check that your triggers get fired in any case that you need auditing.
Alternately, you could also take a look at using the service broker to do async auditing on a dedicated machine. This is more complex and takes a bit of configuring to set up.
Which ever way you do it you need to decide on the format the audit log will take. Normally you would save this log in your database, but you could just save it in a log file or whatever suits your requirements. You could use a single audit table that logs all changes, or you could have an audit table per main table being audited. For large scale implementations you could even consider putting the audit tables in a totally separate database. If your logging into a table, it's common to have a "change type" field which indicates if the audited change was an insert, update or delete style of change, along with the changed data, user who made the change and the date/time the change was made. Don't forget to include the old and new data for update style changes.
Ditto use triggers.
Anyone considering soft deletion should have a read of Richard Dingwall's The trouble with soft delete.
Most universal method would be to create another table for storing versions of record from the first table. Then, you can remove all the data from main table. Suppose you need versioning of a table Person(PersonId, Name, Surname):
CREATE TABLE Person
(
PersonId INT, // PK
CurrentPersonVersion INT // FK
);
CREATE TABLE PersonVersion
(
PersonVersionId INT, // PK
PersonID // FK
Name VARCHAR, // actual data
Surname VARCHAR, // actual data
ChangeDate // logging data
ChangeAuthor // logging data
)
Now any change requires inserting new PersonVersion and updating the CurrentPersonVersionID.
The best way to do this is set up triggers in the database that write to audit tables.
Solution 1: SQL Server Change Data Capture
https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/enable-and-disable-change-data-capture-sql-server?view=sql-server-2017
First you need to enable change data capture on your database
USE AdventureWorks2012
GO
EXEC sys.sp_cdc_enable_db
GO
Then you can query the changes using fn_cdc_get_all_changes_ or fn_cdc_get_net_changes_.
-- ========
-- Enumerate All Changes for Valid Range Template
-- ========
USE AdventureWorks2012;
GO
DECLARE #from_lsn binary(10), #to_lsn binary(10);
SET #from_lsn = sys.fn_cdc_get_min_lsn('HR_Department');
SET #to_lsn = sys.fn_cdc_get_max_lsn();
SELECT * FROM cdc.fn_cdc_get_all_changes_HR_Department
(#from_lsn, #to_lsn, N'all');
Solution 2: SQL Server Database Auditing
Source : https://www.dbaservices.com.au/how-to-configure-sql-server-auditing/
ENABLE DATABASE AUDITING
Database auditing requires that a server audit (although not necessarily server audit specification) to be in place. The DB auditing however is created within the user database that is to be audited, rather than within the master database where the server audit gets created. Database audit specifications can be found within the DB itself under Security –> Database Audit Specifications.
To create a database audit, you’ll need to first USE the database (to select it), then the following provides an example syntax for auditing SELECT, UPDATE and DELETE operations for specific tables within that database;
USE UserDatabase
GO
CREATE DATABASE AUDIT SPECIFICATION [User_Database_Audit_Specification]
FOR SERVER AUDIT [SQL_Server_Audit]
ADD (SELECT , UPDATE , DELETE ON UserDatabase.dbo.Customer_DeliveryAddress BY dbo )
,ADD (SELECT , UPDATE , DELETE ON UserDatabase.dbo.DimCustomer_Email BY dbo )
,ADD (SELECT , UPDATE , DELETE ON UserDatabase.dbo.DimCustomer_Phone BY dbo )
WITH (STATE = ON) ;
GO
The SELECT, UPDATE and DELETE operations aren’t the only things you can add to the audit specification though…
+------------+-------------------------------------------------------------------+
| Action | Description |
+------------+-------------------------------------------------------------------+
| SELECT | This event is raised whenever a SELECT is issued. |
| UPDATE | This event is raised whenever an UPDATE is issued. |
| INSERT | This event is raised whenever an INSERT is issued. |
| DELETE | This event is raised whenever a DELETE is issued. |
| EXECUTE | This event is raised whenever an EXECUTE is issued. |
| RECEIVE | This event is raised whenever a RECEIVE is issued. |
| REFERENCES | This event is raised whenever a REFERENCES permission is checked. |
+------------+-------------------------------------------------------------------+
The full list of database events you can log is available here:
https://learn.microsoft.com/en-us/sql/relational-databases/event-classes/security-audit-event-category-sql-server-profiler?view=sql-server-2017
I was recently faced with a requirement to audit some tables and I opted to use triggers. Like others, I only wanted to see entries in the audit table for those fields that had actually changed, however, when updating the tables, the application was updating all the fields in row whether they'd changed or not, therefore, checking whether the fields had been updated or not availed me nothing - they all had!
What I wanted, therefore, was a method of checking the actual value in each field to see if it had changed or not and only writing it to the audit table if it had. Having been unable to find any solution to this conundrum anywhere, I came up with my own, as follows:
CREATE TRIGGER [dbo].[MyTable_CREATE_AUDIT]
ON [dbo].[MyTable]
AFTER UPDATE
AS
INSERT INTO MyTable_Audit
(ItemID,LastModifiedBy,LastModifiedDate,field1,field2,field3,
field4,field5,AuditDate)
SELECT i.ItemID,i.LastModifiedBy,i.LastModifiedDate,
field1 =
CASE i.field1
WHEN d.field1 THEN NULL
ELSE i.field1
END,
field2 =
CASE i.field2
WHEN d.field2 THEN NULL
ELSE i.field2
END,
field3 =
CASE i.field3
WHEN d.field3 THEN NULL
ELSE i.field3
END,
field4 =
CASE i.field4
WHEN d.field4 THEN NULL
ELSE i.field4
END,
field5 =
CASE i.field5
WHEN d.field5 THEN NULL
ELSE i.field5
END,
GETDATE()
FROM inserted i
INNER JOIN deleted d
ON i.ItemID = d.ItemID
As you can see, I'm comparing the values of each field in the deleted and inserted tables and only writing the field value from the inserted table to the audit table if they differ, otherwise I just write NULL.
It certainly works for me. Can anyone see any issues with this approach? My team own both the application and the database so possible curved balls like schema changes are covered off.
The other way of doing this apart from triggers is this,
Have four columns, UpdFlag, DelFlag, EffectiveDate and TerminatedDate for each table you want to do an audit trail on.
code your sproc's in such a way that when you do an update, to pass in the all of the row's column data into the sproc, update the row by setting the TerminatedDate to the date that was updated, and mark the UpdFlag and to put in the datetime into the column
Then create a new row with the new data (which is really updated). and put in a new date now for the EffectiveDate and the TerminatedDate set to the max date.
Likewise if you want to do a deletion of the row, simply update the row by marking the DelFlag as set, the TerminatedDate with the datetime now. You are in effect doing a soft delete and not an actual sql's Delete.
In that way, when you want to audit the data, and to show a trail of the changes, you can simply filter the rows for those that have the UpdFlag set, or between EffectiveDate and TerminatedDate. Likewise for those that were deleted, you filter for those that have the DelFlag set or between EffectiveDate and TerminatedDate. For the current rows, filter the rows that have both flags set off. The advantage is you don't have to create another table for the audit when the trigger is used!
I'd go triggers route, by creating table with similar structure to updated one, with additional columns for tracking changes like ModifiedAt etc. And then adding on update trigger that will insert changes to that table.
I find it easier to maintain than have everything in the application code. Ofcourse many people tend to forget about triggers when it comes to questions like 'wtf this table is changing' ;) Cheers.
Some how some records in my table are getting updated with value of xyz in a certain column. Out of hundred of stored procedures, functions, triggers, how can I determine which code is doing this action. Is there a way to search through the database some how through each and every script of the code?
Please help.
One approach is to check syscomments
Contains entries for each view, rule,
default, trigger, CHECK constraint,
DEFAULT constraint, and stored
procedure within the database. The
text column contains the original SQL
definition statements..
e.g. select text from syscomments
If you are having trouble finding that literal string, the values could be coming from a table, or they could be being concatenated within a routine.
Try this
Select text from syscomments
where CharIndex('x', text) > 0
and CharIndex('y', text) > 0
and CharIndex('z', text) > 0
That might help you either find the right routine, or further indicate that the values are coming from a table.
This is going to be nearly impossible to do in SQL Server 2000 because the update might very well be from a variable that has that value or a join to another table that has that value and not hard-coded into the stored proc, trigger etc. The update could also be coming from a DTS package, a job, a piece of dynamic code run by the app or even from query analyzer, so the code itself may not be recorded inthe datbase anywhere.
Perhaps a better approach might be to create an audit table for the table in question and have it record the user and the code from the spid that generated the change as well as the old and new values. You'll have to wait until it happens again, but then you would know exactly what changed the value and what value to put it back to if need be.
Alternatively you could run profiler on the system until it happens but profiler tends to hurt performance and is not usually a good idea to run on a production system. If it is happening very often, it might be an acceptable alternative.
Here's a hint as to how you might get some of the info you want for the eventual trigger code you write:
create table #temp (eventtype nvarchar (1000), parameters int, eventinfo nvarchar (4000), myspid int)
declare #myspid int
select #myspid =##spid
insert #temp (eventtype,parameters, eventinfo)
exec ('dbcc inputbuffer (##spid)')
update #temp
set myspid = #myspid
select hostname, program_name, eventinfo
from #temp t
join sysprocesses s on t.myspid = s.spid
WHERE spid = #myspid
You might use sql-profiler to trac the update of a given table / column.