I need to audit data in all tables in database. I use SQL Server 2016. I enabled Change Data Capture for all tables.
How to get changes from all tables chronologically?
Basically the Change Data Capture creates system tables in [cdc] schema to capture change events for each table. something like cdc.[TableSchemaName]_[TableName]_CT, this table will have all the changes done to your actual table in the chronological order. It is basically the data read from DB's transaction log file.
Another point - you need to query Maximum Lsn for the database at any point of time and also the minimum LSN for the table for which you want to read change data. The records between min and max LSN should give you the total changes for a table.
Reference link below:
https://learn.microsoft.com/en-us/sql/relational-databases/system-functions/cdc-fn-cdc-get-all-changes-capture-instance-transact-sql
Related
I upload an Excel file using BCP. (Truncate the current table in DB every day and BCP in from the excel file to repopulate table). It is important for me to keep a log of all the changes made to the rows (could be row additions or changes in columns of current rows). The idea is to keep a log of all the changes made.
I have read a few articles online, where we can create a log table and trigger (have no idea how to do it). A table of logs that has columns like
Date | Field | Old Value | New Value.
Firstly, how to do this?
Secondly, whats a smarter way to not log truncating of table and just the actual changes. I'm thinking of creating a temp table (tbl_Excefile_Temp) where I will import the file and then UPDATE the current table (tbl_Excefile) from the tbl_Excefile_Temp This way all the changes made in the current table will get logged automatically in the logs table.
I know its a big use case, could you please guide.
If you are using SQL server 2016 or higher I would advise you to look into temporal tables. If you stop truncating and use a merge statement you have a very easy way of keeping a log. Whenever you make a change SQL server will write to old values away and add the datetimes when the old row was valid.
With temporal tables you can query your table as they were at a specific datetime. In regular use there is no difference with a non-temporal table.
If CDC has been setup on a table, only tracking Columns A,D,E instead of the entire table, is it possible to add a column Z to the source table, then add column Z to the list of tracked columns for CDC? Is it possible to do this without losing CDC data?
I've looked around and the only examples I find are for tracking the entire table and not for cherry picking columns. I'm hoping for a way to update a table schema and not lose CDC history, without doing the whole copy CDC to temp table then back to CDC process.
Do you always have to create a new instance of CDC for schema changes?
SQL Server 2012
The reason that CDC allows for two capture instances on a given table is exactly this reason. The idea is this:
You have a running instance tracking columns A, B, and C
You now want to start tracking column D
So you set up a second capture instance to track A, B, C, and D.
You then process everything from the original capture instance and note the last LSN that you process
Using the fn_cdc_increment_lsn() function, you set the start point to start processing the new capture instance
Once you're up and running on the new instance, you can drop the old one
Of course, if you're using CDC as history of all changes ever on the table… that's not what it was designed for. CDC was meant to be an ETL facilitator. That is, capturing changes to data so that they could be consumed and then ultimately discarded from the CDC system. If you're using it for a historical system of record, I'd suggest setting up a "dumb" ETL meaning a straight copy out of the CDC tables into a user table. Once you do that, you can implement the above.
I have a set of 16 SQL Server tables, which get updated constantly from a Web UI. I need to track every change happening to these tables and call a separate system every 10 minutes sending each inserted or updated row through a Windows service.
I can duplicate the schema and create another set of 16 similar tables to track changes in the original set. There will be triggers that insert a new row into tracking tables (plus ins/upd flag, timestamp etc fields) every time a corresponding source table is modified.
I am wondering is there any better way I can do this using 1 (or few) common tables that can hold data from multiple tables? Something that does not force me to maintain a duplicate set of tracking tables?
If you have Enterprise Edition, Change Data Capture is an option which is available to you.
Other editions have Change Tracking, which doesn't track history but can get you the net changes.
Comparison: https://technet.microsoft.com/en-us/library/Cc280519(v=SQL.105).aspx
If we receive an update statement that does not check if the value has changed in the where clause, what are the different ways to ignore that update inside a trigger?
I know we can do a comparison of each individual field (handling the ISNULL side as well), but where it's a table that has 50+ fields, is there a faster/easier way to do it?
Note:I want to save each and every event in logs for updated fields.for example i have 50 fields and one of the field is updated(for single row not for entire table),then i want to save only that updated field old value and new value in logs.
Thanks in Advance, RAHUL
If this is more about logging changes to tables, a simpler solution may be to use Change Data Capture (CDC) tables.
Every time a change is made to a table, a row is written to your CDC table. Then you could write a query over the CDC table to bring you back just the data that has changed.
More information is on CDC tables is availble here:
http://msdn.microsoft.com/en-us/library/bb522489(v=sql.105).aspx
Scenario:
I have Database1 (PostgreSQL). For this i) When a record is deleted, the status col. for that record is changed to inactive. ii) When a record is updated, the current record is rendered INACTIVE and a new record is inserted. iii) Insertion happens as usual. There is a timestamp col for each record for all the tables in the database.
I have another database2 (SQLite) which is synced with Database1, and follows the same property of Database1
Database1 gets changed regularly and I would get the CSV files for all the tables. The CSV would include all the data, including new insertions, and updations.
Requirement:
I need to make the data in Database1 consistent with the new CSV.
i) For the records that are not in the CSV, but are there in Database1 (DELETED RECORDS) - These records I have to set the status as inactive.
ii) For the records that are there in the CSV but not there in the Database1 (INSERTED RECORDS) - I need these records to be inserted.
iii) For the records that are updated in the CSVs I need to set status as inactive and insert new records.
Kindly help me with the logical implementation of these!!!
Thanks
Jayakrishnan
I assume you're looking to build software to achieve what you want, not looking for an off-the-shelf solution.
What environments are you able to develop in? C? PHP? Java? C#?
Lots of options in many environments that can all read/write from CSV/SQLite/PostgreSQL.
you could use an ON DELETE trigger to override existing delete behavior.
This strikes me as dangerous however. Someone is going to rely on this and then when the trigger isn't there, you will have actual deletions occur... It's better to encapsulate this behind a view or something and put a trigger on that. Or go through a stored procedure or something.