Task:
Automate database deployment (SSDT/dacpac deployment with CI/CD)
The database is a 3rd party database
It also includes our own customized tables/SP/Fn/Views in separate schemas
Should exclude 3rd party objects while deploying the database project(dacpac) to Production
Thanks to Ed Elliott for the AgileSqlClub.DeploymentFilterContributor. Used the dll to filter out the schema successfully.
Problem:
The 3rd party schema objects(Tables) are defined with unnamed constraints(default / primary key) when creating the tables. Example:
CREATE TABLE [3rdParty].[MainTable]
(ID INT IDENTITY(1,1) NOT NULL,
CreateDate DATETIME DEFAULT(GETDATE())) --There is no name given to default constraint
When I generate the script for deployment using sqlpackage.exe, I see following statements in the generated script.
Generated the script using:
"C:\Program Files\Microsoft SQL Server\150\DAC\bin\sqlpackage.exe" /action:script /sourcefile:C:\Users\User123\source\repos\DBProject\DBProject\bin\Debug\DBProject.dacpac /TargetConnectionString:"Data Source=MyServer; Initial Catalog=MSSQLDatabase; Trusted_Connection=True" /p:AdditionalDeploymentContributorPaths="C:\Program Files\Microsoft SQL Server\150\DAC\bin\AgileSqlClub.SqlPackageFilter.dll" /p:AdditionalDeploymentContributors=AgileSqlClub.DeploymentFilterContributor /p:AdditionalDeploymentContributorArguments="SqlPackageFilter=IgnoreSchema(3rdParty)" /outputpath:"c:\temp\script_AfterDLL.sql"
Script Output:
/*
Deployment script for MyDatabase
This code was generated by a tool.
Changes to this file may cause incorrect behavior and will be lost if
the code is regenerated.
*/
...
...
GO
PRINT N'Dropping unnamed constraint on [3rdParty].[MainTable]...';
GO
ALTER TABLE [3rdParty].[MainTable] DROP CONSTRAINT [DF__MainTabl__Crea__59463169];
...
...
...(towards the end of the script)
ALTER TABLE [3rdParty].[MainTable_2] WITH CHECK CHECK CONSTRAINT [fk_518_t_44_t_9];
I cannot alter 3rd party schema due to company restrictions
There are many lines of unnamed constraint and WITH CHECK CHECK constraints generated in the script.
Question:
How can I be able to remove the lines to DROP unnamed Constraint on 3rd party schemas? - Even though the dll excludes 3rd party schema, it still has these unnamed constraints scripted/deployed. Also, it is not Adding them back too !!
How can I be able to skip/remove generating WITH CHECK CHECK CONSTRAINT on 3rd party schemas
Any suggestions will be greatly helpful.
EDIT:
Also, I found another issue. The deployment will not succeed due to Rows were detected. The schema update is terminating because data loss might occur
Output:
/*
The column [3rdParty].[MainTable_1].[Col1] is being dropped, data loss could occur.
The column [3rdParty].[MainTable_1].[Col2] is being dropped, data loss could occur.
The column [3rdParty].[MainTable_1].[Col3] is being dropped, data loss could occur.
The column [3rdParty].[MainTable_1].[Col4] is being dropped, data loss could occur.
*/
IF EXISTS (select top 1 1 from [3rdParty].[MainTable_1])
RAISERROR (N'Rows were detected. The schema update is terminating because data loss might occur.', 16, 127) WITH NOWAIT
GO
Regarding the unnamed constraints, I couldn't find any solution using sqlpackage.exe.
But Redgate SQL Compare has an option to ignore them called IgnoreSystemNamedConstraintAndIndexNames that ignores system generated constraints and generates a much cleaner script.
For example when comparing 2 dacpacs:
SQLCompare /Scripts1:"\unpacked_dacpac_source_folder" /Scripts2:"\unpacked_dacpac_dest_folder" /options:IgnoreSystemNamedConstraintAndIndexNames /scriptFile:"script_result.sql"
You can find more info here:
Handling System-named Constraints in SQL Compare
I have a table with a trigger newly added to it.
When I try to run a batch file and upload the file using SqlBulkCopy it did not work.
After reading that adding SqlBulkCopyOptions.FireTriggers will fire the triggers that are set to the table, I added it.
But still I get the same error - 'Bulk copy failed. User does not have ALTER TABLE permission on table. ALTER TABLE permission is required on the target table of a bulk copy operation if the table has triggers or check constraints, but 'FIRE_TRIGGERS' or 'CHECK_CONSTRAINTS' bulk hints are not specified as options to the bulk copy command.
'
Any idea what is to be done? Any help would be appreciated
SQL Server requires ALTER TABLE permissions to bulk insert data without the FireTriggers and CheckConstraint options, as well as with the KeepIdentity option, to help safeguard against minimally-privileged users from violating data integrity rules enforced by SQL Server. See What permission do I need to use SqlBulkCopy in SQL Server 2008?.
The FireTriggers and CheckConstraint options of SqlBulkCopy are off by default to maximize performance. It is the application's responsibility to ensure data integrity that would otherwise be enforced by constraints and triggers when these options are off. Foreign key and check constraints become not trusted after data are bulk copied without these options and must be re-validated with ALTER TABLE...CHECK CONSTRAINT` afterwards before SQL Server will trust the constraints again.
No special permissions other than SELECT/INSERT to use SqlBulkCopy when FireTriggers and CheckConstraint options are on, and the KeepIdentity option is off.
I tried adding Checkconstraints as well but it says the same error.
using (var bulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.FireTriggers | SqlBulkCopyOptions.CheckConstraints))
Is there another way to do it?
Background
I'm looking into creating a simple web app, a part of which will display Images associated with Items. I've decided to look into using the FILETABLE feature of SQL Server which will allow binary image data to be uploaded into the exposed share directly. As such there is a use case to allow the deletion of files (rows in a FILETABLE) through Windows Explorer. This example replicates the issue, which stems from having a foreign key relationship to a FILETABLE.
Structure
Having already added an image using File Explorer to the FILETABLE with the path_locator of 0xFF5354649088A1EFEE8F747CD11030F80800170620:
CREATE TABLE [dbo].[Image] AS FILETABLE WITH (FileTable_Directory = 'Images');
GO
CREATE TABLE [dbo].[ImageLink] (
[id] INT NOT NULL IDENTITY(1, 1)
,[path_locator] HIERARCHYID NOT NULL
,FOREIGN KEY ([path_locator]) REFERENCES [dbo].[Image] ([path_locator])
);
GO
INSERT INTO [dbo].[ImageLink] ([path_locator]) VALUES (0xFF5354649088A1EFEE8F747CD11030F80800170620);
Issue
Upon deleting the file through File Explorer...
... the file disappears from the directory as Windows reports the deletion a success but the row is not removed from the FILETABLE.
However, when trying to delete through SQL Server, the familiar reference constraint conflict error is thrown:
DELETE FROM [dbo].[Image] WHERE [path_locator] = 0xFF5354649088A1EFEE8F747CD11030F80800170620;
Msg 547, Level 16, State 0, Line 69
The DELETE statement conflicted with the REFERENCE constraint "FK__ImageLink__path___5070F446". The conflict occurred in database "FileTableTest", table "dbo.ImageLink", column 'path_locator'.
I added an AFTER DELETE trigger to the FILETABLE with the intention of removed the referencing row, but this also does not get executed.
Question
How might I go about propagating the delete through the link table upon deletion through Windows Explorer?
Is there some kind of SQL Server/Windows API hook I can detect and execute DML code that handles the delete?
Update #1
From BOL, the following section kind of confirms the behaviour, although doesn't offer any further information.
Transactional Semantics
When you access the files in a FileTable by using file I/O APIs, these operations are not associated with any user transactions, and have the following additional characteristics:
Since non-transacted access to FILESTREAM data in a FileTable is not associated with any transaction, it does not have any specific isolation semantics. However SQL Server may use internal transactions to enforce locking or concurrency semantics on the FileTable data. Any internal transactions of this type are done with read-committed isolation.
The problem is the foreign key.
Use 'ON CASCADE DELETE' in your foreign key, so when you delete through File Explorer the associated ImageLink is deleted too.
It looks like ,there is problem with the foreign key. As there is foreign key attached to that table so you cannot simply delete the row as foreign key contraint fails.
So first disable foreign key check in sql by :
SET FOREIGN_KEY_CHECKS = 1;
and then try deleting this & yes don't forget to set foreign key check to 0 by:
SET FOREIGN_KEY_CHECKS = 0;
after deleting the row.
Running the following query results in all constraints in our client's database. However, several rows in the result set don't seem to have a parent, i.e. parent_object_id = 0 and OBJECT_NAME(parent_object_id) returns NULL.
SELECT name, type_desc, OBJECT_NAME(parent_object_id), parent_object_id
FROM sys.objects
WHERE is_ms_shipped = 0
AND type_desc LIKE '%_CONSTRAINT'
Does this mean there are orphan constraints in the database? If so, how do I remove these?
From their names, I can see they are leftovers from before a large amount changes were made to the structure.
Using sp_helptext I can see they were created using a CREATE DEFAULT statement, e.g.:
CREATE DEFAULT dbo.MyDefault AS 2
This means they are simply unbound defaults, which can be bound using the sp_binddefault according to MSDN, and removed with a simple DROP DEFAULT statement:
DROP DEFAULT dbo.MyDefault
I wasn't aware of this syntax, which apparently will be removed in a future version of SQL Server, according to the aforementioned MSDN article.
I've been asked to create a simple DataGrid-style application to edit a single table of a database, and that's easy enough. But part of the request is to create an audit trail of changes made, who made them, and the date/time.
How might you solve this kind of thing?
(I'll be using C# in VS2008, ADO.NET connected to SQL Server 2005, WPF and Xceed's DataGrid, if it makes any difference.)
There are two common ways of creating audit trails.
Code your data access layer.
In the database itself using triggers.
There are advantages and disadvantages to both. Some people prefer one over the other. It's often down to the type of app and the type of database use you can expect.
If you do it in your DA layer it's pretty much up to you. You just need to add code to every method that saves to the database to also save a log of the changes. This auditing code could be in your DA layer code, or even in your stored procs in your database if you are using stored procs for everything. Essentially the premise is the same, any time you make a change to the database, log that change.
If you want to go down the triggers route, you can write custom triggers for each table, or fashion a more generic trigger that works the same on lots of tables. Check out this article on audit triggers. This works by firing of triggers whenever a change is made, and the triggers log the changes. Remember that if you want to audit SELECT statements, you can't use triggers, you'll have to do that with in code/stored proc auditing. It's also worth remember that depending on your database, triggers may not fire in all circumstances. For example, most databases don't fire triggers during TRUNCATE statements. Check that your triggers get fired in any case that you need auditing.
Alternately, you could also take a look at using the service broker to do async auditing on a dedicated machine. This is more complex and takes a bit of configuring to set up.
Which ever way you do it you need to decide on the format the audit log will take. Normally you would save this log in your database, but you could just save it in a log file or whatever suits your requirements. You could use a single audit table that logs all changes, or you could have an audit table per main table being audited. For large scale implementations you could even consider putting the audit tables in a totally separate database. If your logging into a table, it's common to have a "change type" field which indicates if the audited change was an insert, update or delete style of change, along with the changed data, user who made the change and the date/time the change was made. Don't forget to include the old and new data for update style changes.
Ditto use triggers.
Anyone considering soft deletion should have a read of Richard Dingwall's The trouble with soft delete.
Most universal method would be to create another table for storing versions of record from the first table. Then, you can remove all the data from main table. Suppose you need versioning of a table Person(PersonId, Name, Surname):
CREATE TABLE Person
(
PersonId INT, // PK
CurrentPersonVersion INT // FK
);
CREATE TABLE PersonVersion
(
PersonVersionId INT, // PK
PersonID // FK
Name VARCHAR, // actual data
Surname VARCHAR, // actual data
ChangeDate // logging data
ChangeAuthor // logging data
)
Now any change requires inserting new PersonVersion and updating the CurrentPersonVersionID.
The best way to do this is set up triggers in the database that write to audit tables.
Solution 1: SQL Server Change Data Capture
https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/enable-and-disable-change-data-capture-sql-server?view=sql-server-2017
First you need to enable change data capture on your database
USE AdventureWorks2012
GO
EXEC sys.sp_cdc_enable_db
GO
Then you can query the changes using fn_cdc_get_all_changes_ or fn_cdc_get_net_changes_.
-- ========
-- Enumerate All Changes for Valid Range Template
-- ========
USE AdventureWorks2012;
GO
DECLARE #from_lsn binary(10), #to_lsn binary(10);
SET #from_lsn = sys.fn_cdc_get_min_lsn('HR_Department');
SET #to_lsn = sys.fn_cdc_get_max_lsn();
SELECT * FROM cdc.fn_cdc_get_all_changes_HR_Department
(#from_lsn, #to_lsn, N'all');
Solution 2: SQL Server Database Auditing
Source : https://www.dbaservices.com.au/how-to-configure-sql-server-auditing/
ENABLE DATABASE AUDITING
Database auditing requires that a server audit (although not necessarily server audit specification) to be in place. The DB auditing however is created within the user database that is to be audited, rather than within the master database where the server audit gets created. Database audit specifications can be found within the DB itself under Security –> Database Audit Specifications.
To create a database audit, you’ll need to first USE the database (to select it), then the following provides an example syntax for auditing SELECT, UPDATE and DELETE operations for specific tables within that database;
USE UserDatabase
GO
CREATE DATABASE AUDIT SPECIFICATION [User_Database_Audit_Specification]
FOR SERVER AUDIT [SQL_Server_Audit]
ADD (SELECT , UPDATE , DELETE ON UserDatabase.dbo.Customer_DeliveryAddress BY dbo )
,ADD (SELECT , UPDATE , DELETE ON UserDatabase.dbo.DimCustomer_Email BY dbo )
,ADD (SELECT , UPDATE , DELETE ON UserDatabase.dbo.DimCustomer_Phone BY dbo )
WITH (STATE = ON) ;
GO
The SELECT, UPDATE and DELETE operations aren’t the only things you can add to the audit specification though…
+------------+-------------------------------------------------------------------+
| Action | Description |
+------------+-------------------------------------------------------------------+
| SELECT | This event is raised whenever a SELECT is issued. |
| UPDATE | This event is raised whenever an UPDATE is issued. |
| INSERT | This event is raised whenever an INSERT is issued. |
| DELETE | This event is raised whenever a DELETE is issued. |
| EXECUTE | This event is raised whenever an EXECUTE is issued. |
| RECEIVE | This event is raised whenever a RECEIVE is issued. |
| REFERENCES | This event is raised whenever a REFERENCES permission is checked. |
+------------+-------------------------------------------------------------------+
The full list of database events you can log is available here:
https://learn.microsoft.com/en-us/sql/relational-databases/event-classes/security-audit-event-category-sql-server-profiler?view=sql-server-2017
I was recently faced with a requirement to audit some tables and I opted to use triggers. Like others, I only wanted to see entries in the audit table for those fields that had actually changed, however, when updating the tables, the application was updating all the fields in row whether they'd changed or not, therefore, checking whether the fields had been updated or not availed me nothing - they all had!
What I wanted, therefore, was a method of checking the actual value in each field to see if it had changed or not and only writing it to the audit table if it had. Having been unable to find any solution to this conundrum anywhere, I came up with my own, as follows:
CREATE TRIGGER [dbo].[MyTable_CREATE_AUDIT]
ON [dbo].[MyTable]
AFTER UPDATE
AS
INSERT INTO MyTable_Audit
(ItemID,LastModifiedBy,LastModifiedDate,field1,field2,field3,
field4,field5,AuditDate)
SELECT i.ItemID,i.LastModifiedBy,i.LastModifiedDate,
field1 =
CASE i.field1
WHEN d.field1 THEN NULL
ELSE i.field1
END,
field2 =
CASE i.field2
WHEN d.field2 THEN NULL
ELSE i.field2
END,
field3 =
CASE i.field3
WHEN d.field3 THEN NULL
ELSE i.field3
END,
field4 =
CASE i.field4
WHEN d.field4 THEN NULL
ELSE i.field4
END,
field5 =
CASE i.field5
WHEN d.field5 THEN NULL
ELSE i.field5
END,
GETDATE()
FROM inserted i
INNER JOIN deleted d
ON i.ItemID = d.ItemID
As you can see, I'm comparing the values of each field in the deleted and inserted tables and only writing the field value from the inserted table to the audit table if they differ, otherwise I just write NULL.
It certainly works for me. Can anyone see any issues with this approach? My team own both the application and the database so possible curved balls like schema changes are covered off.
The other way of doing this apart from triggers is this,
Have four columns, UpdFlag, DelFlag, EffectiveDate and TerminatedDate for each table you want to do an audit trail on.
code your sproc's in such a way that when you do an update, to pass in the all of the row's column data into the sproc, update the row by setting the TerminatedDate to the date that was updated, and mark the UpdFlag and to put in the datetime into the column
Then create a new row with the new data (which is really updated). and put in a new date now for the EffectiveDate and the TerminatedDate set to the max date.
Likewise if you want to do a deletion of the row, simply update the row by marking the DelFlag as set, the TerminatedDate with the datetime now. You are in effect doing a soft delete and not an actual sql's Delete.
In that way, when you want to audit the data, and to show a trail of the changes, you can simply filter the rows for those that have the UpdFlag set, or between EffectiveDate and TerminatedDate. Likewise for those that were deleted, you filter for those that have the DelFlag set or between EffectiveDate and TerminatedDate. For the current rows, filter the rows that have both flags set off. The advantage is you don't have to create another table for the audit when the trigger is used!
I'd go triggers route, by creating table with similar structure to updated one, with additional columns for tracking changes like ModifiedAt etc. And then adding on update trigger that will insert changes to that table.
I find it easier to maintain than have everything in the application code. Ofcourse many people tend to forget about triggers when it comes to questions like 'wtf this table is changing' ;) Cheers.