Task:
Automate database deployment (SSDT/dacpac deployment with CI/CD)
The database is a 3rd party database
It also includes our own customized tables/SP/Fn/Views in separate schemas
Should exclude 3rd party objects while deploying the database project(dacpac) to Production
Thanks to Ed Elliott for the AgileSqlClub.DeploymentFilterContributor. Used the dll to filter out the schema successfully.
Problem:
The 3rd party schema objects(Tables) are defined with unnamed constraints(default / primary key) when creating the tables. Example:
CREATE TABLE [3rdParty].[MainTable]
(ID INT IDENTITY(1,1) NOT NULL,
CreateDate DATETIME DEFAULT(GETDATE())) --There is no name given to default constraint
When I generate the script for deployment using sqlpackage.exe, I see following statements in the generated script.
Generated the script using:
"C:\Program Files\Microsoft SQL Server\150\DAC\bin\sqlpackage.exe" /action:script /sourcefile:C:\Users\User123\source\repos\DBProject\DBProject\bin\Debug\DBProject.dacpac /TargetConnectionString:"Data Source=MyServer; Initial Catalog=MSSQLDatabase; Trusted_Connection=True" /p:AdditionalDeploymentContributorPaths="C:\Program Files\Microsoft SQL Server\150\DAC\bin\AgileSqlClub.SqlPackageFilter.dll" /p:AdditionalDeploymentContributors=AgileSqlClub.DeploymentFilterContributor /p:AdditionalDeploymentContributorArguments="SqlPackageFilter=IgnoreSchema(3rdParty)" /outputpath:"c:\temp\script_AfterDLL.sql"
Script Output:
/*
Deployment script for MyDatabase
This code was generated by a tool.
Changes to this file may cause incorrect behavior and will be lost if
the code is regenerated.
*/
...
...
GO
PRINT N'Dropping unnamed constraint on [3rdParty].[MainTable]...';
GO
ALTER TABLE [3rdParty].[MainTable] DROP CONSTRAINT [DF__MainTabl__Crea__59463169];
...
...
...(towards the end of the script)
ALTER TABLE [3rdParty].[MainTable_2] WITH CHECK CHECK CONSTRAINT [fk_518_t_44_t_9];
I cannot alter 3rd party schema due to company restrictions
There are many lines of unnamed constraint and WITH CHECK CHECK constraints generated in the script.
Question:
How can I be able to remove the lines to DROP unnamed Constraint on 3rd party schemas? - Even though the dll excludes 3rd party schema, it still has these unnamed constraints scripted/deployed. Also, it is not Adding them back too !!
How can I be able to skip/remove generating WITH CHECK CHECK CONSTRAINT on 3rd party schemas
Any suggestions will be greatly helpful.
EDIT:
Also, I found another issue. The deployment will not succeed due to Rows were detected. The schema update is terminating because data loss might occur
Output:
/*
The column [3rdParty].[MainTable_1].[Col1] is being dropped, data loss could occur.
The column [3rdParty].[MainTable_1].[Col2] is being dropped, data loss could occur.
The column [3rdParty].[MainTable_1].[Col3] is being dropped, data loss could occur.
The column [3rdParty].[MainTable_1].[Col4] is being dropped, data loss could occur.
*/
IF EXISTS (select top 1 1 from [3rdParty].[MainTable_1])
RAISERROR (N'Rows were detected. The schema update is terminating because data loss might occur.', 16, 127) WITH NOWAIT
GO
Regarding the unnamed constraints, I couldn't find any solution using sqlpackage.exe.
But Redgate SQL Compare has an option to ignore them called IgnoreSystemNamedConstraintAndIndexNames that ignores system generated constraints and generates a much cleaner script.
For example when comparing 2 dacpacs:
SQLCompare /Scripts1:"\unpacked_dacpac_source_folder" /Scripts2:"\unpacked_dacpac_dest_folder" /options:IgnoreSystemNamedConstraintAndIndexNames /scriptFile:"script_result.sql"
You can find more info here:
Handling System-named Constraints in SQL Compare
Related
I was running some SQL migration that first checked for some conditions to determine if it should be executed and I was surprised to discover that SQL server failed the script even in the case it was not executed. What was even more surprising - only broken INSERTs for existing tables caused the failure, while broken SELECTs and INSERTs for non-existing tables were not causing any failures.
Here is a minimalistic example.
The test table creation script:
CREATE TABLE [dbo].[TableWithManyColumnsTheFirstIsGuid](
[Id] [uniqueidentifier] NOT NULL,
[Value] [int] NOT NULL
)
The test script:
IF 1 = 2 -- to never execute
BEGIN
INSERT INTO NoSuchTable VALUES ('failure');
SELECT * FROM NoSuchTable;
INSERT INTO TableWithManyColumnsTheFirstIsGuid VALUES ('failure');
END
You will receive Column name or number of supplied values does not match table definition. error, which is correct for the third command because there is no way for SQL server to insert a string value in a table with Guid and int columns.
Why does it fail at all if that faulty INSERT statement would be never executed because of the IF condition?
If the answer is 'because SQL server validates commands in scripts even when not executing them', then the question is why do the other two broken commands not cause it to fail?
This behavior can bite you hard in a situation when you have a migration that removes a column and a disabled old migration that inserts some data into that column. The disabled migration will cause your entire migration script to fail. Of course, you can clean up your migration, but that's manual work and not automation-friendly when using migration scripts generated by EntityFramework.
Is this behavior by design? Is it specified in ANSI SQL or is it Microsoft-specific?
We are trying to work with temporal tables in SQL Server 2016. We are developing the SQL scripts in SSDT 15.1.6 in Visual Studio 2017, but we are experiencing issues when trying to deploy the dacpac that is generated during the build.
Our dacpac is deployed using SqlPackage.exe, and we encounter this error when attempting to deploy the dacpac:
Creating [dbo].[TestHISTORY].[ix_TestHISTORY]...
An error occurred while the batch was being executed.
Updating database (Failed)
Could not deploy package.
Error SQL72014: .Net SqlClient Data Provider:
Msg 1913, Level 16, State 1, Line 1
The operation failed because an index or statistics with name 'ix_TestHISTORY' already exists on table 'dbo.TestHistory'.
Error SQL72045: Script execution error. The executed script:
CREATE CLUSTERED INDEX [ix_TestHISTORY]
ON [dbo].[TestHistory]([SysStart] ASC, [SysEnd] ASC);
When we create the temporal table in SSDT we have the following:
CREATE TABLE [dbo].[Test]
(
[Id] INT NOT NULL PRIMARY KEY,
[SysStart] DATETIME2 (7) GENERATED ALWAYS AS ROW START NOT NULL,
[SysEnd] DATETIME2 (7) GENERATED ALWAYS AS ROW END NOT NULL,
PERIOD FOR SYSTEM_TIME ([SysStart], [SysEnd])
)
WITH (SYSTEM_VERSIONING = ON(HISTORY_TABLE=[dbo].[TestHISTORY], DATA_CONSISTENCY_CHECK=ON))
As far as I can tell the issue is with the dacpac creation. After the project is built, the dacpac created looks like this:
CREATE TABLE [dbo].[test]
(
[Id] INT NOT NULL PRIMARY KEY CLUSTERED ([Id] ASC),
[SysStart] DATETIME2 (7) GENERATED ALWAYS AS ROW START NOT NULL,
[SysEnd] DATETIME2 (7) GENERATED ALWAYS AS ROW END NOT NULL,
PERIOD FOR SYSTEM_TIME ([SysStart], [SysEnd])
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE=[dbo].[testHISTORY], DATA_CONSISTENCY_CHECK=ON));
GO
CREATE TABLE [dbo].[testHISTORY]
(
[Id] INT NOT NULL,
[SysStart] DATETIME2 (7) NOT NULL,
[SysEnd] DATETIME2 (7) NOT NULL
);
GO
CREATE CLUSTERED INDEX [ix_testHISTORY]
ON [dbo].[testHISTORY]([SysEnd] ASC, [SysStart] ASC);
GO
I suspect because we are using a temporal table with a default history table we can't have the dacpac create those extra creation statements. Since this is effectively causing SQL Server to try to create those items twice, leading to the above error.
Does anyone know what we might be missing? Or if you are deploying temporal tables using a dacpac, is your only option to use user-defined history tables?
We've had a number of issues between temporal tables and DACPAC's. A few tips that will go a long way:
Explicitly declare history tables - This goes way further than one would think. When adding/removing columns, you can define a default on history tables, allowing you to bypass a number of issues that arise when data is already in the tables.
Add defaults to EVERYTHING - This cannot be overstated. Defaults are the best friend of a DACPAC.
Review the scripts - It's nice to think of DACFx as hands off, but it's not. Review the scripts once in a while, and you'll gain a ton of insight (it appears you already are!)
Explicitly name your indices - DACFx sometimes uses temporary names for indices/tables/other stuff. Consistency is king, right?
Review ALL publish profile options - Sometimes, there are settings you didn't think of in the profile. It took us a lot of manual intervention before we realized there was a setting for transactional scripts in the publish profile.
Also look into who is turning your DACPAC into a script. VS uses SqlPackage.exe, but I sometimes get different results from the DACFx DLLs. It's likely a config thing that's different between the two, but it's tough to find out. Just try both, and see if one works better.
Best of luck! Hope this helps!
One potential hacky work around you can try is pre-deployment scripts;
https://msdn.microsoft.com/en-us/library/jj889461(v=vs.103).aspx
They are executed between 'Generation of deployment script' & 'Execution of the deployment script'. So if you can't avoid collision on index name, you can probably rename existing index before upgrade, This is hacky and i am assuming you are deploying/updating schema of a live DB and not creating a new DB
BTW, Where are the column names 'ValidFrom' & 'ValidTo' found in error message coming from?, If it is auto generated then it should be 'SysEnd' & 'SysStart'
Looking for a workaround for:
Error: SQL71609: System-versioned current and history tables do not have matching schemes. Mismatched column: 'XXXX'.
When trying to use SQL 2016 System-Versioned (Temporal) tables in SSDT for Visual Studio 2015.
I've defined a basic table:
CREATE TABLE [dbo].[Example] (
[ExampleId] INT NOT NULL IDENTITY(1,1) PRIMARY KEY,
[ExampleColumn] VARCHAR(50) NOT NULL,
[SysStartTime] datetime2 GENERATED ALWAYS AS ROW START HIDDEN NOT NULL,
[SysEndTime] datetime2 GENERATED ALWAYS AS ROW END HIDDEN NOT NULL,
PERIOD FOR SYSTEM_TIME (SysStartTime,SysEndTime)
)
WITH (SYSTEM_VERSIONING=ON(HISTORY_TABLE=[history].[Example]))
GO
(Assuming the [history] schema is properly created in SSDT). This builds fine the first time.
If I later make a change:
CREATE TABLE [dbo].[Example] (
[ExampleId] INT NOT NULL IDENTITY(1,1) PRIMARY KEY,
[ExampleColumn] CHAR(50) NOT NULL, -- NOTE: Changed datatype
[SysStartTime] datetime2 GENERATED ALWAYS AS ROW START HIDDEN NOT NULL,
[SysEndTime] datetime2 GENERATED ALWAYS AS ROW END HIDDEN NOT NULL,
PERIOD FOR SYSTEM_TIME (SysStartTime,SysEndTime)
)
WITH (SYSTEM_VERSIONING=ON(HISTORY_TABLE=[history].[Example]))
GO
Then the build fails with the error message above. Any change to the data type, length, precision, or scale will result in this error. (Including changing from VARCHAR to CHAR and VARCHAR(50) to VARCHAR(51); changing NOT NULL to NULL does not produce the error.) Doing a Clean does not fix things.
My current workaround is to make sure I have the latest version checked in to source control, then open the SQL Server Object Explorer, expand the Projects - XXXX folder and navigate to the affected table, then delete it. Then I have to restore the code (which SSDT deletes) from source control. This procedure is tedious, dangerous, and not what I want to be doing.
Has anyone found a way to fix this? Is it a bug?
I'm using Microsoft Visual Studio Professional 2015, Version 14.0.25431.01 Update 3 with SQL Server Data Tools 14.0.61021.0.
I can reproduce this problem. We (the SQL Server tools team) will work to get this fixed in a future version of SSDT. In the meantime, I believe you can work around this by explicitly defining the history table (i.e. add the history table with its desired schema to the project), and then manually keep the schema of the current and history table in sync.
If you encounter problems with explicitly defining the history table, try closing Visual Studio, deleting the DBMDL file in the project root, and then re-opening the project.
We just experienced this issue. We found a workaround by commenting out the system versioning elements of the table (effectively making it a normal table), building the project with the schema change we needed (which succeeds), and then putting the system versioning lines back in place (which also succeeds).
Just in case someone faced the same issue:
The fix is to go to [YourDatabaseProject]/bin/Debug folder and clear it and then build without removing anything.
Hope this helps!
I'm trying to update a database that is maintained and deployed using a database project (.sqlproj) in Visual Studio 2012. This is easier with SQL Server Management Studio, but in this case I have to deploy using a DACPAC.
What is the correct way to change a column to not be nullable, using DACPAC and without risking data loss?
A nullable column was added to a table. Now I need to publish an update that sets the column to not null and sets a default. Because there are rows in the table, the update fails. There is a setting to 'allow data loss' but that isn't an option for us and this update should not result in data loss. Here's a simple example that shows the problem:
CREATE TABLE [dbo].[Hello]
(
[Id] INT IDENTITY(100,1) NOT NULL PRIMARY KEY,
[HelloString] NVARCHAR(50) NULL ,
[Language] NCHAR(2) NOT NULL
)
Now publish that database and add rows, at least one row should have a null for HelloString.
Change the table definition to be:
CREATE TABLE [dbo].[Hello]
(
[Id] INT IDENTITY(100,1) NOT NULL PRIMARY KEY,
[HelloString] NVARCHAR(50) NOT NULL DEFAULT 'Hello' ,
[Language] NCHAR(2) NOT NULL
)
This cannot be published.
Error:
Rows were detected. The schema update is terminating because data loss might occur.
Next, I tried to add a pre-deployment script to set all NULL to be 'Hello':
UPDATE Hello SET HelloString = 'Hello' WHERE HelloString IS NULL
This publish attempt also fails, with the same error. Looking at the auto generated publish script it is clear why, but this seems to be incorrect behavior.
The NOT NULL alteration is applied BEFORE the default is added
The script checks for ANY rows, it doesn't matter whether there are
nulls or not.
The advice in the comment (To avoid this issue, you must add values to this column for all rows) doesn't solve this.
/*
The column HelloString on table [dbo].[Hello] must be changed from NULL to NOT NULL. If the table contains data, the ALTER script may not work. To avoid this issue, you must add values to this column for all rows or mark it as allowing NULL values, or enable the generation of smart-defaults as a deployment option.
*/
IF EXISTS (select top 1 1 from [dbo].[Hello])
RAISERROR (N'Rows were detected. The schema update is terminating because data loss might occur.', 16, 127) WITH NOWAIT
GO
PRINT N'Altering [dbo].[Hello]...';
GO
ALTER TABLE [dbo].[Hello] ALTER COLUMN [HelloString] NVARCHAR (50) NOT NULL;
GO
PRINT N'Creating Default Constraint on [dbo].[Hello]....';
GO
ALTER TABLE [dbo].[Hello]
ADD DEFAULT 'hello' FOR [HelloString];
Seen in SQL Server 2012 (v11.0.5343), SQL Server Data Tools 11.1.31009.1
When publishing a dacpac using SSMS, you'll not have access to the full set of publish options that are available when publishing from SqlPackage.exe or Visual Studio. I would suggest publishing with either SqlPackage.exe or with Visual Studio and enabling the "Generate smart defaults, where applicable" option. In the case of SqlPackage.exe, you would run a command like:
"C:\Program Files (x86)\Microsoft SQL Server\120\DAC\bin\SqlPackage.exe" /a:publish /sf:"C:\MyDacpac.dacpac" /tcs:"Data Source=MYSERVER;Initial Catalog=MYDATABASE;Integrated Security=true" /p:GenerateSmartDefaults=true
In the case of Visual Studio, you'd check the Generate smart defaults option in the Advanced publish options dialog.
I receive the following error (taken from replication monitor):
The option 'FILETABLE_STREAMID_UNIQUE_CONSTRAINT_NAME' is only valid when used on a FileTable. Remove the option from the statement. (Source: MSSQLServer, Error number: 33411)
The command attempted is:
CREATE TABLE [dbo].[WP_CashCenter_StreamLocationLink](
[id] [bigint] NOT NULL,
[Stream_id] [int] NOT NULL,
[Location_id] [numeric](15, 0) NOT NULL,
[UID] [uniqueidentifier] NOT NULL
)
WITH
(
FILETABLE_STREAMID_UNIQUE_CONSTRAINT_NAME=[UC_StreamLocation]
)
Now, for me there's two things unclear here.
Table already existed on subscriber, and I've set #pre_creation_cmd = N'delete' for the article. So I don't expect the table to be dropped and re-created. In fact, table still exists on subscriber side, although create table command failed to complete. What am I missing? Where does this create table command come from and why?
I don't understand why does this FILETABLE_STREAMID_UNIQUE_CONSTRAINT_NAME option appear in creation script. I tried generating create table script from table in SSMS and indeed, it's there. But what's weird, I can't drop and re-create the table this way - I get the very same error message.
EDIT: Ok, I guess now I know why the table is still there - I noticed begin tran in sql server profiler.
If your table on the publisher is truly not defined as a FileTable, then the issue has to do with the column named "Stream_id". I believe there is a known issue in SQL 2012 where if you have a column named "Stream_id", which is kind of reserved for FileTable/FileStream, it will automatically add that constraint, and unfortunately break Replication. The workaround here is to rename the column to something other than "Stream_id".
Another workaround is to set the schema option to not replicate constraints (guessing this will work). If you require constraints on the subscriber, you can then try to manually apply them on the sbuscriber after the fact (or script them out and use #post_snaphsot_script).