Exclude a post deployment script in Visual studio 2013 SQL Server project. I am not explaining the whole requirement why I have to do this. I am explaining part of my issue; please bear it.
I have the following script:
CREATE SCHEMA [College]
CREATE TABLE [College].[Department]
(
[Name] varchar(50) NOT NULL,
)
In a post deployment script I have a script file which is not included in the build, and I am inserting a value in the department table:
INSERT INTO [College].[Department]
([DeptId], [DeptName])
Values('77ACE81C-7C41-4A31-8F39-9814E36548A1', 'Computer')
As my department table doesn't have Department Id, I want to exclude this script, so based on a condition I am including this file by taking the help of a command-line variable. My post deployment file has the following entry:
IF('$(UseDepartMent)' = 1)
BEGIN
PRINT N'Command Line UseDepartment is 1'
:r .\Script.DepartMentTableEntry.sql
END
GO
(90,1): SQL72014: .NET SqlClient Data Provider: Msg 207, Level 16, State 1, Line 47 Invalid column name 'DeptId'.
Related
I am using EF Core (3.1.15). In a previous migration (also created in 3.1.15), a column was referenced that was dropped later on. The idempotent script does check if the migration was performed on the database (which it is, and the reference still shows in the __EFMigrationsHistory table). However the check doesn't have the expected result and the script due to the inexistent column.
Q: why is the inexistent column tripping the execution of the SQL script?
Script was created with
dotnet-ef migrations script -i -o migrations.sql
Relevant part of the automated script that fails, where ReferenceToLedgerId is the column dropped in later migration:
IF NOT EXISTS(SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20210612052003_CLedger')
BEGIN
UPDATE LedgerTable SET LedgerId = ReferenceToLedgerId
END;
Error:
Msg 207, Level 16, State 1, Line 3
Invalid column name 'ReferenceToLedgerId'
When running the following SQL query, the result comes back as expected:
SELECT *
FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20210612052003_CLedger'
MigrationId
ProductVersion
20210612052003_CLedger
3.1.15
The database is Azure SQL Database. Script doesn't fail on local SQL dev database. A dozen migrations have been applied since then, and only now the script fails.
Below was the call that created the specific script:
migrationBuilder.Sql("UPDATE LedgerTable set LedgerId = ReferenceToLedgerId", true);
I tried to place the table and column names in square brackets, but that made no difference (eg. [ReferenceToLedgerId]. The script fails in Azure DevOps release when using SQLCMD and also fails when using Azure Data Studio, both accessing the Azure SQL Database.
Additional check
I changed the script to do a quick check:
PRINT '#Before IF'
IF NOT EXISTS(SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20210612052003_CLedger')
BEGIN
PRINT '#Within IF'
--UPDATE LedgerTable SET LedgerId = ReferenceToLedgerId
END;
PRINT '#After IF'
To which I get the following result:
Started executing query at Line 1
#Before IF #After IF
Total execution time: 00:00:00.010
If I uncomment the UPDATE statement it fails again. So I can only conclude that the code path works as intended, but that the server still checks for the existence of the column. I am not familiar with SQL to understand why this would be, or why it only fails for this one line while the column itself is referenced in other lines of the SQL script without it failing.
That batch will fail on every version of SQL Server. eg
use tempdb
go
create table __EFMigrationsHistory(MigrationId nvarchar(200))
create table LedgerTable(LedgerId int)
go
insert into __EFMigrationsHistory(MigrationId) values (N'20210612052003_CLedger')
go
IF NOT EXISTS(SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20210612052003_CLedger')
BEGIN
UPDATE LedgerTable SET LedgerId = ReferenceToLedgerId
END;
Fails with
Msg 207, Level 16, State 1, Line 8
Invalid column name 'ReferenceToLedgerId'.
Because the batch cannot be parsed and compiled. It's simply not legal to reference a non-existent table or column in a TSQL batch.
You can work around this by using dynamic SQL, so that the batch referencing the non-existent column is not parsed and compiled unless the migration is being applied.
migrationBuilder.Sql("exec('UPDATE LedgerTable set LedgerId = ReferenceToLedgerId')", true);
This is documented here:
Tip
Use the EXEC function when a statement must be the first or only
one in a SQL batch. It might also be needed to work around parser
errors in idempotent migration scripts that can occur when referenced
columns don't currently exist on a table.
https://learn.microsoft.com/en-us/ef/core/managing-schemas/migrations/operations
Using SQL Server 2016. Have a locally hosted database that uses the Windows login for the sa, which is what I am using to login.
Yesterday I ran
CREATE TABLE [Otis].[AnalyzerGroups]
(
[id] [int] IDENTITY,
[Name] [varchar](50) NULL
);
and got command successfully completed. Today I tried selected from this table but got an error:
Msg 208, Level 16, State 1, Line 1
Invalid object name 'Otis.AnalyzerGroups'
So I thought I misremembered and tried running the create statement again but then got the error -
Msg 15530, Level 16, State 1, Line 1
The object with name "AnalyzerGroups" already exists.
The statement has been terminated.
So then I tried DROP TABLE [Otis].[AnalyzerGroups]; and got
Msg 3701, Level 11, State 5, Line 1
Cannot drop the table 'Otis.AnalyzerGroups', because it does not exist or you do not have permission.
I tried making a new test table and the same thing. The first time I run the create statement command successfully completes, but then I can't select / insert / drop from the table, and I cannot see it in the Object explorer either.
I assume this must be some permissions issue but I don't know what property is keeping me from viewing these tables - its not like I'm putting security on these tables, and I can see every other table in our database. Any thoughts?
You put it most likely in the wrong database. Try this.
sp_MSforeachdb 'SELECT "?" AS DB, * FROM [?].sys.tables WHERE name like
''%AnalyzerGroups%'''
There was a trigger in the SQL Server database that fired any time a new table was created under any schema, and would then transfer it to the dbo schema. So my tables did exist, but they were under the schema I was expecting because of this trigger. Why does this trigger exist? Got me. But it's in production and has been for over a decade so there's some reason/it's definitely not changing. Thanks for the help though everyone.
I'm setting up a new VM on a server to offload SQL Server database loading from my laptop. In doing so, I'd like to be able to execute stored procedures (no params, just 'exec storedprocedure') in my database via Python, but it's not working.
Stored procedure call worked when using sqlcmd via a batch file and in SSMS, but i'd like to make it all python based.
The stored procedure is appending fact tables follows the below general format:
--staging tbl drop and creation
if object_id(stagingtbl) is not null drop tabl stagingtbl
create table stagingtbl
(fields datatypes nullable
)
--staging tbl load
bulk insert stagingtbl
from 'c:\\filepath\\filename.csv'
with (
firstrow = 2
, rowterminator = '\n'
,fieldterminator = ','
, tablock /*don't know what tablock does but it works...*/
)
--staging table transformation
; with cte as (
/*ETL process to transform csv file into my tbl structure*/
)
--final table load
insert final_tbl
select * from cte
/*
T-SQL update the final table's effect to date, based on subsequent effect from date.
eg:
id, effectfromdate, effecttodate
1,1/1/19, 1/1/3000
1,1/10/19, 1/1/3000
becomes
id, effectfromdate, effecttodate
1,1/1/19, 1/10/19
1,1/10/19, 1/1/3000
*/
The stored procedure works fine with sqlcmd and in ssms but in python (pyodbc) executing the query 'exec storedprocedure', I get the error message:
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]
Cannot bulk load because the file "c:\filepath\filename.csv" could not be opened.
Operating system error code 3(The system cannot find the path specified.). (4861) (SQLExecDirectW)')
When the csv file is there, no misspellings in path or filename, and I can open the csv when double clicking on it, and no one has the csv open.
With continued experimentation I've established the problem is not with python or pyodbc. In SSMS on my laptop (host machine of db) the stored procedures work just fine, but in SSMS on the VM the stored procedures cause the same error. This tells me that my question isn't the root problem and I had more digging to do. The error (in SSMS) is below.
Msg 4861, Level 16, State 1, Procedure Append_People, Line 71 [Batch Start Line 0]
Cannot bulk load because the file "N:\path\filename.csv" could not be opened. Operating system error code 3(The system cannot find the path specified.).
Once I established the problem is in SSMS, I broadened my search and discovered the issue is the path for the bulk insert command has to be relative to the machine hosting the database. So in the VM (client machine until I migrate the db) when I use the path c:\ thinking it's the VM's c:\ drive the stored procedure is looking at the c:\ of my laptop since it's the host machine. With that I also learned that on a shared drive (N:\) access is delegated and that is its own issue (https://dba.stackexchange.com/questions/44524/bulk-insert-through-network).
So I'm going to focus on migrating the database first, then that'll solve my problem. Thanks to all who tried to help
How do we configure our TFS gated build validation to catch parser validation errors?
For example, we had a stored procedure with an invalid statement that kept passing the gated TFS check-in XAML build validation and even passed SSDT publish to a SQL Server 2008R2 database for two whole years!
CREATE PROCEDURE This_Should_Fail
AS BEGIN
DECLARE #TableVariable TABLE(
ID INT IDENTITY(1, 1),
Name VARCHAR(MAX)
)
SELECT DISTINCT TOP 1 Name
FROM #TableVariable
ORDER BY ID
END
When I run this statement I get the appropriate error, however when I run the full version in TFS there is no error!
Msg 145, Level 15, State 1, Procedure This_Should_Fail, Line 7
ORDER BY items must appear in the select list if SELECT DISTINCT is specified.
But the error was never encountered until recently when the definition changed to be like this, which doesn't even throw any errors when run!
CREATE PROCEDURE This_Should_Fail
AS BEGIN
DECLARE #TableVariable TABLE(
ID INT IDENTITY(1, 1),
Name VARCHAR(MAX),
ForeignKeyID INT
)
--create some sample data
INSERT INTO #TableVariable(Name, ForeignKeyID)
VALUES ('Obj1', 999), ('Obj2', 999), ('Obj3', 0), ('Obj4', 0)
DECLARE #ForeignKeyID INT = 999--some lookup here
SELECT DISTINCT TOP 1 Name
FROM #TableVariable
WHERE ForeignKeyID = #ForeignKeyID
END
And it took a whole month after that change was made before the change in the TFS branch failed to deploy anywhere. In both cases, there was no ordering by all selected columns.
This is confusing me even more because neither version in TFS (which I can't include because it's my company's IP) throws any error even when manually run in SSMS 2014!
Thanks.
Gated check-in is a form of continuous integration build. In TFS, it creates a shelveset containing the code that's being validated, then runs a build of that code. Only if that code builds successfully and all configured unit tests pass does the code actually get checked in.
In other words, the validation is controlled by your build/test directly not by TFS. If the error is not caught by your build or not test out by your test. Then it will effect after actually deployed to a server.
One way to do this is catching the log info of your build and then use Logging Commands and exit 1 to fail the build. More details you could refer this question: How to fail the build from a PowerShell task in TFS 2015 (for vNext Build)
When I generate a migration script using the ef 7 powershell commands
dnx ef migrations add Initial -c MyDbContext
dnx ef migrations script -c MyDbContext
I get a script that has, in the first few lines, the following;
Using context 'MyDbContext'.
Generating up script for migration '20151001104737_Initial'.
IF OBJECT_ID(N'__MigrationHistory') IS NULL
CREATE TABLE [__MigrationHistory] (
[MigrationId] nvarchar(150) NOT NULL,
[ProductVersion] nvarchar(32) NOT NULL,
CONSTRAINT [PK_HistoryRow] PRIMARY KEY ([MigrationId])
);
GO
When I run the script on my SQL Databases to apply my migration I get the error
Msg 102, Level 15, State 1, Line 1
Incorrect syntax near 'MyDbContext'.
Msg 208, Level 16, State 1, Line 98
Invalid object name '__MigrationHistory'.
I have run this script on SQL Server database versions 11, 12 & 13 and I get the same error throughout. It is clearly complaining that the table name __MigrationHistory isnt a valid table name it seems.
Figured out the problem
I was piping the ef command into a file and the first two lines are just descriptions of what is going on while the command is running
so if you are doing something like
dnx ef migrations script -c MyDbContext > myscript.sql
then make sure to remove the first two lines before running the script