I am using MSSQL and evaluating Liquibase to use for migrations. So, I wanted to generate my first changelog using generateChangeLog. My database has two schemas: the default schema and another called 'blah'. I have a table in each schema with the same table name: test1. I ran:
liquibase --dataOutputDirectory=./data/ --schemas=blah,dbo --changeLogFile=changelog.mssql.sql --includeSchema=true generateChangeLog
It completed and I looked at the generated SQL:
-- liquibase formatted sql
-- changeset bmccord2:1604068236633-1
CREATE TABLE blah.test1 (id int NOT NULL, name varchar(255), CONSTRAINT PK__test1__3213E83F4F883C7C PRIMARY KEY (id));
-- changeset bmccord2:1604068236633-2
INSERT INTO blah.test1 (id, name) VALUES (1, 'Brian'),(2, 'Kim');;
-- changeset bmccord2:1604068236633-3
CREATE TABLE dbo.test1 (id int NOT NULL, name varchar(255), CONSTRAINT PK__test1__3213E83F6FD50901 PRIMARY KEY (id));
-- changeset bmccord2:1604068236633-4
INSERT INTO dbo.test1 (id, name) VALUES (1, 'Brian'),(2, 'Kim');;
At first, it looks ok, but then I noticed that the data being inserted into the blah.test1 table is not the data that is actually in that table. The data in that table is:
"id","name"
"1","Miranda"
"2","Kyle"
So, it is using the second table's data for both tables. It is also only generating one .csv file in the data folder.
Obviously, this isn't my real database. I simplified the problem down to the smallest thing that causes the problem.
Is there any way to make this work properly?
Checking their forum they state that:
The way that Liquibase is designed, it only works with a single schema at a time.
If it fits your use case you could try to define two separate migrations and apply them one by one, e.g:
liquibase --dataOutputDirectory=./data/ --schemas=blah --changeLogFile=changelog.blah.mssql.sql --includeSchema=true generateChangeLog
liquibase --dataOutputDirectory=./data/ --schemas=dbo --changeLogFile=changelog.dbo.mssql.sql --includeSchema=true generateChangeLog
or if you would prefer to apply the exact same changelog for both database you can call it twice with the same changeLogFile. In that case only the --schemas needs to be adjusted. (--schemas=dbo and --schemas=blah)
Related
I have two tables in SQL Server:
Person
ID (PK, int, IDENTITY)
Name (varchar(100))
UploadedBy (varchar(50))
DateAdded (datetime)
PersonFile
ID (PK, int, IDENTITY)
PersonId (FK, int)
PersonFile (varchar(max))
I am reading in a large file (150MB), and I have a script component that can successfully parse the file into several columns. The issue is that I need to insert the first 3 columns of my parsed data row into my Person table first, then use the ID of that Row to insert the final column into my PersonFile table. Is there an easy way to do this in SSIS?
I suppose I could technically script everything out to handle inserts in the database, but I feel like in that case, I might as well just skip SSIS altogether and user powershell. I also thought about writing a procedure in SQL server and then passing the information to the procedure to handle inserts. But again, this seems very inefficient.
What's the best way for me to insert a row of data into two tables, if one of them has a foreign key constraint?
I think the best way is to use a stage table in the database to hold the parsed source file and then use stored procedures or SQL-query to load your tables. There is a lookup component in SSIS that can be used for your case but I try avoiding it for various reasons.
Create a table resembeling the source file, something like:
CREATE TABLE dbo.[SourceFileName](
Name nvarchar(100) NULL,
UploadedBy nvarchar(50) NULL,
DateAdded datetime NULL,
PersonFile nvarchar(max) NULL
)
Truncate the stage table. Use a dataflow component to get the source data. Use script or stored procedures to insert the source data in your destination table (begin with Person and the load PersonFile). Your SSIS dataflow should look something like this:
For the insert script for person do something like:
INSERT INTO dbo.Person (Name, UploadedBy,DateAdded)
SELECT Name,UploadedBy,DateAdded
FROM dbo.SourceFileName;
For the insert for PersonFile make a join to the destination table:
INSERT INTO dbo.PersonFile(PersonId,PersonFile)
SELECT
Person.ID,
SourceFile.PersonFile
FROM dbo.SourceFileName SourceFile
JOIN dbo.Person Person
ON Person.Name = SourceFile.Name
You should also add a UNIQUE CONSTRAINT to the column that identifies the person (Name for example).
One very common thing to do would be to stage the data first.
So you insert all columns into a table on the server, which also has an extra nullable column for the PersonID.
Then you’d have a stored procedure which inserts unique Person records into the Person table, and updates the staging table with the resulting PersonID, which is the extra field you need for the PersonFile insert, which could then be performed either in the same procedure or another one. (You’d call these procedures in SSIS with an Execute SQL Task.)
I suppose this could possibly be done purely in SSIS, for example with a Script Destination that performs an insert and retrieves the PersonID for a second insert, but I’m fairly sure performance would take a huge hit with an approach like that.
After rebuilding all of the tables in one of my SQL SERVER databases, into a new database, I failed to set the 'ID' column to IDENTITY and PRIMARY KEY for many of the tables. Most of them have data.
I discovered this T-SQL, and have successfully implemented it for a couple of the tables already. The new/replaced ID column contains the same values from the previous column (simply because they were from an auto-incremented column in the table I imported from), and my existing stored procedures all still work.
Alter Table ExistingTable
Add NewID Int Identity(1, 1)
Go
Alter Table ExistingTable Drop Column ID
Go
Exec sp_rename 'ExistingTable.NewID', 'ID', 'Column'
--Then open the table in Design View, and set the new/replaced column as the PRIMARY KEY
--I understand that I could set the PK when I create the new IDENTITY column
The new/replaced ID column is now the last column in the table, and so far, I haven't ran into issues with the ASP.Net/C# data access objects that call the stored procedures.
As mentioned, each of these tables had no PRIMARY KEY (nor FOREIGN KEY) set. With that in mind, are there any additional steps I should take to ensure the integrity of the database?
I ran across this SO post, which suggests that I should run the 'ALTER TABLE REBUILD' statement, but since there was no PK already set, do I really need to do this?
Ultimately, I just want to be sure I'm not creating issues that won't appear until later in the game, and be sure the methods I'm implementing are sound, logical, and ensure data integrity.
I suppose it might be a better option to DROP/RECREATE the table with the proper PK/IDENTITY column, and I could write some T-SQL to dump the existing data into a TEMP table, then drop/recreate, and re-populate the new table with data from the TEMP table. I specifically avoided this option as it seems much more aggressive, and I don't fully understand what it means for the Stored Procedures/Functions, etc., that depend on these tables.
Here is an example of one of the tables I've performed this on. You can see the NewID values are identical to the original ID.enter image description here
Give this a go; it's rummaged up from a script we used a few years ago in a similar situation, can't remember what version of SQLS it was used against.. If it works out for your scenario you can adapt it to your tables..
SELECT MAX(Id)+1 FROM causeCodes -- run and use value below
CREATE TABLE [dbo].[CauseCodesW]( [ID] [int] NOT NULL IDENTITY(put_maxplusone_here,1), [Code] [varchar](50) NOT NULL, [Description] [varchar](500) NULL, [IsActive] [bit] NOT NULL )
ALTER TABLE CauseCodes SWITCH TO CauseCodesW;
DROP TABLE CauseCodes;
EXEC sp_rename 'CauseCodesW','CauseCodes';
ALTER TABLE CauseCodes ADD CONSTRAINT PK_CauseCodes_Id PRIMARY KEY CLUSTERED (Id);
SELECT * FROM CauseCodes;
You can now find any tables that have FKs to this table and recreate those relationships..
I had created identical databases in different environments: Dev and QA. While doing the development, I have changed a few tables in the Dev database. How do I change the QA database to make it again identical to the Dev database in terms of tables (and constraints)?
I checked the below link:
[Copy one database to another database
Steps in the above link did not directly work because the tables and constraints already existed in the second database. I did modification in the sql file after the steps
I followed the below steps:
Right-click on the database you want to copy
Choose 'Tasks' > 'Generate scripts'
'Select specific database objects' and Check 'Tables'
Click on Next. Again click on Next.
This exports .sql file to the path shown while following the above steps.
I edited the script file and changed the database name to the QA database name (at the top of the script).
After this added the below line above every create table statement as the table exist.
DROP TABLE IF EXISTS tablename;
On running the query, I get an error message saying
Could not drop object tablename because it is referenced by a FOREIGN
KEY constraint.
How do I change the second database to make it identical to the first database in terms of tables (and constraints)?
Thank You
Well, the most straight forward solution would be to drop all constraints first. You could add a drop constraint per constraint above your drop table lines, though that may be tedious.
An answer to this question has a script that drops every constraint in a database and table. You could omit the table name param in the where.
But, since you're destroying everything in the database, it might be easiest to delete and recreate the database. Then you wouldn't need to add the drop table statements to the create script from dev.
In your scripts, separate tables creation, from insertion of records, for later.
Example (before):
create table one ...
insert into table one ...
create table two ...
insert into table two ..
create table three ...
insert into table three ..
Example (after)
create table one ...
create table two ...
create table three ...
insert into table one ...
insert into table two ..
insert into table three ..
If you have foreign constraints, check that the destination tables (primary table or master table), are created and filled first, than the source tables (secondary table or slave table).
Example (before):
create table one
(int onekey primary key, int threekey foreing key)
create table three ...
(int threekey primary key)
insert into table one ...
insert into table three ..
Example (after):
create table three ...
(int threekey primary key)
create table one
(int onekey primary key, int threekey foreing key)
insert into table three ..
insert into table one ...
And, finally, if you use automatic of self generated keys, turn off before table insertion, and turn on back, after table insertion, because the DB server may assign new keys to destination tables, and source tables, may expect the previous keys.
I created a table into an external server
CREATE FOREIGN TABLE external_table (
field_1 varchar(15) NULL,
field_2 int4 NULL
)
SERVER server_name
OPTIONS(compression 'pglz', stripe_row_count '500000');
Now I want to insert into external_table, but if I run this query
INSERT INTO external_table (field_1, field_2) VALUES ('test',1);
It return this error
ERROR: operation is not supported
How can I add record into a foreign table?
I've tried with the following insert
INSERT INTO external_table (field_1, field_2) select 'test',1;
It works, but I can't use a INSERT INTO with SELECT statment.
Looks like the extension you are using supports "insert into ... select .." but not direct inserts.
you can use you should probably ask this question while specifying the extension.
PS: It looks like the extension you use is cstore_fdw. It does not support direct inserts, because it completely cancels benefits of using columnar storage and create some extra overhead. If you are using cstore_fdw, try to use bulk inserts instead of single row ones. Inserting into a regular table and moving data into cstore_fdw table when data reaches certain size (i.e. stripe_row_count number of rows) is much better option.
I am trying to change a primary key Id to identity to increment 1 on each entry. But the column has been referenced already by other tables. Is there any way to set primary key to auto increment without dropping the foreign keys from other tables?
If the table isn't that large generate script to create an identical table but change the schema it created to:
CREATE TABLE MYTABLE_NEW (
PK INT PRIMARY KEY IDENTITY(1,1),
COL1 TYPEx,
COL2 TYPEx,
COLn
...)
Set your database to single-user mode or make sure no one is in the
database or tables you're changing or change the table you need to
change to READ/ONLY.
Import your data into MYTABLE_NEW from MYTABLE using set IDENTITY_INSERT on
Script your foreign key constraints and save them--in case you need
to back out of your change later and/or re-implement them.
Drop all the constraints from MYTABLE
Rename MYTABLE to MYTABLE_SAV
Rename MYTABLE_NEW to MYTABLE
Run constraint scripts to re-implement constraints on MYTABLE
p.s.
you did ask if there was a way to not drop the foreign key constraints. Here's something to try on your test system. on Step 4 run
ALTER TABLE MYTABLE NOCHECK CONSTRAINT ALL
and on Step 7 ALTER TABLE MYTABLE CHECK CONSTRAINT ALL. I've not tried this myself -- interesting to see if this would actually work on renamed tables.
You can script all this ahead of time on a test SQL Server or even a copy of the database staged on a production server--to make implementation day a no-brainer and gauge your SLAs for any change control procedures for your company.
You can do a similar methodology by deleting the primary key and re-adding it back, but you'll need to have the same data inserted in the new column before you delete the old column. So you'll be deleting and inserting schema and inserting primary key data with this approach. I like to avoid touching a production table if at all possible and having MYTABLE_SAV around in case "anything" unexpected occurs is a comfort to me personally--as I can tell management "the production data was not touched". But some tables are simply too large for this approach to be worthwhile and, also, tastes and methodologies differ largely from DBA to DBA.