Idempotent PostgreSQL DDL scripts - database

I'm looking for a way to script postgreSQL schema changes in an idempotent manner.
In MSSQL I could do something like this:
if(not exists(select * from information_schema.columns where table_name = 'x' and column_name = 'y'))
begin
alter table x add y int
end
go
PostgreSQL doesn't seem to allow ad-hoc pl/pgsql in the same way MSSQL does with T-SQL so I can't use control structures in a SQL script and run it with psql -f x.sql.
I know PostgreSQL will throw an error if the object already exists but I don't want to have to ignore errors.
I could use some schema versioning technique like dbdeploy but I really like the simplicity of running a set of files through psql without incurring any unwanted side effects.
Is this possible?
Thanks,
Mark

Disclaimer: I know, this is a very old question and already has an accepted answer.
But I'd like to register here a totally idempotent script, without external links.
A simple script demonstrating PostgreSQL 9.5+ idempotent built-in capabilities. You can run this script as many times as you wish.
Here we go:
--Table
CREATE TABLE IF NOT EXISTS person (
id integer NOT NULL,
person_name character varying(40) NOT NULL,
updated_date date,
CONSTRAINT person_pkey PRIMARY KEY (id)
);
--Index
CREATE INDEX IF NOT EXISTS idx_person_name ON person (person_name);
--Sequence
CREATE SEQUENCE IF NOT EXISTS seq_person_inc;
--Function
CREATE OR REPLACE FUNCTION simple_sum(a_integer int, b_integer int) RETURNS INT
AS $$ SELECT a_integer+b_integer $$
LANGUAGE SQL;
--View
CREATE OR REPLACE VIEW vw_select_1 AS
SELECT 1;
--Role
DO $$
BEGIN
CREATE ROLE rick_deckard;
EXCEPTION
WHEN duplicate_object THEN
RAISE NOTICE 'Role already exists. Ignoring...';
END$$;
--Simple insert
INSERT INTO person (id, person_name) VALUES (1, 'HAL-9000');
--Upsert (insert + update)
INSERT INTO person (id, person_name) VALUES (1, 'Betrayer') ON CONFLICT ON CONSTRAINT person_pkey DO UPDATE SET person_name = EXCLUDED.person_name;
--Upsert (ignoring duplicate error)
INSERT INTO person (id, person_name) VALUES (1, 'HAL-9000') ON CONFLICT ON CONSTRAINT person_pkey DO NOTHING;
--Upsert (ignoring any error)
INSERT INTO person (id, person_name) VALUES (1, 'HAL-9000') ON CONFLICT DO NOTHING;
--Field
DO $$
BEGIN
ALTER TABLE person ADD COLUMN id_another_person INTEGER;
EXCEPTION
WHEN duplicate_column THEN
RAISE NOTICE 'Field already exists. Ignoring...';
END$$;
--Constraint
DO $$
BEGIN
ALTER TABLE person ADD CONSTRAINT person_id_another_person_fkey FOREIGN KEY (id_another_person) REFERENCES person (id);
EXCEPTION
WHEN duplicate_object THEN
RAISE NOTICE 'Constraint already exists. Ignoring...';
END$$;
--Trigger
CREATE OR REPLACE FUNCTION person_trigger_function() RETURNS trigger AS $BODY$
BEGIN
--Something complex here =)
RETURN NEW;
END;
$BODY$
LANGUAGE plpgsql;
DO $$
BEGIN
CREATE TRIGGER person_trigger BEFORE INSERT OR UPDATE ON person FOR EACH ROW EXECUTE PROCEDURE person_trigger_function();
EXCEPTION
WHEN duplicate_object THEN
RAISE NOTICE 'Trigger already exists. Ignoring...';
END$$;
--Drop
DROP TRIGGER IF EXISTS person_trigger ON person;
DROP INDEX IF EXISTS idx_person_name;
ALTER TABLE person DROP COLUMN IF EXISTS person_name;
ALTER TABLE person DROP CONSTRAINT IF EXISTS person_id_another_person_fkey;
DROP ROLE IF EXISTS rick_deckard;
DROP VIEW IF EXISTS vw_select_1;
DROP FUNCTION IF EXISTS simple_sum(integer, integer);
DROP FUNCTION IF EXISTS person_trigger_function();
DROP TABLE IF EXISTS person;
DROP SEQUENCE IF EXISTS seq_person_inc;

you might find this blogpost helpful: http://www.depesz.com/index.php/2008/06/18/conditional-ddl/

You should be able to use plpgsql:
create language plpgsql;
create function f() ... as $$
<plpgsql code>
$$language plpgsql;
select f();

Related

Create / Alter procedure referencing non-existing table or view

According to MSDN:
A procedure can reference tables that do not yet exist. At creation
time, only syntax checking is performed. The procedure is not compiled
until it is executed for the first time. Only during compilation are
all objects referenced in the procedure resolved. Therefore, a
syntactically correct procedure that references tables that do not
exist can be created successfully; however, the procedure fails at
execution time if the referenced tables do not exist.
I was always under the impression that referenced tables are checked. For example if you reference an incorrect column in an existing view it complains at compile time:
So what is going on here, why are columns checked within the procedure but not views / tables?
The document is referencing that, at the time to Stored Procedure is created, it isn't validated. Take this batch:
USE Sandbox;
GO
CREATE PROC dbo.MyProc #ID int AS
BEGIN
SELECT *
FROM dbo.MyTable
WHERE ID = #ID;
END;
GO
CREATE TABLE dbo.MyTable (ID int,
SomeValue varchar(20));
INSERT INTO dbo.MyTable
VALUES(1,'asdjka'),
(2,'asdkj');
GO
EXEC dbo.MyProc 1;
GO
DROP TABLE dbo.MyTable;
DROP PROC dbo.MyProc;
Notice that I CREATE dbo.MyProc before dbo.MyTable, even though it references that object. That's because the validity of the objects in the procedure isn't checked at the point the procedure is created or altered.
A VIEW on the other hand, is checked at the time that it is created.
CREATE TABLE dbo.MyTable (ID int,
SomeValue varchar(20));
INSERT INTO dbo.MyTable
VALUES(1,'asdjka'),
(2,'asdkj');
GO
--Fails
CREATE VIEW dbo.MyView AS
SELECT ID,
SomeValue,
SomeInt
FROM dbo.MyTable;
GO
--Fails
CREATE VIEW dbo.MyOtherView AS
SELECT *
FROM dbo.MyOtherTable;
GO
DROP TABLE dbo.MyTable;
Depending on the object type depends on what is validated and isn't when the DDL statement is issued.
Edit: It seems that what the OP is questioning is why do that get an error if the object does exist, but they reference a column that doesn't. For exmaple take the below batch:
USE Sandbox;
GO
CREATE TABLE dbo.MyTable (ID int,
SomeValue varchar(20));
INSERT INTO dbo.MyTable
VALUES(1,'asdjka'),
(2,'asdkj');
GO
CREATE PROC dbo.MyProc #ID int AS
BEGIN
SELECT ID,
SomeValue,
AnotherValue
FROM dbo.MyTable
WHERE ID = #ID;
END;
GO
DROP TABLE dbo.MyTable;
GO
DROP PROC dbo.MyProc;
GO
This fails as the column AnotherValue does not exist. This is actually covered in the very documentation you quote:
A procedure can reference tables that do not yet exist. At creation time, only syntax checking is performed.
It explicitly states you can reference a table that does not exist. It makes no mention of objects that don't, or (more specifically) columns in a table/object. Referencing a table that does exist will be validated at creation time.

Creating temp tables in sybase

I am running into an issue with creating temp tables in Sybase db. We have a sql where we create a temp table, insert/update it and do a select * from it at the end of get some results. We are invoking this sql from the service layer using spring jdbc tmplate. The first run works fine, but the next subsequesnt runs fails with error
cannot create temporary table <name>. Prefix name is already in use by another temorary table
This is how I am checking if table exists:
if object_id('#temp_table') is not null
drop table #temp_table
create table #temp_table(
...
)
Anything I am missing here?
Might not be a great response, but I also have that problem and I have 2 ways around it.
1. Do the IF OBJECT_ID Drop Table as a separate execute prior to the query
2. Do the Drop Table without the IF OBJECT_ID() right after your query.
You are really close but temp tables require using the db name before too.
IF OBJECT_ID('tempdb..#Results') IS NOT NULL
DROP TABLE #Results
GO
It would be the same if you were checking if a user table in another database existed.
IF OBJECT_ID('myDatabase..myTable') IS NOT NULL
DROP TABLE myDatabase..myTable
GO
NOTE: A bit more info on BigDaddyO's first suggestion ...
The code snippet you've provided, when submitted as a SQL batch, is parsed as a single unit of work prior to the execution. Net result is that if #temp_table already exists when the batch is submitted, then the compilation of the create table command will generate the error. This behavior can be seen in the following example:
create table #mytab (a int, b varchar(30), c datetime)
go
-- your code snippet; during compilation the 'create table' generates the error
-- because ... at the time of compilation #mytab already exists:
if object_id('#mytab') is not NULL
drop table #mytab
create table #mytab (a int, b varchar(30), c datetime)
go
Msg 12822, Level 16, State 1:
Server 'ASE200', Line 3:
Cannot create temporary table '#mytab'. Prefix name '#mytab' is already in use by another temporary table '#mytab'.
-- same issue occurs if we pull the 'create table' into its own batch:
create table #mytab (a int, b varchar(30), c datetime)
go
Msg 12822, Level 16, State 1:
Server 'ASE200', Line 1:
Cannot create temporary table '#mytab'. Prefix name '#mytab' is already in use by another temporary table '#mytab'.
As BigDaddyO has suggested, one way to get around this is to break your code snippet into two separate batches, eg:
-- test/drop the table in one batch:
if object_id('#mytab') is not NULL
drop table #mytab
go
-- create the table in a new batch; during compilation we don't get an error
-- because #mytab does not exist at this point:
create table #mytab (a int, b varchar(30), c datetime)
go

Best way how to validate field when insert/update

I am trying to apply custom validation using CHECK + UDF, nonetheless, it doesn't work properly in my case, moreover after searching there is an opinion that such approach may impact performance and it is not robust.
Is in the T-SQL some existing tools to resolve my issue?
Here is my sql:
DROP TABLE IF EXISTS dbo.test_name_uniq;
CREATE TABLE dbo.test_name_uniq (
name VARCHAR(255),
state VARCHAR(255)
)
ALTER TABLE dbo.test_name_uniq DROP CONSTRAINT IF EXISTS test_name_uniq_constraint;
DROP FUNCTION IF EXISTS dbo.validate_test_name_uniq;
GO
CREATE FUNCTION dbo.validate_test_name_uniq(#name VARCHAR(255))
RETURNS BIT
AS
BEGIN
DECLARE #unique_name BIT = 0;
SELECT #unique_name = CASE
WHEN COUNT(*) > 0
THEN 0
ELSE 1
END
FROM dbo.test_name_uniq i
WHERE i.name = #name AND i.state <> 'Removed';
RETURN #unique_name;
END;
GO
ALTER TABLE dbo.test_name_uniq
WITH NOCHECK ADD CONSTRAINT test_name_uniq_constraint CHECK (dbo.validate_test_name_uniq(name) = 1 );
GO
DELETE FROM dbo.test_name_uniq;
GO
INSERT INTO dbo.test_name_uniq (name, state) VALUES
('Test application', 'Active');
Every time when I try to insert a row, I get the error:
The INSERT statement conflicted with the CHECK constraint "test_name_uniq_constraint". The conflict occurred in database "data_local_test", table "dbo.test_name_uniq", column 'name'.
Thanks in advance!
You can try with a UNIQUE FILTERED INDEX on column name where state<>'Removed':
CREATE UNIQUE INDEX uix_name
ON dbo.test_name_uniq (name)
WHERE where state<> 'Removed';
Hope it helps.

TRANSACTION rollback not working as expected

I'm doing some DB schema re-structuring.
I have a script that looks broadly like this:
BEGIN TRAN LabelledTransaction
--Remove FKs
ALTER TABLE myOtherTable1 DROP CONSTRAINT <constraintStuff>
ALTER TABLE myOtherTable2 DROP CONSTRAINT <constraintStuff>
--Remove PK
ALTER TABLE myTable DROP CONSTRAINT PK_for_myTable
--Add replacement id column with new type and IDENTITY
ALTER TABLE myTable ADD id_new int Identity(1, 1) NOT NULL
GO
ALTER TABLE myTable ADD CONSTRAINT PK_for_myTable PRIMARY KEY CLUSTERED (id_new)
GO
SELECT * FROM myTable
--Change referencing table types
ALTER TABLE myOtherTable1 ALTER COLUMN col_id int NULL
ALTER TABLE myOtherTable2 ALTER COLUMN col_id int NOT NULL
--Change referencing table values
UPDATE myOtherTable1 SET consignment_id = Target.id_new FROM myOtherTable1 AS Source JOIN <on key table>
UPDATE myOtherTable2 SET consignment_id = Target.id_new FROM myOtherTable2 AS Source JOIN <on key table>
--Replace old column with new column
ALTER TABLE myTable DROP COLUMN col_id
GO
EXEC sp_rename 'myTable.id_new', 'col_id', 'Column'
GO
--Reinstate any OTHER PKs disabled
ALTER TABLE myTable ADD CONSTRAINT <PK defn>
--Reinstate FKs
ALTER TABLE myOtherTable1 WITH CHECK ADD CONSTRAINT <constraintStuff>
ALTER TABLE myOtherTable2 WITH CHECK ADD CONSTRAINT <constraintStuff>
SELECT * FROM myTable
-- Reload out-of-date views
EXEC sp_refreshview 'someView'
-- Remove obsolete sequence
DROP SEQUENCE mySeq
ROLLBACK TRAN LabelledTransaction
Obviously that's all somewhat redacted, but the fine detail isn't the important thing in here.
Naturally, it's quite hard to locate all the things that need to be turned off/editted before the core change (even with some meta-queries to help me), so I don't always get the script correct first time.
But I put in the ROLLBACK in order to ensure that the failed attempts left the DB unchanged.
But what I actually see is that the ROLLBACK doesn't occur if there were errors in the TRAN. I think I get errors about "no matching TRAN for the rollback"?
My first instinct was that it was about the GO statements, but https://stackoverflow.com/a/11121382/1662268 suggests that labeling the TRAN should have fixed that?
What's happening? Why don't the changes get rolled back properly if there are errors.
How can I write and test these scripts in such a way that I don't have to manually revert any partial changes if the script isn't perfect first time?
EDIT:
Additional comments based on the first answer.
If the linked answer is not applicable to this query, could you expand on why that is, and why it's different from the example that they had given in their answer?
I can't (or rather, I believe that I can't) remove the GOs, because the script above requires the GOs in order to compile. If I remove the GOs then later statements that depend on the newly added/renamed columns don't compile. and the query can't run.
Is there any way to work around this, to remove the GOs?
If you have any error which automatically causes the transaction to be rolled back then the transaction will roll back as part of the current batch.
Then, control will return back to the client tool which will then send the next batch to the server and this next batch (and subsequent ones) will not be wrapped in any transaction.
Finally, when the final batch is executed that tries to run the rollback then you'll get the error message you received.
So, you need to protect each batch from running when its not protected by a transaction.
One way to do it would be to insert our old fried GOTO:
GO
IF ##TRANCOUNT=0 GOTO NBATCH
...Rest of Code
NBATCH:
GO
or SET FMTONLY:
GO
IF ##TRANCOUNT=0 BEGIN
SET FMTONLY ON
END
...Rest of Code
GO
Of course, this won't address all issues - some statements need to be the first or only statement in a batch. To resolve these, we have to combine one of the above techniques with an EXEC of some form:
GO
IF ##TRANCOUNT=0 BEGIN
SET FMTONLY ON
END
EXEC sp_executesql N'/*Code that needs to be in its own batch*/'
GO
(You'll also have to employ this technique if a batch of code relies on work a previous batch has performed which introduces new database objects (tables, columns, etc), since if that previous batch never executed, the new object will not exist)
I've also just discovered the existence of the -b option for the sqlcmd tool. The following script generates two errors when run through SSMS:
begin transaction
go
set xact_abort on
go
create table T(ID int not null,constraint CK_ID check (ID=4))
go
insert into T(ID) values (3)
go
rollback
Errors:
Msg 547, Level 16, State 0, Line 7
The INSERT statement conflicted with the CHECK constraint "CK_ID". The conflict occurred in database "TestDB", table "dbo.T", column 'ID'.
Msg 3903, Level 16, State 1, Line 9
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION.
However, the same script saved as Abortable.sql and run with the following commandline:
sqlcmd -b -E -i Abortable.sql -S .\SQL2014 -d TestDB
Generates the single error:
Msg 547, Level 16, State 1, Server .\SQL2014, Line 1
The INSERT statement conflicted with the CHECK constraint "CK_ID". The conflict
occurred in database "TestDB", table "dbo.T", column 'ID'.
So, it looks like running your scripts from the commandline and using the -b option may be another approach to take. I've just scoured the SSMS options/properties to see if I can find something equivalent to -b but I've not found it.
Remove the 'GO', that finishes the transaction
Only ROLLBACK if completes - just use TRY/CATCH:
BEGIN TRANSACTION;
BEGIN TRY
--Remove FKs
ALTER TABLE myOtherTable1 DROP CONSTRAINT <constraintStuff>
ALTER TABLE myOtherTable2 DROP CONSTRAINT <constraintStuff>
--Remove PK
ALTER TABLE myTable DROP CONSTRAINT PK_for_myTable
--Add replacement id column with new type and IDENTITY
ALTER TABLE myTable ADD id_new int Identity(1, 1) NOT NULL
ALTER TABLE myTable ADD CONSTRAINT PK_for_myTable PRIMARY KEY CLUSTERED (id_new)
SELECT * FROM myTable
--Change referencing table types
ALTER TABLE myOtherTable1 ALTER COLUMN col_id int NULL
ALTER TABLE myOtherTable2 ALTER COLUMN col_id int NOT NULL
--Change referencing table values
UPDATE myOtherTable1 SET consignment_id = Target.id_new FROM myOtherTable1 AS Source JOIN <on key table>
UPDATE myOtherTable2 SET consignment_id = Target.id_new FROM myOtherTable2 AS Source JOIN <on key table>
--Replace old column with new column
ALTER TABLE myTable DROP COLUMN col_id
EXEC sp_rename 'myTable.id_new', 'col_id', 'Column'
--Reinstate any OTHER PKs disabled
ALTER TABLE myTable ADD CONSTRAINT <PK defn>
--Reinstate FKs
ALTER TABLE myOtherTable1 WITH CHECK ADD CONSTRAINT <constraintStuff>
ALTER TABLE myOtherTable2 WITH CHECK ADD CONSTRAINT <constraintStuff>
SELECT * FROM myTable
-- Reload out-of-date views
EXEC sp_refreshview 'someView'
-- Remove obsolete sequence
DROP SEQUENCE mySeq
ROLLBACK TRANSACTION
END TRY
BEGIN CATCH
print 'Error caught'
select ERROR_NUMBER() AS ErrorNumber, ERROR_MESSAGE() AS ErrorMessage;
END CATCH;

Set auto id to primary key column [duplicate]

This question already has answers here:
How to add identity to the column in SQL Server?
(4 answers)
Closed 8 years ago.
I have a table and primary key is already set to that table and now I want that column to be autoincrement. Table has many records. Is it possible? or which one is fastest way to do that?
I think you have to make some effort for this as you cannot create identity column on existing column. However you may have a workaround for this like first try this to add a new column having identity field:
ALTER TABLE dbo.Table_name
ADD ID INT IDENTITY
and then make your ID as primary key like this:
ALTER TABLE dbo.Table_name
ADD CONSTRAINT PK_YourTable
PRIMARY KEY(ID)
And yes you have to remove the old dependencies before performing the above steps like this:
ALTER TABLE Table_name
DROP CONSTRAINT PK_Table1_Col1
EDIT:-
From the source:
We can use ALTER TABLE...SWITCH to work around this by only modifying metadata. See Books Online for restrictions on using the SWITCH method presented below. The process is practically instant even for the largest tables.
USE tempdb;
GO
-- A table with an identity column
CREATE TABLE dbo.Source (row_id INTEGER IDENTITY PRIMARY KEY NOT NULL, data SQL_VARIANT NULL);
GO
-- Some sample data
INSERT dbo.Source (data)
VALUES (CONVERT(SQL_VARIANT, 4)),
(CONVERT(SQL_VARIANT, 'X')),
(CONVERT(SQL_VARIANT, {d '2009-11-07'})),
(CONVERT(SQL_VARIANT, N'áéíóú'));
GO
-- Remove the identity property
BEGIN TRY;
-- All or nothing
BEGIN TRANSACTION;
-- A table with the same structure as the one with the identity column,
-- but without the identity property
CREATE TABLE dbo.Destination (row_id INTEGER PRIMARY KEY NOT NULL, data SQL_VARIANT NULL);
-- Metadata switch
ALTER TABLE dbo.Source SWITCH TO dbo.Destination;
-- Drop the old object, which now contains no data
DROP TABLE dbo.Source;
-- Rename the new object to make it look like the old one
EXECUTE sp_rename N'dbo.Destination', N'Source', 'OBJECT';
-- Success
COMMIT TRANSACTION;
END TRY
BEGIN CATCH
-- Bugger!
IF XACT_STATE() <> 0 ROLLBACK TRANSACTION;
PRINT ERROR_MESSAGE();
END CATCH;
GO
-- Test the the identity property has indeed gone
INSERT dbo.Source (row_id, data)
VALUES (5, CONVERT(SQL_VARIANT, N'This works!'))
SELECT row_id,
data
FROM dbo.Source;
GO
-- Tidy up
DROP TABLE dbo.Source;

Resources