Make Postgres choose the next minimal available id - database

I would like to make Postgres choose the first next available id so that no error occurs in the following case:
CREATE TABLE test(
id serial PRIMARY KEY,
name varchar
);
Then:
INSERT INTO test VALUES (2,'dd');
INSERT INTO test (name) VALUES ('aa');
INSERT INTO test (name) VALUES ('bb');
This will give a constraint error since id is primary.
How can I tell Postgres to insert the record with the next free id?

Generally it's best to never overrule the default in a serial column. If you sometimes need to provide id values manually, replace the standard DEFAULT clause nextval('sequence_name') of the serial column with a custom function that omits existing values.
Based on this dummy table:
CREATE TABLE test (test_id serial PRIMARY KEY, test text);
Function:
CREATE OR REPLACE FUNCTION f_test_test_id_seq(OUT nextfree bigint) AS
$func$
BEGIN
LOOP
SELECT INTO nextfree val
FROM nextval('test_test_id_seq'::regclass) val -- use actual name of sequence
WHERE NOT EXISTS (SELECT 1 FROM test WHERE test_id = val);
EXIT WHEN FOUND;
END LOOP;
END
$func$ LANGUAGE plpgsql;
Alter default:
ALTER TABLE test ALTER COLUMN test_id SET DEFAULT f_test_test_id_seq();
It's not strictly a serial any more, but serial is only a convenience feature anyway:
Safely and cleanly rename tables that use serial primary key columns in Postgres?
And if you build this on top of a serial column the SEQUENCE is automatically "owned" by the table column, which is probably a good thing.
This is a slightly faster variant of:
Autoincrement, but omit existing values in the column
Table and sequence name are hard coded here. You could easily parametrize the sequence name (like in the linked answer) and even the table name - and test existence with a dynamic statement using EXECUTE. Would give you a generic function, but the call would be a bit more expensive.
CREATE OR REPLACE FUNCTION f_nextfree(_tbl regclass
, _col text
, _seq regclass
, OUT nextfree bigint) AS
$func$
BEGIN
LOOP
EXECUTE '
SELECT val FROM nextval($1) val WHERE NOT EXISTS (
SELECT 1 FROM ' || _tbl || ' WHERE ' || quote_ident(_col) || ' = val)'
INTO nextfree
USING _seq;
EXIT WHEN nextfree IS NOT NULL;
END LOOP;
END
$func$ LANGUAGE plpgsql;
ALTER TABLE test2 ALTER COLUMN test2_id
SET DEFAULT f_nextfree('test2', 'test2_id', 'test2_test2_id_seq');
SQL Fiddle.

Related

What does 'Unnecessary table scan' mean exactly

I have a table of names with Ids
Create table Names (
Id int,
Name nvarchar(500)
)
I'm trying to create a procedure that would select 1 name that matches the provided Id if Id is provided or select all names if no Id is provided
Create Procedure SelectNames
#Id int = null
AS
BEGIN
Select * From Names
Where IsNull(#Id, 0) = 0
Or Id = #Id
END
GO
But I get an error: 'Error: SR0015 : Microsoft.Rules.Data : Deterministic function call (ISNULL) might cause an unnecessary table scan.'
What does the 'unnecessary table scan' refer to in this instance?
And is there a better way to write the procedure?
The simplest way to remove the table scan is to create an index (probably unique) on your Id column. In general, one wouldn't expect a nullable Id value. With that index in place, finding a name by Id will not require scanning (or iterating through every row in) the table.
Regarding "better way to write the procedure" - once the nullability is removed, a simple SELECT without the WHERE should be fine.

postgres 9.3+: constraint to ensure table and column exist in database

How can I ensure a new record contains values that refer to a schema, table, and column that currently exist in the database?
For example, given a table:
CREATE TEMP TABLE "column_reference" (
"gid" SERIAL PRIMARY KEY
, "val" INTEGER
, "schema" TEXT
, "table" TEXT
, "column" TEXT
);
how can I ensure schema.table.column exists?
I tried a fkey to information_schema.columns, but, of course, foreign keys to views are disallowed.
It also appears from the columns view definition that I need several tables in order to get the schema, table, and column names so I can't create a single foreign key to the source tables.
My current workaround is to manually create a __columns table from the information_schema.columns view and reference it instead. This works given the control I happen to have on this project at this point in time, but am looking for a permanent, dynamic solution.
Is there a different constraint or method I could use?
You can create a trigger function that checks what you want, and associate this function with a trigger which is fired BEFORE an INSERT or an UPDATE of the table:
This could be your trigger function:
CREATE FUNCTION column_reference_check()
RETURNS trigger
LANGUAGE 'plpgsql'
AS
$BODY$
begin
/* Check for the existence of the required column */
if EXISTS (
SELECT *
FROM information_schema.columns
WHERE
table_schema = new.schema
AND table_name = new.table
AND column_name = new.column )
then
/* Everything Ok */
return new ;
else
/* This is approx. what would happen if you had a constraint */
RAISE EXCEPTION 'Trying to insert non-matching (%, %, %)', new.schema, new.table, new.column ;
/* As an alternative, you could also just return NULL
As a result, the row is *not* inserted, but execution continues */
return NULL ;
end if ;
end ;
$BODY$;
To associate this function with a trigger, you'd use:
CREATE TRIGGER column_reference_check_trg
BEFORE INSERT OR UPDATE OF "schema", "table", "column"
ON column_reference
FOR EACH ROW
EXECUTE PROCEDURE column_reference_check();
Now you can try to perform the following INSERT, that should succeed:
INSERT INTO column_reference
VALUES (2, 1, 'pg_catalog', 'pg_statistic', 'starelid');
But if you try this one:
INSERT INTO column_reference
VALUES (-1, 1, 'false_schema', 'false_table', 'false_column');
... you get an exception:
ERROR: Trying to insert non-matching (false_schema, false_table, false_column)
CONTEXT: PL/pgSQL function column_reference_check() line 16 at RAISE

Create Unique ID in Stored Procedure to Match Legacy Data

I'm creating CRUD procedures that duplicate a legacy program that generates a unique ID based on a 'Next ID' field in a separate table. Rather than duplicate the use of a separate table I have written a stored procedure that reads the number of rows in the table.
CREATE PROCEDURE [TLA_CreateItem]
#SiteReference varchar(50)
,#ItemID varchar(4)
,#NewUniqueID varchar(68) OUTPUT
AS
BEGIN
DECLARE #Rows varchar(12)
SET #Rows = (CONVERT(varchar(12), (SELECT Count(UniqueID) FROM [TLA_Items]) + 1))
SET #NewUniqueID = #ItemID + #SiteReference + #Rows
INSERT INTO [TLA_Items] ([ItemID], [UniqueID])
VALUES (#ItemID, #NewUniqueID)
SELECT #NewUniqueID
END
I've simplified the code above but what's not shown is that the TLA_Items table also has an IDENTITY column and that it needs to work with SQL Server 2008.
The UniqueID field has to match the pattern of the legacy program: ItemID + SiteReference + (integer representing number of previous records)
However when testing this I've found a flaw in my logic. If rows are deleted then it's possible to create a unique Id which matches an existing row. This doesn't happen in the legacy system as rows are rarely deleted and the separate table stores the next number in the sequence.
Other than store the next ID value in a separate table, is there a better technique, to create a unique ID that matches the legacy pattern?
You could have your procedure store only the prefix (#ItemID + #SiteReference) into UniqueID and use a FOR INSERT trigger to append the IDENTITY value as the rows component immediately after the row is inserted, something like this:
CREATE TRIGGER TLA_Items_Adjust
ON dbo.TLA_Items
FOR INSERT
AS
BEGIN
UPDATE t
SET t.UniqueID = i.UniqueID + CAST(t.IdentityColumn AS varchar(10))
FROM dbo.TLA_Items AS t
INNER JOIN inserted AS i
ON t.IdentityColumn = i.IdentityColumn
;
END
To read and return the newly generated UniqueID value as the OUTPUT parameter as well as a row, you could use a table variable and the OUTPUT clause in the INSERT statement, like this:
CREATE PROCEDURE [TLA_CreateItem]
#SiteReference varchar(50)
,#ItemID varchar(4)
,#NewUniqueID varchar(68) OUTPUT
AS
BEGIN
DECLARE #GeneratedUniqueID TABLE (UniqueID varchar(68));
INSERT INTO dbo.[TLA_Items] ([ItemID], [UniqueID])
OUTPUT inserted.UniqueID INTO #GeneratedUniqueID (UniqueID)
VALUES (#ItemID, #ItemID + #SiteReference);
SELECT #NewUniqueID = UniqueID FROM #GeneratedUniqueID;
SELECT #NewUniqueID;
END
Although instead of using OUTPUT you could probably just read the value from the row matching the SCOPE_IDENTITY() result:
CREATE PROCEDURE [TLA_CreateItem]
#SiteReference varchar(50)
,#ItemID varchar(4)
,#NewUniqueID varchar(68) OUTPUT
AS
BEGIN
INSERT INTO dbo.[TLA_Items] ([ItemID], [UniqueID])
VALUES (#ItemID, #ItemID + #SiteReference);
SELECT #NewUniqueID = UniqueID
FROM dbo.TLA_Items
WHERE IdentityColumn = SCOPE_IDENTITY();
SELECT #NewUniqueID;
END
Here is another option, but please bear in mind that it would affect existing UniqueID values.
If you can afford a slight change to the table schema, you could add a column called something like UniqueIDPrefix:
ALTER TABLE dbo.TLA_Items
ADD UniqueIDPrefix varchar(56) NOT NULL;
and redefine the UniqueID column to be a computed column:
ALTER TABLE dbo.TLA_Items
DROP COLUMN UniqueID;
GO
ALTER TABLE dbo.TLA_Items
ADD UniqueID AS UniqueIDPrefix + CAST(IdentiyColumn AS varchar(12));
In your stored procedure, you would then need to populate UniqueIDPrefix instead of UniqueID (with just the result of #ItemID + #SiteReference)
INSERT INTO dbo.[TLA_Items] ([ItemID], [UniqueIDPrefix])
VALUES (#ItemID, #ItemID + #SiteReference);
and read the value of UniqueID using either OUTPUT or SCOPE_IDENTITY(), as in my other answer.
It sounds like you are on SQL 2008, but if you were on 2012, you could use a sequence to store an incrementing value.
How about never delete? You could add a flag to the table for logical deletes.

TSQL ID generation

I have a question regarding locking in TSQL. Suppose I have a the following table:
A(int id, varchar name)
where id is the primary key, but is NOT an identity column.
I want to use the following pseudocode to insert a value into this table:
lock (A)
uniqueID = GenerateUniqueID()
insert into A values (uniqueID, somename)
unlock(A)
How can this be accomplished in terms of T-SQL? The computation of the next id should be done with the table A locked in order to avoid other sessions to do the same operation at the same time and get the same id.
If you have custom logic that you want to apply in generating the ids, wrap it up into a user defined function, and then use the user defined function as the default for the column. This should reduce concurrency issue similarly to the provided id generators by deferring the generation to the point of insert and piggy backing on the insert locking behavior.
create table ids (id int, somval varchar(20))
Go
Create function GenerateUniqueID()
returns int as
Begin
declare #ret int
select #ret = max(isnull(id,1)) * 2 from ids
if #ret is null set #ret = 2
return #ret
End
go
alter table ids add Constraint DF_IDS Default(dbo.GenerateUniqueID()) for Id
There are really only three ways to go about this.
Change the ID column to be an IDENTITY column where it auto increments by some value on each insert.
Change the ID column to be a GUID with a default constraint of NEWID() or NEWSEQUENTIALID(). Then you can insert your own value or let the table generate one for you on each insert.
On each insert, start a transaction. Then get the next available ID using something like select max(id)+1 . Do this in a single sql statement if possible in order to limit the possibility of a collision.
On the whole, most people prefer option 1. It's fast, easy to implement, and most people understand it.
I tend to go with option 2 with the apps I work on simply because we tend to scale out (and up) our databases. This means we routinely have apps with a multi-master situation. Be aware that using GUIDs as primary keys can mean your indexes are routinely trashed.
I'd stay away from option 3 unless you just don't have a choice. In which case I'd look at how the datamodel is structured anyway because there's bound to be something wrong.
You use the NEWID() function and you do not need any locking mechanism
You tell a column to be IDENTITY and you do not need any locking mechanism
If you generate these IDs manually and there is a chance parallel calls could generate the same IDs then something like this:
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
#NextID = GenerateUniqueID()
WHILE EXISTS (SELECT ID FROM A WHERE ID = #NextID)
BEGIN
#NextID = GenerateUniqueID()
END
INSERT INTO A (ID, Text) VALUES (#NextID , 'content')
COMMIT TRANSACTION
#Markus, you should look at using either IDENTITY or NEWID() as noted in the other answers. if you absolutely can't, here's an option for you...
DECLARE #NewID INT
BEGIN TRAN
SELECT #NewID = MAX(ID) + 1
FROM TableA (tablockx)
INSERT TableA
(ID, OtherFields)
VALUES (#NewID, OtherFields)
COMMIT TRAN
If you're using SQL2005+, you can use the OUTPUT clause to do what you're asking, without any kind of lock (The table Test1 simulates the table you're inserted into, and since OUTPUT requires a temp table and not a variable to hold the results, #Result will do that):
create table test1( test INT)
create table #result (LastValue INT)
insert into test1
output INSERTED.test into #result(test)
select GenerateUniqueID()
select LastValue from #result
Just to update an old post. It is now possible with SQL Server 2012 to use a feature called Sequence. Sequences are created in much the same way a function and it is possible to specify the range, direction(asc, desc) and rollover point. After which it's possible to invoke the NEXT VALUE FOR method to generate the next value in the range.
See the following documentation from Microsoft.
http://technet.microsoft.com/en-us/library/ff878091.aspx

SQL: How to get the id of values I just INSERTed?

I inserted some values into a table. There is a column whose value is auto-generated. In the next statement of my code, I want to retrieve this value.
Can you tell me how to do it the right way?
##IDENTITY is not scope safe and will get you back the id from another table if you have an insert trigger on the original table, always use SCOPE_IDENTITY()
This is how I do my store procedures for MSSQL with an autogenerated ID.
CREATE PROCEDURE [dbo].[InsertProducts]
#id INT = NULL OUT,
#name VARCHAR(150) = NULL,
#desc VARCHAR(250) = NULL
AS
INSERT INTO dbo.Products
(Name,
Description)
VALUES
(#name,
#desc)
SET #id = SCOPE_IDENTITY();
This works very nicely in SQL 2005:
DECLARE #inserted_ids TABLE ([id] INT);
INSERT INTO [dbo].[some_table] ([col1],[col2],[col3],[col4],[col5],[col6])
OUTPUT INSERTED.[id] INTO #inserted_ids
VALUES (#col1,#col2,#col3,#col4,#col5,#col6)
It has the benefit of returning all the IDs if your INSERT statement inserts multiple rows.
If your using PHP and MySQL you can use the mysql_insert_id() function which will tell you the ID of item you Just instered.
But without your Language and DBMS I'm just shooting in the dark here.
Again no language agnostic response, but in Java it goes like this:
Connection conn = Database.getCurrent().getConnection();
PreparedStatement ps = conn.prepareStatement(insertSql, Statement.RETURN_GENERATED_KEYS);
try {
ps.executeUpdate();
ResultSet rs = ps.getGeneratedKeys();
rs.next();
long primaryKey = rs.getLong(1);
} finally {
ps.close();
}
If you are working with Oracle:
Inset into Table (Fields....) values (Values...) RETURNING (List of Fields...) INTO (variables...)
example:
INSERT INTO PERSON (NAME) VALUES ('JACK') RETURNING ID_PERSON INTO vIdPerson
or if you are calling from... Java with a CallableStatement (sry, it's my field)
INSERT INTO PERSON (NAME) VALUES ('JACK') RETURNING ID_PERSON INTO ?
and declaring an autput parameter for the statement
There's no standard way to do it (just as there is no standard way to create auto-incrementing IDs). Here are two ways to do it in PostgreSQL. Assume this is your table:
CREATE TABLE mytable (
id SERIAL PRIMARY KEY,
lastname VARCHAR NOT NULL,
firstname VARCHAR
);
You can do it in two statements as long as they're consecutive statements in the same connection (this will be safe in PHP with connection pooling because PHP doesn't give the connection back to the pool until your script is done):
INSERT INTO mytable (lastname, firstname) VALUES ('Washington', 'George');
SELECT lastval();
lastval() gives you the last auto-generated sequence value used in the current connection.
The other way is to use PostgreSQL's RETURNING clause on the INSERT statement:
INSERT INTO mytable (lastname) VALUES ('Cher') RETURNING id;
This form returns a result set just like a SELECT statement, and is also handy for returning any kind of calculated default value.
An important note is that using vendor SQL queries to retrieve the last inserted ID are safe to use without fearing about concurrent connections.
I always thought that you had to create a transaction in order to INSERT a line and then SELECT the last inserted ID in order to avoid retrieving an ID inserted by another client.
But these vendor specific queries always retrieve the last inserted ID for the current connection to the database. It means that the last inserted ID cannot be affected by other client insertions as long as they use their own database connection.
For SQL 2005:
Assuming the following table definition:
CREATE TABLE [dbo].[Test](
[ID] [int] IDENTITY(1,1) NOT NULL,
[somevalue] [nchar](10) NULL,
)
You can use the following:
INSERT INTO Test(somevalue)
OUTPUT INSERTED.ID
VALUES('asdfasdf')
Which will return the value of the ID column.
From the site i found out the following things:
SQL SERVER – ##IDENTITY vs SCOPE_IDENTITY() vs IDENT_CURRENT – Retrieve Last Inserted Identity of Record
March 25, 2007 by pinaldave
SELECT ##IDENTITY
It returns the last IDENTITY value produced on a connection, regardless of the table that produced the value, and regardless of the scope of the statement that produced the value.
##IDENTITY will return the last identity value entered into a table in your current session. While ##IDENTITY is limited to the current session, it is not limited to the current scope. If you have a trigger on a table that causes an identity to be created in another table, you will get the identity that was created last, even if it was the trigger that created it.
SELECT SCOPE_IDENTITY()
It returns the last IDENTITY value produced on a connection and by a statement in the same scope, regardless of the table that produced the value.
SCOPE_IDENTITY(), like ##IDENTITY, will return the last identity value created in the current session, but it will also limit it to your current scope as well. In other words, it will return the last identity value that you explicitly created, rather than any identity that was created by a trigger or a user defined function.
SELECT IDENT_CURRENT(‘tablename’)
It returns the last IDENTITY value produced in a table, regardless of the connection that created the value, and regardless of the scope of the statement that produced the value.
IDENT_CURRENT is not limited by scope and session; it is limited to a specified table. IDENT_CURRENT returns the identity value generated for a specific table in any session and any scope.
Remember that ##IDENTITY returns the most recently created identity for your current connection, not necessarily the identity for the recently added row in a table. You should always use SCOPE_IDENTITY() to return the identity of the recently added row.
What database are you using? As far as I'm aware, there is no database agnostic method for doing this.
This is how I've done it using parameterized commands.
MSSQL
INSERT INTO MyTable (Field1, Field2) VALUES (#Value1, #Value2);
SELECT SCOPE_IDENTITY();
MySQL
INSERT INTO MyTable (Field1, Field2) VALUES (?Value1, ?Value2);
SELECT LAST_INSERT_ID();
sql = "INSERT INTO MyTable (Name) VALUES (#Name);" +
"SELECT CAST(scope_identity() AS int)";
SqlCommand cmd = new SqlCommand(sql, conn);
int newId = (int)cmd.ExecuteScalar();
Ms SQL Server: this is good solution even if you inserting more rows:
Declare #tblInsertedId table (Id int not null)
INSERT INTO Test ([Title], [Text])
OUTPUT inserted.Id INTO #tblInsertedId (Id)
SELECT [Title], [Text] FROM AnotherTable
select Id from #tblInsertedId
Rob's answer would be the most vendor-agnostic, but if you're using MySQL the safer and correct choise would be the built-in LAST_INSERT_ID() function.
SELECT ##Scope_Identity as Id
There is also ##identity, but if you have a trigger, it will return the results of something that happened during the trigger, where scope_identity respects your scope.
insert the row with a known guid.
fetch the autoId-field with this guid.
This should work with any kind of database.
An Environment Based Oracle Solution:
CREATE OR REPLACE PACKAGE LAST
AS
ID NUMBER;
FUNCTION IDENT RETURN NUMBER;
END;
/
CREATE OR REPLACE PACKAGE BODY LAST
AS
FUNCTION IDENT RETURN NUMBER IS
BEGIN
RETURN ID;
END;
END;
/
CREATE TABLE Test (
TestID INTEGER ,
Field1 int,
Field2 int
)
CREATE SEQUENCE Test_seq
/
CREATE OR REPLACE TRIGGER Test_itrig
BEFORE INSERT ON Test
FOR EACH ROW
DECLARE
seq_val number;
BEGIN
IF :new.TestID IS NULL THEN
SELECT Test_seq.nextval INTO seq_val FROM DUAL;
:new.TestID := seq_val;
Last.ID := seq_val;
END IF;
END;
/
To get next identity value:
SELECT LAST.IDENT FROM DUAL
In TransactSQL, you can use OUTPUT clause to achieve that.
INSERT INTO my_table(col1,col2,col3) OUTPUT INSERTED.id VALUES('col1Value','col2Value','col3Value')
FRI: http://msdn.microsoft.com/en-us/library/ms177564.aspx
Simplest answer:
command.ExecuteScalar()
by default returns the first column
Return Value
Type: System.Object
The first column of the first row in the result set, or a null reference (Nothing in Visual Basic) if the result set is empty. Returns a maximum of 2033 characters.
Copied from MSDN

Resources