PostgreSQL & Liquibase - default id value from sequence - database

I generated a changelog.xml from an existent database using Liquibase.
Liquibase 4.9.1
PostgreSQL 13.2
The original database was built using the below commands. One company table that has its primary key id defaulted to the my_company_id_seq sequence nextval
The Liquibase generate-changelog command created two change sets (company table & sequence) but the configured company default id, that used to point to the my_company_id_seq sequence, was lost, now it uses GENERATED BY DEFAULT AS IDENTITY.
I understand GENERATED BY DEFAULT AS IDENTITY is the recommended approach from PostgreSQl 10+ on instead of SERIAL, but not for my use case where sometimes I need to generate the ID in advance (calling nextValue from the sequence) and later pass it at time to insert the data into company table, and other times I can just let the id to be generated by the configured DEFAULT nextval('my_company_id_seq')
Do you know the rationale of it?
Or maybe there is no and it can be considered a bug?
Any insights and/or suggestions will be really appreciated
DDL from existent database:
CREATE SEQUENCE my_company_id_seq
INCREMENT BY 1
START WITH 1;
CREATE TABLE company (
id INT NOT NULL DEFAULT nextval('my_company_id_seq') PRIMARY KEY,
name VARCHAR NOT NULL
);
changelog.xml generated from existent database:
<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" xmlns:pro="http://www.liquibase.org/xml/ns/pro" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/pro http://www.liquibase.org/xml/ns/pro/liquibase-pro-4.6.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.6.xsd">
<changeSet author="liquibase (generated)" id="1656837328215-1">
<createTable tableName="company">
<column autoIncrement="true" name="id" type="INTEGER">
<constraints nullable="false" primaryKey="true" primaryKeyName="company_pkey"/>
</column>
<column name="name" type="VARCHAR">
<constraints nullable="false"/>
</column>
</createTable>
</changeSet>
<changeSet author="liquibase (generated)" id="1656837328215-2">
<createSequence cacheSize="1" cycle="false" dataType="bigint" incrementBy="1" maxValue="9223372036854775807" minValue="1" sequenceName="my_company_id_seq" startValue="1"/>
</changeSet>
</databaseChangeLog>
Generated sequence & table from Liquibase changelog:
CREATE SEQUENCE IF NOT EXISTS app.my_company_id_seq
INCREMENT 1
START 1
MINVALUE 1
MAXVALUE 9223372036854775807
CACHE 1;
CREATE TABLE IF NOT EXISTS app.company
(
id integer NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 2147483647 CACHE 1 ),
name character varying COLLATE pg_catalog."default" NOT NULL,
CONSTRAINT company_pkey PRIMARY KEY (id)
)

Related

How to fix creation of table with reserved keywork (key)?

I use liquibase to initilize my h2 database for unit testing, but there is a problem with one table using "key" as name of column.
I tried "_key" as name, it works but the database (which I cannot change its schema) use "key" as a column name, so i have to put #Column("\"key\"") in my domain class. This works for SQL Server (production database) but doesn't for h2 (in memory db for testing)
In Liquibase xml:( for unit test)
<column name='_key' type="varchar(100)">
<constraints nullable="true" />
</column>
My domain:
#Column(name = "_key", length = 100)
//#Column(name = "\"key\"", length = 100)
private String key;
So I have to uncomment the first option and comment the second to make the unit test works and do the opposite to make the application work.
Using the
#Column(name = "\"key\"", length = 100)
with
<column name="key" type="varchar(100)"> or
<column name="\"key\"" type="varchar(100)"> or
<column name='"key"' type="varchar(100)">
returns an error:
Schema-validation: missing column [key] in table [displaymessages]
and the query that creates the table is:
CREATE TABLE PUBLIC.displaymessages (
id BIGINT AUTO_INCREMENT NOT NULL,
"'key'" VARCHAR(100),
value VARCHAR(1000),
type INT NOT NULL,
createdon date NOT NULL,
modifiedon date,
isdeleted INT NOT NULL,
CONSTRAINT PK_DISPLAYMESSAGES PRIMARY KEY (id)
)
On the changeset that creates the table/column, you can add an attribute objectQuotingStrategy with the value QUOTE_ALL_OBJECTS, as shown in the example below:
<changeSet author="Sir Cuke Umber" id="1::addTable::labelA" labels="AAA" objectQuotingStrategy="QUOTE_ALL_OBJECTS">
<createTable tableName="LABELA">
<column name="message" type="TEXT">
<constraints nullable="true" primaryKey="false" unique="false"/>
</column>
</createTable>
</changeSet>
This attribute can also be added at the root changelog level if you want every identifier to be quoted.

Simple SQL Bulk Insert not working

I'm trying to create a simple Bulk Insert command to import a fixed width text file into a table. Once I have this working I'll then expand on it to get my more complex import working.
I'm currently receiving the error...
Msg 4866, Level 16, State 7, Line 1
The bulk load failed. The column is too long in the data file for row 1, column 1. Verify that the field terminator and row terminator are specified correctly.
Obviously I have checked the terminator in the file. For test data I just typed a 3 line text file in Notepad. At this stage I'm just trying to import one column per line. I have padded the first two lines so each one is 18 characters long.
Test.txt
This is line one
This is line two
This is line three
When I view the file in Notepad++ and turn on all characaters I see CRLF on the end of each line and no blank lines at the end of the file.
This is the SQL I'm using:
USE [Strata]
GO
drop table VJR_Bulk_Staging
Create Table [dbo].[VJR_Bulk_Staging](
[rowid] int Identity(1,1) Primary Key,
[raw] [varchar](18) not null)
GO
Bulk Insert [VJR_Bulk_Staging]
From 'c:\temp\aba\test.txt'
with (FormatFile='c:\temp\aba\test2.xml')
Here is the format XML file. I have tried several variations. This one was created using the BCP command.
bcp strata.dbo.vjr_bulk_staging format nul -f test2.xml -x -n -T -S Server\Instance
This created a record and a row entry for my rowid column which I thought was a problem as that is an identity field, so I removed it.
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="18" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="raw" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>
I'm testing on SQL Server 2008 R2 Express.
Any ideas where I'm going wrong?
I think the problem is with your prefix being 2 bytes long:
xsi:type="CharPrefix" PREFIX_LENGTH="2"
From what you have posted you don't have a prefix in your data file. Set the PREFIX_LENGTH to 0 in your format file, or provide the proper prefix in your data file.
You can find more information about prefix datatypes and what the prefix is about in the documentation: Specify Prefix Length in Data Files by Using bcp (SQL Server).
I think what you really wanted is type CharTerm with a proper TERMINATOR (/r/n in your case).
This works.
Option 1: Non-XML Format File
9.0
1
1 SQLCHAR 0 18 "\r\n" 2 raw SQL_Latin1_General_CP1_CI_AS
or simply
9.0
1
1 SQLCHAR "" "" "\r\n" 2 "" ""
Option 2: XML Format File
Ugly as hell work-around
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="\r" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\n"/>
</RECORD>
<ROW>
<COLUMN SOURCE="2" xsi:type="SQLINT"/>
<COLUMN SOURCE="1" xsi:type="SQLCHAR"/>
</ROW>
</BCPFORMAT>
P.s.
It seems to me that there is a bug in the design of the XML format file.
Unlike the Non-XML format file, there is no option to indicate the position of the loaded column (and the names are just for the clarity of the scripts, they have no real meanning).
The XML example in the documentation does not work
Use a Format File to Skip a Table Column (SQL Server)
Could you please try the following command and check if the BULK insert is happening.please note I have added the last line mentioning the delimiter.
USE [Strata]
GO
drop table VJR_Bulk_Staging
Create Table [dbo].[VJR_Bulk_Staging](
[rowid] int Identity(1,1) Primary Key,
[raw] [varchar](18) not null)
GO
Bulk Insert [VJR_Bulk_Staging]
From 'c:\temp\aba\test.txt'
WITH ( FIELDTERMINATOR ='\t', ROWTERMINATOR ='\n',FIRSTROW=1 )

Liquibase: SQL does not generate

I am working on learning Liquibase. I am working on trying to generate the SQL from the changelog that I have. For some reason, the only SQL it is generating is the locking for the DATABASECHANGELOGLOCK table.
ChangeLog
This is located in com/example/db/changelog
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">
<changeSet id="drop-tables" author="ascalonian">
<dropTable tableName="liquibase" />
</changeSet>
<changeSet id="create-tables" author="ascalonian">
<createTable tableName="liquibase">
<column name="id" type="NUMERIC(19,0)" autoIncrement="true">
<constraints primaryKey="true" nullable="false" />
</column>
<column name="firstname" type="VARCHAR(50)" />
<column name="lastname" type="VARCHAR(50)">
<constraints nullable="false" />
</column>
</createTable>
</changeSet>
<changeSet id="update-tables" author="ascalonian">
<addColumn tableName="liquibase">
<column name="username" type="VARCHAR(10)" />
</addColumn>
</changeSet>
</databaseChangeLog>
Command Line
java -jar liquibase.jar
--driver=net.sourceforge.jtds.jdbc.Driver
--classpath=/Users/<username>/.m2/repository/net/sourceforge/jtds/jtds/1.3.1/jtds-1.3.1.jar
--changeLogFile=com/example/db/changelog/db.changelog-master.xml
--url=jdbc:jtds:sqlserver://localhost:1433/TestDB
--username=username
--password=password
updateSQL
SQL Output
-- *********************************************************************
-- Update Database Script
-- *********************************************************************
-- Change Log: com/example/db/changelog/db.changelog-create.xml
-- Ran at: 8/11/15 4:28 PM
-- Against: username#jdbc:jtds:sqlserver://localhost:1433/TestDB
-- Liquibase version: 3.4.1
-- *********************************************************************
-- Lock Database
UPDATE [dbo].[DATABASECHANGELOGLOCK] SET [LOCKED] = 1, [LOCKEDBY] = 'fe80:0:0:0:541c:15ff:fe8f:7826%9 (fe80:0:0:0:541c:15ff:fe8f:7826%9)', [LOCKGRANTED] = '2015-08-11T16:28:18.090' WHERE [ID] = 1 AND [LOCKED] = 0
GO
-- Release Database Lock
UPDATE [dbo].[DATABASECHANGELOGLOCK] SET [LOCKED] = 0, [LOCKEDBY] = NULL, [LOCKGRANTED] = NULL WHERE [ID] = 1
GO
I was expecting the SQL for the drop, create and update table but don't see anything.
Liquibase: Version 3.4.1
Database: MS SQL Server
Cannot reproduce your issue. Have you already run the "update" command against your database? That would explain the lack of SQL (because the changes were already applied to the target DB).
The DATABASECHANGELOG table records which changesets have been applied. Checkout the rollbackSQL command, that will show you the undo actions

SQL Server BIT data type reports differently for View and Table query

I need to export data from SQL Server 2012 based on a view. While testing the export for a downstream system, I was manually extracting the data out of the table that the view is based on and the BIT data type columns were reporting as 1/0.
However, once I setup the view against the table, I noticed that the BIT data type columns reported as TRUE/FALSE. This happens whether I perform a select against the view or export from it.
Why does this happen and how can I maintain the same results in the view as the data table (1/0)?
The bit data type is interpreted by clients differently. SSMS, will report back a 1 or 0 for a bit while the same 1/0 is interpreted by an SSIS's Data Flow as True or False.
Whether the source is a table or a view makes no matter for SSIS unless you explicitly change the data type.
For setup, I created 2 tables and a view
CREATE TABLE dbo.BaseTable
(
SomeBit bit NOT NULL
, RowDescription varchar(50) NOT NULL
);
CREATE TABLE dbo.TargetTable
(
SomeBit bit NOT NULL
, RowDescription varchar(50) NOT NULL
, SourcePackage nvarchar(100) NOT NULL
);
GO
CREATE VIEW dbo.MyView
AS
SELECT
BT.SomeBit
, BT.RowDescription
FROM
dbo.BaseTable AS BT;
GO
INSERT INTO
dbo.BaseTable
(
SomeBit
, RowDescription
)
VALUES
(CAST(0 AS bit), 'Falsification')
, (CAST(1 AS bit), 'True dat');
GO
At this point, if I use SSMS and query either dbo.BaseTable or dbo.MyView, I will get back a 1 and 0. But again, these are just artifacts of presentation. In C, 0 is false and any numeric value that isn't 0 is true. Excel will present it as FALSE and TRUE. Every client will interpret the value into whatever the local representation of a boolean value is. SSIS chose True and False.
I built out a simple package that pulls data from BaseTable or MyView and writes it to a text file and a table.
The basic control flow looks thus
The data flow looks complex but it's not.
I select from either my table or view, add a description for my target table, use a multicast so I can send the same data to multiple destinations and then write to a file and table.
If I query SSMS for my sources and destinations, you'll see that the destination libraries handle the translation between the local and foreign representation of the data type.
There is no such translation available for a flat file because there's no "standard" for the representation of a boolean. I might like Y/N. Even so, the
I tried a number of things to coerce a 1/0 to be written to the flat file. I set my data types to
Boolean DT_BOOL
Single byte signed int DT_I1
Four byte signed int DT_I4
String DT_STR
but it never mattered (which actually seems odd given how persnickety SSIS is about data types) --- my output was always the same
False,Falsification
True,True dat
Ultimately, if I wanted a 0 or a 1 in that output file, I needed to change my data type: either in the source query with an explicit cast or through a Derived Column component using the ternary operator SomeBit ? (DT_I1)1 : (DT_I1)0. Use DT_I1/I2/I4/I8 as you see fit
Fun trivia note: if you chose to use the Data Conversion component you're going to get 0 for False, -1 for True or if you use a lazy cast in the Derived Component (DT_I1) SomeBit It seems they follow the C interpretation of boolean values.
Biml it
No need to take my word for it. Using the above table definitions and population of values, if you install the free addon BIDS Helper you can generate the same code for any version of SSIS.
After installing BIDS Helper, right click on an SSIS project and in the context menu, select Add Biml file. Replace the contents of that file with the below code; save and then right-click to generate a new package.
You will need to edit the values for the Flat File Connection to point to valid locations as well as point the ole db connection string to wherever you spin up your tables.
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<FlatFileConnection FilePath="C:\ssisdata\so_29244868.table.csv" FileFormat="FFF_table" Name="FF_Table" />
<FlatFileConnection FilePath="C:\ssisdata\so_29244868.view.csv" FileFormat="FFF_table" Name="FF_View" />
<OleDbConnection Name="CM_OLE" ConnectionString="Data Source=localhost\dev2014;Initial Catalog=tempdb;Provider=SQLNCLI11.0;Integrated Security=SSPI;" />
</Connections>
<FileFormats>
<FlatFileFormat
Name="FFF_table" IsUnicode="false" CodePage="1252"
FlatFileType="RaggedRight">
<Columns>
<Column Name="SomeBit" DataType="Boolean" Delimiter="," />
<Column Name="RowDescription" DataType="AnsiString" Length="50" Delimiter="CRLF"/>
</Columns>
</FlatFileFormat>
</FileFormats>
<Packages>
<Package ConstraintMode="Parallel" Name="so_29244868">
<Tasks>
<Dataflow Name="DFT Table example">
<Transformations>
<OleDbSource ConnectionName="CM_OLE" Name="OLE_SRC dbo_BaseTable">
<ExternalTableInput Table="dbo.BaseTable" />
</OleDbSource>
<DerivedColumns Name="DER Package name">
<Columns>
<Column DataType="String" Name="SourcePackage" Length="100">"DFT Table example"</Column>
</Columns>
</DerivedColumns>
<Multicast Name="MC Dupe">
<OutputPaths>
<OutputPath Name="FF" />
<OutputPath Name="Table" />
</OutputPaths>
</Multicast>
<FlatFileDestination ConnectionName="FF_Table" Name="FF_DST table">
<InputPath OutputPathName="MC Dupe.FF" />
</FlatFileDestination>
<OleDbDestination
ConnectionName="CM_OLE"
Name="OLE_DST Table"
TableLock="false">
<InputPath OutputPathName="MC Dupe.Table" />
<ExternalTableOutput Table="[dbo].[TargetTable]"></ExternalTableOutput>
</OleDbDestination>
</Transformations>
</Dataflow>
<Dataflow Name="DFT View example">
<Transformations>
<OleDbSource ConnectionName="CM_OLE" Name="OLE_SRC dbo_MyView">
<ExternalTableInput Table="dbo.MyView" />
</OleDbSource>
<DerivedColumns Name="DER Package name">
<Columns>
<Column DataType="String" Name="SourcePackage" Length="100">"DFT View example"</Column>
</Columns>
</DerivedColumns>
<Multicast Name="MC Dupe">
<OutputPaths>
<OutputPath Name="FF" />
<OutputPath Name="Table" />
</OutputPaths>
</Multicast>
<FlatFileDestination ConnectionName="FF_View" Name="FF_DST view">
<InputPath OutputPathName="MC Dupe.FF" />
</FlatFileDestination>
<OleDbDestination
ConnectionName="CM_OLE"
Name="OLE_DST view"
TableLock="false"
>
<InputPath OutputPathName="MC Dupe.Table" />
<ExternalTableOutput Table="[dbo].[TargetTable]"></ExternalTableOutput>
</OleDbDestination>
</Transformations>
</Dataflow>
</Tasks>
</Package>
</Packages>
</Biml>
I've run into the same problem using Entity Framework.
Try casting the bit field to a bit.

Is there a way to impose a limit on the XML-type of a typed XML column in SQL Server?

I have two XML Schemas where one depends on the other, and I'd like to create a typed XML column in SQL Server where the type is defined by one of my schemas (actually I have hundreds of schemas but in principle it's the same). When I create the column it accepts all the types defined in my entire schema collection, and I'd really like to limit the type for one column to just one of the types.
<schema targetnamespace="b">
<simpleType name="b">
<restriction base="string"><pattern value="(0[1-9])"/></restriction>
</simpleType>
</schema>
<schema targetnamespace="a" xmlns:b="b">
<element name="b" type="b:b" />
<complexType name="aType">
<sequence>
<element ref="b"/>
</sequence>
</complexType>
<element name="a" type="aType"/>
</schema>
So first I create the schema collection with
CREATE XML SCHEMA COLLECTION Foo AS N'<above schemas>'
and then I create the column with
ALTER TABLE FooTable ADD fooCol [xml] (Document [Foo])
Now I can insert values into the column of XML types "a" as well as "b", but I'd really like to limit the column type to just "a".
Is there any way of achieving that, short of hand-writing customized schema collections for each different type?
Updated:
Regarding the check constraint suggested - according to http://blogs.msdn.com/b/denisruc/archive/2006/08/22/713342.aspx
the check constraint would then be something along the lines of
CREATE FUNCTION dbo.checkTopElmntIsTypeA(#x XML(MySchemaCollection))
RETURNS bit
AS
BEGIN
RETURN ~(#x.exist('/a:a'))
END
go
-and the table definition to limit content of a column to XML type a
CREATE TABLE T(xmlCol XML(MySchemaCollection)
CHECK (1 = dbo.checkTopElmntIsTypeA(xmlCol)))
go
I guess that could do, even if it's not quite as nice. :-)

Resources