I am working on learning Liquibase. I am working on trying to generate the SQL from the changelog that I have. For some reason, the only SQL it is generating is the locking for the DATABASECHANGELOGLOCK table.
ChangeLog
This is located in com/example/db/changelog
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">
<changeSet id="drop-tables" author="ascalonian">
<dropTable tableName="liquibase" />
</changeSet>
<changeSet id="create-tables" author="ascalonian">
<createTable tableName="liquibase">
<column name="id" type="NUMERIC(19,0)" autoIncrement="true">
<constraints primaryKey="true" nullable="false" />
</column>
<column name="firstname" type="VARCHAR(50)" />
<column name="lastname" type="VARCHAR(50)">
<constraints nullable="false" />
</column>
</createTable>
</changeSet>
<changeSet id="update-tables" author="ascalonian">
<addColumn tableName="liquibase">
<column name="username" type="VARCHAR(10)" />
</addColumn>
</changeSet>
</databaseChangeLog>
Command Line
java -jar liquibase.jar
--driver=net.sourceforge.jtds.jdbc.Driver
--classpath=/Users/<username>/.m2/repository/net/sourceforge/jtds/jtds/1.3.1/jtds-1.3.1.jar
--changeLogFile=com/example/db/changelog/db.changelog-master.xml
--url=jdbc:jtds:sqlserver://localhost:1433/TestDB
--username=username
--password=password
updateSQL
SQL Output
-- *********************************************************************
-- Update Database Script
-- *********************************************************************
-- Change Log: com/example/db/changelog/db.changelog-create.xml
-- Ran at: 8/11/15 4:28 PM
-- Against: username#jdbc:jtds:sqlserver://localhost:1433/TestDB
-- Liquibase version: 3.4.1
-- *********************************************************************
-- Lock Database
UPDATE [dbo].[DATABASECHANGELOGLOCK] SET [LOCKED] = 1, [LOCKEDBY] = 'fe80:0:0:0:541c:15ff:fe8f:7826%9 (fe80:0:0:0:541c:15ff:fe8f:7826%9)', [LOCKGRANTED] = '2015-08-11T16:28:18.090' WHERE [ID] = 1 AND [LOCKED] = 0
GO
-- Release Database Lock
UPDATE [dbo].[DATABASECHANGELOGLOCK] SET [LOCKED] = 0, [LOCKEDBY] = NULL, [LOCKGRANTED] = NULL WHERE [ID] = 1
GO
I was expecting the SQL for the drop, create and update table but don't see anything.
Liquibase: Version 3.4.1
Database: MS SQL Server
Cannot reproduce your issue. Have you already run the "update" command against your database? That would explain the lack of SQL (because the changes were already applied to the target DB).
The DATABASECHANGELOG table records which changesets have been applied. Checkout the rollbackSQL command, that will show you the undo actions
Related
I generated a changelog.xml from an existent database using Liquibase.
Liquibase 4.9.1
PostgreSQL 13.2
The original database was built using the below commands. One company table that has its primary key id defaulted to the my_company_id_seq sequence nextval
The Liquibase generate-changelog command created two change sets (company table & sequence) but the configured company default id, that used to point to the my_company_id_seq sequence, was lost, now it uses GENERATED BY DEFAULT AS IDENTITY.
I understand GENERATED BY DEFAULT AS IDENTITY is the recommended approach from PostgreSQl 10+ on instead of SERIAL, but not for my use case where sometimes I need to generate the ID in advance (calling nextValue from the sequence) and later pass it at time to insert the data into company table, and other times I can just let the id to be generated by the configured DEFAULT nextval('my_company_id_seq')
Do you know the rationale of it?
Or maybe there is no and it can be considered a bug?
Any insights and/or suggestions will be really appreciated
DDL from existent database:
CREATE SEQUENCE my_company_id_seq
INCREMENT BY 1
START WITH 1;
CREATE TABLE company (
id INT NOT NULL DEFAULT nextval('my_company_id_seq') PRIMARY KEY,
name VARCHAR NOT NULL
);
changelog.xml generated from existent database:
<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" xmlns:pro="http://www.liquibase.org/xml/ns/pro" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/pro http://www.liquibase.org/xml/ns/pro/liquibase-pro-4.6.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.6.xsd">
<changeSet author="liquibase (generated)" id="1656837328215-1">
<createTable tableName="company">
<column autoIncrement="true" name="id" type="INTEGER">
<constraints nullable="false" primaryKey="true" primaryKeyName="company_pkey"/>
</column>
<column name="name" type="VARCHAR">
<constraints nullable="false"/>
</column>
</createTable>
</changeSet>
<changeSet author="liquibase (generated)" id="1656837328215-2">
<createSequence cacheSize="1" cycle="false" dataType="bigint" incrementBy="1" maxValue="9223372036854775807" minValue="1" sequenceName="my_company_id_seq" startValue="1"/>
</changeSet>
</databaseChangeLog>
Generated sequence & table from Liquibase changelog:
CREATE SEQUENCE IF NOT EXISTS app.my_company_id_seq
INCREMENT 1
START 1
MINVALUE 1
MAXVALUE 9223372036854775807
CACHE 1;
CREATE TABLE IF NOT EXISTS app.company
(
id integer NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 2147483647 CACHE 1 ),
name character varying COLLATE pg_catalog."default" NOT NULL,
CONSTRAINT company_pkey PRIMARY KEY (id)
)
I am having a problem with using bulk insert. The issue is that the source files (tab delimited) that I'm dealing with contain rows that end in cr/lf without filling in values of the empty columns with tab for the rest of the row. So when the data is pulled into SQL Server, it's combining those shortened lines into the previous line. so basically it's combining multiple rows into one rather than writing it as two separate rows with nulls at the end of the first row.
Example to illustrate the problem: sample .txt file
column1 column2 column3 column4 column5
1 2 3 4 5
2 5 4 6
4 4 6 4
4 5 6 4 6
SQL to create table and bulk insert
CREATE TABLE test (
[column1] varchar(MAX) NULL,
[column2] varchar(MAX) NULL,
[column3] varchar(MAX) NULL,
[column4] varchar(MAX) NULL,
[column5] varchar(MAX) NULL
)
BULK INSERT test
FROM 'c:\temp\testimport.txt'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = '\t',
ROWTERMINATOR = '\r'
);
The really strange thing is that I can use the data import wizard and it imports the data perfectly, without any issue, and handles the lack of tabs for the columns just fine. But I don't know what the wizard is doing behind the scenes to make this happen. I would love to have the code it uses to create the table and do the insert as that would probably answer my question for me. At the end of the day I can't use the wizard as this will eventually be part of an automated task I'll be running against an SQL Server Express database on multiple files with different names but the same column header.
Maybe bulk insert isn't the way to go here? Or there is something obvious I'm missing that someone else might know off the top of their head. Either way, all help is appreciated and thanks in advance.
As Tim H suggested I've made a few attempts at creating a format file to accommodate the data. Results so far are as follows.
Using
bcp temp.dbo.test format nul -x -f test_format.xml -n -T
produces
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="2" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="column1" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="2" NAME="column2" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="3" NAME="column3" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="4" NAME="column4" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="5" NAME="column5" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>
Using this temp file as is produces......
Msg 4866, Level 16, State 7, Line 31
The bulk load failed. The column is too long in the data file for row 1, column 1. Verify that the field terminator and row terminator are specified correctly.
My attempt to edit the XML to work.....
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="column1" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="2" NAME="column2" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="3" NAME="column3" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="4" NAME="column4" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="5" NAME="column5" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>
Which does insert the data, but unfortunately still produces the same jumbled insert with overlapping lines in the same row.
Do you have control over the source files? If not, are the width of each column a fixed width or variable width? I know your create table example uses varchar(max)'s. The bulk insert feature in SQL Server allows you to use a format file that better defines how the expected input should be formatted, by column, including whether a column is nullable. Microsoft's doc for bulk inserts is actually pretty helpful (https://learn.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?redirectedfrom=MSDN&view=sql-server-ver15), especially the link at the end of the page to format files.
This page (https://learn.microsoft.com/en-us/sql/relational-databases/import-export/keep-nulls-or-use-default-values-during-bulk-import-sql-server?view=sql-server-ver15) directly deals with null values, which would be your predicament.
A better answer would be to add the following to your BULK INSERT...WITH statement: KEEPNULLS. It does as you would expect: it keeps null values instead of tossing them. The bulk insert utility will toss nulls by default.
Never found a direct solution from SQL express for this. I ended up going with PowerShell scripting to solve the problem. Import-CSV pulled the data from the files uniformly and without issue. Not sure why, but it handled the data far better than SQL did. From there I used variables for each line and Invoke-SQLCmd and some SQL scripting to import them into the DB. Worked like a charm. Since this process is all on the local server there aren't any security issues to worry about, so it was an acceptable solution. Thanks again for all the suggestions and help though.
Is there an option to create a database table (SQL Server) automatically depends on a large XML data? Instead of creating the table manually.
, let say you have the xml with 50 nodes
i need a pattern to create a table with the mentioned 50 columns automatically for one time Because i have a huge xml tree . My needs is to save time
You can use Liquibase for XML based Database management on SQL Server.
First, you need to integrate your code base with Liquibase then need to convert your existing XML to Liquibase version XML like below. So, you can start to manage your database changes (including DDL and DML operations) via a DBMS Tool.
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd">
<changeSet id="1" author="bob">
<createTable tableName="department">
<column name="id" type="int">
<constraints primaryKey="true" nullable="false"/>
</column>
<column name="name" type="varchar(50)">
<constraints nullable="false"/>
</column>
<column name="active" type="boolean"
defaultValueBoolean="true"/>
</createTable>
</changeSet>
</databaseChangeLog>
https://www.liquibase.org/documentation/xml_format.html
https://www.liquibase.org/documentation/tutorials/mssql.html
I have been able to execute a Stored Procedure in hibernate to map the result query into a Java Bean.
For example, I have this file called Profesor.hbm.xml:
<hibernate-mapping>
<class name="model.Profesor" table="Profesor" catalog="dbo">
<id name="idProfesor" type="int">
<column name="idProfesor" />
<generator class="assigned" />
</id>
<property name="Nombre" type="string">
<column name="Nombre" length="25" not-null="false" />
</property>
<property name="ApellidoP" type="string">
<column name="ApellidoP" length="20" not-null="false" />
</property>
<property name="ApellidoM" type="string">
<column name="ApellidoM" length="20" not-null="false" />
</property>
</class>
<sql-query name="getProfesors" callable="true">
<return alias="getProfesors" class="model.Profesor">
<return-property name="idProfesor" column="idProfesor"/>
<return-property name="Nombre" column="Nombre"/>
<return-property name="ApellidoP" column="ApellidoP"/>
<return-property name="ApellidoM" column="ApellidoM"/>
</return>
exec getProfesors :idTT
</sql-query>
</hibernate-mapping>
And then in my Profesor class I have the following code:
SessionFactory sf = new Configuration().configure().buildSessionFactory();
session = sf.getCurrentSession();
public List <Profesor> getProfesors(String idTT){
session.beginTransaction();
query.setString("idTT", idTT);
return query.list();
}
This works perfectly, I have no problem, the stored procedure executes and it populates my model Class named Profesor with the results.
Now I have another stored procedure which INSERTs data to create a new Profesor in the Database.
I tried something like this with no success:
session.beginTransaction();
Connection c = session.connection();
CallableStatement cs = c.prepareCall( "{call addProfesor(?,?,?)}" ) ;
cs.setString(1, "George");
cs.setString(2, "Williams");
cs.setString(3, "Mathematics");
cs.executeUpdate();
It doesn't even show me an error message, the data just won't insert.
I was also reading that just like there is a <"sql-query"> tag, there is a <"sql-insert">
but then I can't see the way to call that <"sql-insert"> because it doesn't have a "name" attribute just like sql-query does.
With <"sql-query"> we could do:
Query query =session.getNamedQuery("getProfesors");
Just like I showed before, but since sql-insert doesnt have that attribute name I don't know how to do it. I'm also forced to use Stored procedures since it's a very special requirement, otherwise I would have used other Hibernate persistance features.
Any help would be really appreciated.
You can specify "custom sql" for insert, update and delete. See the documentation about it.
These are not named queries. They are always used when Hibernate inserts, updates or deletes this entity.
Another way to do it is call the stored proc from a prepared statement. This is potentially useful if the stored procedure updates more than one record (batch updates).
PreparedStatement ps = getSession().connection().prepareStatement("{exec sp_batchRateUpdate}");
ps.execute();
Please note though: hibernate sessions' connection() method has been deprecated since Hibernate 3.2.4 (https://hibernate.onjira.com/browse/HHH-2603)
An alternative to this deprecation temporarily, we can use:
PreparedStatement ps = ((SessionFactoryImplementor)sessionFactory).getConnectionProvider()
.getConnection().prepareStatement("{exec sp_batchRateUpdate}");
ps.execute();
Good afternoon to all, I have this scenario:
I am using SQL Server 'BulkInsert' command to insert data in a table from a positional (.txt) file.
I use, to define structure of the file, a .xml file that defines position (and lenght) of the fields and their names.
These are 2 sample rows of the .txt positional file:
AAA111111Surname 1 Name 1
BBB222222Surname 23 Name 99
My .xml format file is defined as below:
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharFixed" LENGTH="3" />
<FIELD ID="2" xsi:type="CharFixed" LENGTH="6" />
<FIELD ID="3" xsi:type="CharFixed" LENGTH="20" />
<FIELD ID="4" xsi:type="CharFixed" LENGTH="20" />
<FIELD ID="5" xsi:type="CharTerm" TERMINATOR="\r\n" />
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="AlfaCode" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="2" NAME="NumericCode" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="3" NAME="Surname" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="4" NAME="Name" xsi:type="SQLNVARCHAR"/>
</ROW>
</BCPFORMAT>
My SQL Server code is:
DELETE from MY_TABLE
BULK INSERT MY_TABLE FROM 'C:\Directory\InputFile.txt'
WITH (
FORMATFILE = 'C:\Directory\FormatFile.xml'
)
But when I run in SQL Server the sp, I have the following error:
Msg 4866, Level 16, State 1, Line 3
The bulk load failed. The column is too long in the data file for row 1, column 58. Verify that the field terminator and row terminator are specified correctly.
Msg 7399, Level 16, State 1, Line 3
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 3
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".
This has always run perfectly until 2 months ago, then some wrong data were introduced into the file and procedure failed. Now data into the InputFile.txt are correct again, but procedure doesen't work
I checked more than 1 time InputFile.txt, FormatFile.xml and, to be sure, also MY_TABLE, but all seems perfect.
I am desperate because all seems ok, I compared also old .xml files substiuted adding only some fields.
Please answer ASAP and sorry if my english is very bad.
Don't esitate to tell me other informations.
Thanks to all
I think it's most likely your input file must still be wrong. (even though you think you fixed it).
In your example, you have 50 characters on line 1 before the line break.
Your XML says you should only have 49 chars! (3+6+20+20)
Your second line in the example only has 39 characters before the linebreak.
It's probably also worth opening your txt file in a text editor that will show you the line breaks. For instance, in notepad++, go to View -> Show Symbol -> Show All Characters. Then you can see the CR and LF characters, to verify they are there.