Is there an option to create a database table (SQL Server) automatically depends on a large XML data? Instead of creating the table manually.
, let say you have the xml with 50 nodes
i need a pattern to create a table with the mentioned 50 columns automatically for one time Because i have a huge xml tree . My needs is to save time
You can use Liquibase for XML based Database management on SQL Server.
First, you need to integrate your code base with Liquibase then need to convert your existing XML to Liquibase version XML like below. So, you can start to manage your database changes (including DDL and DML operations) via a DBMS Tool.
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd">
<changeSet id="1" author="bob">
<createTable tableName="department">
<column name="id" type="int">
<constraints primaryKey="true" nullable="false"/>
</column>
<column name="name" type="varchar(50)">
<constraints nullable="false"/>
</column>
<column name="active" type="boolean"
defaultValueBoolean="true"/>
</createTable>
</changeSet>
</databaseChangeLog>
https://www.liquibase.org/documentation/xml_format.html
https://www.liquibase.org/documentation/tutorials/mssql.html
Related
I am having a problem with using bulk insert. The issue is that the source files (tab delimited) that I'm dealing with contain rows that end in cr/lf without filling in values of the empty columns with tab for the rest of the row. So when the data is pulled into SQL Server, it's combining those shortened lines into the previous line. so basically it's combining multiple rows into one rather than writing it as two separate rows with nulls at the end of the first row.
Example to illustrate the problem: sample .txt file
column1 column2 column3 column4 column5
1 2 3 4 5
2 5 4 6
4 4 6 4
4 5 6 4 6
SQL to create table and bulk insert
CREATE TABLE test (
[column1] varchar(MAX) NULL,
[column2] varchar(MAX) NULL,
[column3] varchar(MAX) NULL,
[column4] varchar(MAX) NULL,
[column5] varchar(MAX) NULL
)
BULK INSERT test
FROM 'c:\temp\testimport.txt'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = '\t',
ROWTERMINATOR = '\r'
);
The really strange thing is that I can use the data import wizard and it imports the data perfectly, without any issue, and handles the lack of tabs for the columns just fine. But I don't know what the wizard is doing behind the scenes to make this happen. I would love to have the code it uses to create the table and do the insert as that would probably answer my question for me. At the end of the day I can't use the wizard as this will eventually be part of an automated task I'll be running against an SQL Server Express database on multiple files with different names but the same column header.
Maybe bulk insert isn't the way to go here? Or there is something obvious I'm missing that someone else might know off the top of their head. Either way, all help is appreciated and thanks in advance.
As Tim H suggested I've made a few attempts at creating a format file to accommodate the data. Results so far are as follows.
Using
bcp temp.dbo.test format nul -x -f test_format.xml -n -T
produces
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="2" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharPrefix" PREFIX_LENGTH="2" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="column1" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="2" NAME="column2" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="3" NAME="column3" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="4" NAME="column4" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="5" NAME="column5" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>
Using this temp file as is produces......
Msg 4866, Level 16, State 7, Line 31
The bulk load failed. The column is too long in the data file for row 1, column 1. Verify that the field terminator and row terminator are specified correctly.
My attempt to edit the XML to work.....
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharTerm" TERMINATOR="\t" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="column1" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="2" NAME="column2" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="3" NAME="column3" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="4" NAME="column4" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="5" NAME="column5" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>
Which does insert the data, but unfortunately still produces the same jumbled insert with overlapping lines in the same row.
Do you have control over the source files? If not, are the width of each column a fixed width or variable width? I know your create table example uses varchar(max)'s. The bulk insert feature in SQL Server allows you to use a format file that better defines how the expected input should be formatted, by column, including whether a column is nullable. Microsoft's doc for bulk inserts is actually pretty helpful (https://learn.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?redirectedfrom=MSDN&view=sql-server-ver15), especially the link at the end of the page to format files.
This page (https://learn.microsoft.com/en-us/sql/relational-databases/import-export/keep-nulls-or-use-default-values-during-bulk-import-sql-server?view=sql-server-ver15) directly deals with null values, which would be your predicament.
A better answer would be to add the following to your BULK INSERT...WITH statement: KEEPNULLS. It does as you would expect: it keeps null values instead of tossing them. The bulk insert utility will toss nulls by default.
Never found a direct solution from SQL express for this. I ended up going with PowerShell scripting to solve the problem. Import-CSV pulled the data from the files uniformly and without issue. Not sure why, but it handled the data far better than SQL did. From there I used variables for each line and Invoke-SQLCmd and some SQL scripting to import them into the DB. Worked like a charm. Since this process is all on the local server there aren't any security issues to worry about, so it was an acceptable solution. Thanks again for all the suggestions and help though.
I have a CSV file with the two columns :
Employee_Name,EmpID
"Harry, Watson",1
"Amy, Black",2
"O'hare, Lynn",3
Jeremy Prater,4
,,
,,
I am using biml to generate my package :
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<FileFormats>
<FlatFileFormat Name="FlatFile" CodePage="1252" TextQualifer="""
ColumnNamesInFirstDataRow="true" IsUnicode="false">
<Columns>
<Column Name="Employee_Name" DataType="AnsiString" Length="255" Delimiter="Comma" />
<Column Name="EmpID" DataType="Int64" Delimiter="Comma" />
</Columns>
</FlatFileFormat>
</FileFormats>
<Connections>
<FlatFileConnection Name="importexcel"
FilePath="HR.csv"
FileFormat="FlatFile" />
<Connection Name="AppSTG"
ConnectionString="XXXX"></Connection>
</Connections>
<Databases>
<Database Name="STG_App" ConnectionName="AppSTG"></Database>
</Databases>
<Schemas>
<Schema Name="HR" DatabaseName="STG_App"></Schema>
</Schemas>
<Tables>
<Table Name="Employee" SchemaName="STG_App.HR">
<Columns>
<Column Name="Employee_Name" DataType="AnsiString" Length="255" />
<Column Name="EmpID" DataType="Int64" />
</Columns>
</Table>
</Tables>
<Packages>
<Package Name="Load Flat File Data" >
<Tasks>
<Dataflow Name="Load Flat File Data">
<Transformations>
<FlatFileSource ConnectionName="importexcel" Name="FlatFile"/>
<OleDbDestination Name="Target" ConnectionName="AppSTG">
<TableOutput TableName="STG_App.HR.Employee" />
</OleDbDestination>
</Transformations>
</Dataflow>
</Tasks>
</Package>
</Packages>
</Biml>
To create the following data flow :
When I try to execute the package, I have the following error :
« Data conversion failed. The data conversion for column "EmpID"
returned status value 2 and status text "The value could not be
converted because of a potential loss of data.". »
You have defined your EmpID field as being Int64 which will work great when you have a digits there but in the case of no data (but a row still being present), SSIS is going to try to convert the empty string to a number and that will fail.
If you add an error pathway from the Flat File Source for truncation/error/etc you'd see rows 5+ going down that path. For this data, I'd define everything as string as you need to get the data into the pipeline and then you'll need to take action on it based on whatever business rules make sense (no name/id, trash it).
As #alex points out in the comment, the final rows indicate there are three columns of data whereas you've defined two so when the flat file source gets to that, you'll blow up. SSIS won't be able to handle inconsistent file formats like that.
I am working on learning Liquibase. I am working on trying to generate the SQL from the changelog that I have. For some reason, the only SQL it is generating is the locking for the DATABASECHANGELOGLOCK table.
ChangeLog
This is located in com/example/db/changelog
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">
<changeSet id="drop-tables" author="ascalonian">
<dropTable tableName="liquibase" />
</changeSet>
<changeSet id="create-tables" author="ascalonian">
<createTable tableName="liquibase">
<column name="id" type="NUMERIC(19,0)" autoIncrement="true">
<constraints primaryKey="true" nullable="false" />
</column>
<column name="firstname" type="VARCHAR(50)" />
<column name="lastname" type="VARCHAR(50)">
<constraints nullable="false" />
</column>
</createTable>
</changeSet>
<changeSet id="update-tables" author="ascalonian">
<addColumn tableName="liquibase">
<column name="username" type="VARCHAR(10)" />
</addColumn>
</changeSet>
</databaseChangeLog>
Command Line
java -jar liquibase.jar
--driver=net.sourceforge.jtds.jdbc.Driver
--classpath=/Users/<username>/.m2/repository/net/sourceforge/jtds/jtds/1.3.1/jtds-1.3.1.jar
--changeLogFile=com/example/db/changelog/db.changelog-master.xml
--url=jdbc:jtds:sqlserver://localhost:1433/TestDB
--username=username
--password=password
updateSQL
SQL Output
-- *********************************************************************
-- Update Database Script
-- *********************************************************************
-- Change Log: com/example/db/changelog/db.changelog-create.xml
-- Ran at: 8/11/15 4:28 PM
-- Against: username#jdbc:jtds:sqlserver://localhost:1433/TestDB
-- Liquibase version: 3.4.1
-- *********************************************************************
-- Lock Database
UPDATE [dbo].[DATABASECHANGELOGLOCK] SET [LOCKED] = 1, [LOCKEDBY] = 'fe80:0:0:0:541c:15ff:fe8f:7826%9 (fe80:0:0:0:541c:15ff:fe8f:7826%9)', [LOCKGRANTED] = '2015-08-11T16:28:18.090' WHERE [ID] = 1 AND [LOCKED] = 0
GO
-- Release Database Lock
UPDATE [dbo].[DATABASECHANGELOGLOCK] SET [LOCKED] = 0, [LOCKEDBY] = NULL, [LOCKGRANTED] = NULL WHERE [ID] = 1
GO
I was expecting the SQL for the drop, create and update table but don't see anything.
Liquibase: Version 3.4.1
Database: MS SQL Server
Cannot reproduce your issue. Have you already run the "update" command against your database? That would explain the lack of SQL (because the changes were already applied to the target DB).
The DATABASECHANGELOG table records which changesets have been applied. Checkout the rollbackSQL command, that will show you the undo actions
I am trying to import .csv file but i am getting "BULK LOAD DATA CONVERSION ERROR" for last column. File looks like:
"123456","123","001","0.00"
I have tried below rowterminator:
ROW TERMINATOR = "\"\r\n"
Nothing is working. Any ideas on what is causing this record to have this error? Thanks
As per given example below, remove the quotes in your csv and use the terminator as "\r\n".
Always use format xml when doing bulk insert. It provides several advantages such as validation of data files etc.
The format file maps the fields of the data file to the columns of the table. You can use a non-XML or XML format file to bulk import data when using a bcp command or a BULK INSERT or INSERT or Transact-SQL command
Considering the input file given by you, suppose you have a table as given below :
CREATE TABLE myTestFormatFiles (
Col1 smallint,
Col2 nvarchar(50),
Col3 nvarchar(50),
Col4 nvarchar(50)
);
Your sample Data File will be as follows :
10,Field2,Field3,Field4
15,Field2,Field3,Field4
46,Field2,Field3,Field4
58,Field2,Field3,Field4
Sample format XML file will be :
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="7"/>
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="Col1" xsi:type="SQLSMALLINT"/>
<COLUMN SOURCE="2" NAME="Col2" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="3" NAME="Col3" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="4" NAME="Col4" xsi:type="SQLNVARCHAR"/>
</ROW>
</BCPFORMAT>
If you are unfamiliar with format files, check XML Format Files (SQL Server).
Example is illustrated here
I have been able to execute a Stored Procedure in hibernate to map the result query into a Java Bean.
For example, I have this file called Profesor.hbm.xml:
<hibernate-mapping>
<class name="model.Profesor" table="Profesor" catalog="dbo">
<id name="idProfesor" type="int">
<column name="idProfesor" />
<generator class="assigned" />
</id>
<property name="Nombre" type="string">
<column name="Nombre" length="25" not-null="false" />
</property>
<property name="ApellidoP" type="string">
<column name="ApellidoP" length="20" not-null="false" />
</property>
<property name="ApellidoM" type="string">
<column name="ApellidoM" length="20" not-null="false" />
</property>
</class>
<sql-query name="getProfesors" callable="true">
<return alias="getProfesors" class="model.Profesor">
<return-property name="idProfesor" column="idProfesor"/>
<return-property name="Nombre" column="Nombre"/>
<return-property name="ApellidoP" column="ApellidoP"/>
<return-property name="ApellidoM" column="ApellidoM"/>
</return>
exec getProfesors :idTT
</sql-query>
</hibernate-mapping>
And then in my Profesor class I have the following code:
SessionFactory sf = new Configuration().configure().buildSessionFactory();
session = sf.getCurrentSession();
public List <Profesor> getProfesors(String idTT){
session.beginTransaction();
query.setString("idTT", idTT);
return query.list();
}
This works perfectly, I have no problem, the stored procedure executes and it populates my model Class named Profesor with the results.
Now I have another stored procedure which INSERTs data to create a new Profesor in the Database.
I tried something like this with no success:
session.beginTransaction();
Connection c = session.connection();
CallableStatement cs = c.prepareCall( "{call addProfesor(?,?,?)}" ) ;
cs.setString(1, "George");
cs.setString(2, "Williams");
cs.setString(3, "Mathematics");
cs.executeUpdate();
It doesn't even show me an error message, the data just won't insert.
I was also reading that just like there is a <"sql-query"> tag, there is a <"sql-insert">
but then I can't see the way to call that <"sql-insert"> because it doesn't have a "name" attribute just like sql-query does.
With <"sql-query"> we could do:
Query query =session.getNamedQuery("getProfesors");
Just like I showed before, but since sql-insert doesnt have that attribute name I don't know how to do it. I'm also forced to use Stored procedures since it's a very special requirement, otherwise I would have used other Hibernate persistance features.
Any help would be really appreciated.
You can specify "custom sql" for insert, update and delete. See the documentation about it.
These are not named queries. They are always used when Hibernate inserts, updates or deletes this entity.
Another way to do it is call the stored proc from a prepared statement. This is potentially useful if the stored procedure updates more than one record (batch updates).
PreparedStatement ps = getSession().connection().prepareStatement("{exec sp_batchRateUpdate}");
ps.execute();
Please note though: hibernate sessions' connection() method has been deprecated since Hibernate 3.2.4 (https://hibernate.onjira.com/browse/HHH-2603)
An alternative to this deprecation temporarily, we can use:
PreparedStatement ps = ((SessionFactoryImplementor)sessionFactory).getConnectionProvider()
.getConnection().prepareStatement("{exec sp_batchRateUpdate}");
ps.execute();