.NET: How to insert XML document into SQL Server - sql-server

I want to insert arbitrary XML into SQL Server. The XML is contained in an XmlDocument object.
The column I want to insert into is either nvarchar, ntext, or xml column (If it makes your life easier then you can pick which type it is. Really it's an xml column.)
Prototype
void SaveXmlToDatabase(DbConnection connection,
XmlDocument xmlToSave,
String tableName, String columnName);
{
}
The reason I ask is because I'm trying to find the proper way to turn the XmlDocument into something the database can take - being sure to keep the encodings right:
I have to make sure the encoding using during insert matches what the database takes
I have to synchronize the <?xml version="1.0" encoding="windows-1252"?> element
I know ntext, nvarchar, or xml are stored as UTF-16 inside SQL Server. So I have to be sure to give the data to SQL Server as UTF-16. This isn't a problem for Strings in .NET, since they are unicode UTF-16.
The 2nd problem, synchronizing the encoding attribute, is a tougher nut to crack. I have to figure out how to find the declaration element through the XmlDocument object:
<?xml version="1.0" encoding="windows-1252"?> (or whatever the encoding may be)
and adjust it to UTF-16
<?xml version="1.0" encoding="UTF-16"?>
My naive attempt (that fails)
Ignoring the encoding in the XML declaration, and just figuring out how to save anything into SQL Server:
void SaveXmlToDatabase(DbConnection connection,
XmlDocument xmlToSave,
String tableName, String columnName);
{
String sql = "INSERT INTO "+tableName+" ("+columnName+")
VALUES ('"+xmlToSave.ToString()+"')";
using (DbCommand command = connection.CreateCommand())
{
command.CommandText = sql;
DbTransaction trans = connection.BeginTransaction();
try
{
command.ExecuteNonQuery();
trans.Commit();
}
catch (Exception)
{
trans.Rollback();
throw;
}
}
}
This fails because the sql I try to run is:
INSERT INTO LiveData (RawXML)
VALUES ('System.Xml.XmlDocument')
This is because XmlDocument.ToString() returns "System.Xml.XmlDocument". Peeking at the implementation, it see that it literally is calling:
this.GetType().ToString();
Aside: Microsoft seems to have gone out of their way to prevent you
from getting the Xml as a string -
presumably because it leads to bugs
(But they don't tell us what bugs, why
they're bugs, or the right way to
convert an XmlDocument into a
String!)
See also
C#/SQL - What’s wrong with SqlDbType.Xml in procedures ?
How to convert XmlDocument to XmlReader for SqlXml data type.

You have to use an SqlParameter.
I would recommend doing it like that:
command.Parameters.Add(
new SqlParameter("#xml", SqlDbType.Xml)
{Value = new SqlXml(new XmlTextReader(xmlToSave.InnerXml
, XmlNodeType.Document, null)) })
and the SQL should look like:
String sql = "INSERT INTO "+tableName+" ("+columnName+") VALUES (#xml)";
And since the first child is always the xml node, you can replace the encoding with the following statement.
xmlToSave.FirstChild.InnerText = "version=\"1.0\" encoding=\"UTF-16\"";
All in all it would look like that:
void SaveXmlToDatabase(DbConnection connection,
XmlDocument xmlToSave,
String tableName, String columnName);
{
String sql = "INSERT INTO "+tableName+" ("+columnName+") VALUES (#xml)";
using (DbCommand command = connection.CreateCommand())
{
xmlToSave.FirstChild.InnerText = "version=\"1.0\" encoding=\"UTF-16\"";
command.CommandText = sql;
command.Parameters.Add(
new SqlParameter("#xml", SqlDbType.Xml)
{Value = new SqlXml(new XmlTextReader(xmlToSave.InnerXml
, XmlNodeType.Document, null)) });
DbTransaction trans = connection.BeginTransaction();
try
{
command.ExecuteNonQuery();
trans.Commit();
}
catch (Exception)
{
trans.Rollback();
throw;
}
}
}

The simplest solution is to use the OuterXml attribute of the document. I came across your question while trying to solve the problem under the condition that I could not executed a stored procedure. OuterXml returns the string of text you were expecting to get from .Value or .ToString()
void SaveXmlToDatabase(DbConnection connection,
XmlDocument xmlToSave,
String tableName, String columnName);
{
String sql = "INSERT INTO "+tableName+" ("+columnName+")
VALUES ('"+xmlToSave.OuterXml+"')";
using (DbCommand command = connection.CreateCommand())
{
command.CommandText = sql;
DbTransaction trans = connection.BeginTransaction();
try
{
command.ExecuteNonQuery();
trans.Commit();
}
catch (Exception)
{
trans.Rollback();
throw;
}
}
}

Better late than never... I think you're looking for something like this:
void SaveXmlToDatabase(DbConnection connection,
XmlDocument xmlToSave,
String tableName, String columnName)
{
String sql = "INSERT INTO "+tableName+" ("+columnName+")
VALUES (#XmlVal)";
using (DbCommand command = connection.CreateCommand())
{
command.CommandText = sql;
command.Parameters.AddWithValue("XmlVal", new SqlXml(new XmlNodeReader(xmlToSave)));
DbTransaction trans = connection.BeginTransaction();
try
{
command.ExecuteNonQuery();
trans.Commit();
}
catch (Exception)
{
trans.Rollback();
throw;
}
}
}
The XmlNodeReader object traverses and properly encodes the XmlDocument (or any other XmlNode), and the SqlXml object encapsulates that into the SqlDbType suitable for use with the parameter. This is safer and probably more efficient than using a string intermediary.

Just wanted to add an SQLConnection equivalent to Jonathan Overholt's solution:
private void StoreXMLInDatabase(XmlDocument doc)
{
// xmlvalue is the name of the database column you want to insert into
String sql = "INSERT INTO [tablename] (xmlvalue) VALUES (#xmlvalue)";
// In order to read out you connection string from app.config, remember first to add a reference to System.Configuration in references,
// and set using System.Configuration; in the top of the file
string connString = ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString;
using (SqlConnection conn = new SqlConnection(connString))
{
if (conn.State == ConnectionState.Closed)
{
conn.Open();
}
SqlTransaction transaction = conn.BeginTransaction();
try
{
// Remember to specify the SqlTransaction *name* (transaction)
SqlCommand cobj = new SqlCommand(sql, conn, transaction);
cobj.CommandText = sql;
cobj.Parameters.AddWithValue("xmlvalue", new SqlXml(new XmlNodeReader(doc)));
cobj.ExecuteNonQuery();
transaction.Commit();
}
catch (SqlException ex)
{
//2627 = inserting duplicate key fail. Lets the procedure continue although an identity insert fail occurs
if (ex.Number == 2627)
{
transaction.Rollback();
}
}
} // end using
}
Tested OK in Visual studio 2017 and 2013 with .net up to version 4.6.2, and against MS SQL Server 2008 R2, 2012 and 2016
If an identity insert fail and you want to stop the procedure then, just insert a line after transaction.Rollback() like this:
throw;
Or you can stop the procedure in any circumstance by putting the transaction.Rollback(); and throw; lines just outside the conditional (the if)

Related

How to use scope_identity in jdbc [duplicate]

I want to INSERT a record in a database (which is Microsoft SQL Server in my case) using JDBC in Java. At the same time, I want to obtain the insert ID. How can I achieve this using JDBC API?
If it is an auto generated key, then you can use Statement#getGeneratedKeys() for this. You need to call it on the same Statement as the one being used for the INSERT. You first need to create the statement using Statement.RETURN_GENERATED_KEYS to notify the JDBC driver to return the keys.
Here's a basic example:
public void create(User user) throws SQLException {
try (
Connection connection = dataSource.getConnection();
PreparedStatement statement = connection.prepareStatement(SQL_INSERT,
Statement.RETURN_GENERATED_KEYS);
) {
statement.setString(1, user.getName());
statement.setString(2, user.getPassword());
statement.setString(3, user.getEmail());
// ...
int affectedRows = statement.executeUpdate();
if (affectedRows == 0) {
throw new SQLException("Creating user failed, no rows affected.");
}
try (ResultSet generatedKeys = statement.getGeneratedKeys()) {
if (generatedKeys.next()) {
user.setId(generatedKeys.getLong(1));
}
else {
throw new SQLException("Creating user failed, no ID obtained.");
}
}
}
}
Note that you're dependent on the JDBC driver as to whether it works. Currently, most of the last versions will work, but if I am correct, Oracle JDBC driver is still somewhat troublesome with this. MySQL and DB2 already supported it for ages. PostgreSQL started to support it not long ago. I can't comment about MSSQL as I've never used it.
For Oracle, you can invoke a CallableStatement with a RETURNING clause or a SELECT CURRVAL(sequencename) (or whatever DB-specific syntax to do so) directly after the INSERT in the same transaction to obtain the last generated key. See also this answer.
Create Generated Column
String generatedColumns[] = { "ID" };
Pass this geneated Column to your statement
PreparedStatement stmtInsert = conn.prepareStatement(insertSQL, generatedColumns);
Use ResultSet object to fetch the GeneratedKeys on Statement
ResultSet rs = stmtInsert.getGeneratedKeys();
if (rs.next()) {
long id = rs.getLong(1);
System.out.println("Inserted ID -" + id); // display inserted record
}
When encountering an 'Unsupported feature' error while using Statement.RETURN_GENERATED_KEYS, try this:
String[] returnId = { "BATCHID" };
String sql = "INSERT INTO BATCH (BATCHNAME) VALUES ('aaaaaaa')";
PreparedStatement statement = connection.prepareStatement(sql, returnId);
int affectedRows = statement.executeUpdate();
if (affectedRows == 0) {
throw new SQLException("Creating user failed, no rows affected.");
}
try (ResultSet rs = statement.getGeneratedKeys()) {
if (rs.next()) {
System.out.println(rs.getInt(1));
}
rs.close();
}
Where BATCHID is the auto generated id.
I'm hitting Microsoft SQL Server 2008 R2 from a single-threaded JDBC-based application and pulling back the last ID without using the RETURN_GENERATED_KEYS property or any PreparedStatement. Looks something like this:
private int insertQueryReturnInt(String SQLQy) {
ResultSet generatedKeys = null;
int generatedKey = -1;
try {
Statement statement = conn.createStatement();
statement.execute(SQLQy);
} catch (Exception e) {
errorDescription = "Failed to insert SQL query: " + SQLQy + "( " + e.toString() + ")";
return -1;
}
try {
generatedKey = Integer.parseInt(readOneValue("SELECT ##IDENTITY"));
} catch (Exception e) {
errorDescription = "Failed to get ID of just-inserted SQL query: " + SQLQy + "( " + e.toString() + ")";
return -1;
}
return generatedKey;
}
This blog post nicely isolates three main SQL Server "last ID" options:
http://msjawahar.wordpress.com/2008/01/25/how-to-find-the-last-identity-value-inserted-in-the-sql-server/ - haven't needed the other two yet.
Instead of a comment, I just want to answer post.
Interface java.sql.PreparedStatement
columnIndexes « You can use prepareStatement function that accepts columnIndexes and SQL statement.
Where columnIndexes allowed constant flags are Statement.RETURN_GENERATED_KEYS1 or Statement.NO_GENERATED_KEYS[2], SQL statement that may contain one or more '?' IN parameter placeholders.
SYNTAX «
Connection.prepareStatement(String sql, int autoGeneratedKeys)
Connection.prepareStatement(String sql, int[] columnIndexes)
Example:
PreparedStatement pstmt =
conn.prepareStatement( insertSQL, Statement.RETURN_GENERATED_KEYS );
columnNames « List out the columnNames like 'id', 'uniqueID', .... in the target table that contain the auto-generated keys that should be returned. The driver will ignore them if the SQL statement is not an INSERT statement.
SYNTAX «
Connection.prepareStatement(String sql, String[] columnNames)
Example:
String columnNames[] = new String[] { "id" };
PreparedStatement pstmt = conn.prepareStatement( insertSQL, columnNames );
Full Example:
public static void insertAutoIncrement_SQL(String UserName, String Language, String Message) {
String DB_URL = "jdbc:mysql://localhost:3306/test", DB_User = "root", DB_Password = "";
String insertSQL = "INSERT INTO `unicodeinfo`( `UserName`, `Language`, `Message`) VALUES (?,?,?)";
//"INSERT INTO `unicodeinfo`(`id`, `UserName`, `Language`, `Message`) VALUES (?,?,?,?)";
int primkey = 0 ;
try {
Class.forName("com.mysql.jdbc.Driver").newInstance();
Connection conn = DriverManager.getConnection(DB_URL, DB_User, DB_Password);
String columnNames[] = new String[] { "id" };
PreparedStatement pstmt = conn.prepareStatement( insertSQL, columnNames );
pstmt.setString(1, UserName );
pstmt.setString(2, Language );
pstmt.setString(3, Message );
if (pstmt.executeUpdate() > 0) {
// Retrieves any auto-generated keys created as a result of executing this Statement object
java.sql.ResultSet generatedKeys = pstmt.getGeneratedKeys();
if ( generatedKeys.next() ) {
primkey = generatedKeys.getInt(1);
}
}
System.out.println("Record updated with id = "+primkey);
} catch (InstantiationException | IllegalAccessException | ClassNotFoundException | SQLException e) {
e.printStackTrace();
}
}
I'm using SQLServer 2008, but I have a development limitation: I cannot use a new driver for it, I have to use "com.microsoft.jdbc.sqlserver.SQLServerDriver" (I cannot use "com.microsoft.sqlserver.jdbc.SQLServerDriver").
That's why the solution conn.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS) threw a java.lang.AbstractMethodError for me.
In this situation, a possible solution I found is the old one suggested by Microsoft:
How To Retrieve ##IDENTITY Value Using JDBC
import java.sql.*;
import java.io.*;
public class IdentitySample
{
public static void main(String args[])
{
try
{
String URL = "jdbc:microsoft:sqlserver://yourServer:1433;databasename=pubs";
String userName = "yourUser";
String password = "yourPassword";
System.out.println( "Trying to connect to: " + URL);
//Register JDBC Driver
Class.forName("com.microsoft.jdbc.sqlserver.SQLServerDriver").newInstance();
//Connect to SQL Server
Connection con = null;
con = DriverManager.getConnection(URL,userName,password);
System.out.println("Successfully connected to server");
//Create statement and Execute using either a stored procecure or batch statement
CallableStatement callstmt = null;
callstmt = con.prepareCall("INSERT INTO myIdentTable (col2) VALUES (?);SELECT ##IDENTITY");
callstmt.setString(1, "testInputBatch");
System.out.println("Batch statement successfully executed");
callstmt.execute();
int iUpdCount = callstmt.getUpdateCount();
boolean bMoreResults = true;
ResultSet rs = null;
int myIdentVal = -1; //to store the ##IDENTITY
//While there are still more results or update counts
//available, continue processing resultsets
while (bMoreResults || iUpdCount!=-1)
{
//NOTE: in order for output parameters to be available,
//all resultsets must be processed
rs = callstmt.getResultSet();
//if rs is not null, we know we can get the results from the SELECT ##IDENTITY
if (rs != null)
{
rs.next();
myIdentVal = rs.getInt(1);
}
//Do something with the results here (not shown)
//get the next resultset, if there is one
//this call also implicitly closes the previously obtained ResultSet
bMoreResults = callstmt.getMoreResults();
iUpdCount = callstmt.getUpdateCount();
}
System.out.println( "##IDENTITY is: " + myIdentVal);
//Close statement and connection
callstmt.close();
con.close();
}
catch (Exception ex)
{
ex.printStackTrace();
}
try
{
System.out.println("Press any key to quit...");
System.in.read();
}
catch (Exception e)
{
}
}
}
This solution worked for me!
I hope this helps!
You can use following java code to get new inserted id.
ps = con.prepareStatement(query, Statement.RETURN_GENERATED_KEYS);
ps.setInt(1, quizid);
ps.setInt(2, userid);
ps.executeUpdate();
ResultSet rs = ps.getGeneratedKeys();
if (rs.next()) {
lastInsertId = rs.getInt(1);
}
It is possible to use it with normal Statement's as well (not just PreparedStatement)
Statement statement = conn.createStatement();
int updateCount = statement.executeUpdate("insert into x...)", Statement.RETURN_GENERATED_KEYS);
try (ResultSet generatedKeys = statement.getGeneratedKeys()) {
if (generatedKeys.next()) {
return generatedKeys.getLong(1);
}
else {
throw new SQLException("Creating failed, no ID obtained.");
}
}
Most others have suggested to use JDBC API for this, but personally, I find it quite painful to do with most drivers. When in fact, you can just use a native T-SQL feature, the OUTPUT clause:
try (
Statement s = c.createStatement();
ResultSet rs = s.executeQuery(
"""
INSERT INTO t (a, b)
OUTPUT id
VALUES (1, 2)
"""
);
) {
while (rs.next())
System.out.println("ID = " + rs.getLong(1));
}
This is the simplest solution for SQL Server as well as a few other SQL dialects (e.g. Firebird, MariaDB, PostgreSQL, where you'd use RETURNING instead of OUTPUT).
I've blogged about this topic more in detail here.
With Hibernate's NativeQuery, you need to return a ResultList instead of a SingleResult, because Hibernate modifies a native query
INSERT INTO bla (a,b) VALUES (2,3) RETURNING id
like
INSERT INTO bla (a,b) VALUES (2,3) RETURNING id LIMIT 1
if you try to get a single result, which causes most databases (at least PostgreSQL) to throw a syntax error. Afterwards, you may fetch the resulting id from the list (which usually contains exactly one item).
In my case ->
ConnectionClass objConnectionClass=new ConnectionClass();
con=objConnectionClass.getDataBaseConnection();
pstmtGetAdd=con.prepareStatement(SQL_INSERT_ADDRESS_QUERY,Statement.RETURN_GENERATED_KEYS);
pstmtGetAdd.setString(1, objRegisterVO.getAddress());
pstmtGetAdd.setInt(2, Integer.parseInt(objRegisterVO.getCityId()));
int addId=pstmtGetAdd.executeUpdate();
if(addId>0)
{
ResultSet rsVal=pstmtGetAdd.getGeneratedKeys();
rsVal.next();
addId=rsVal.getInt(1);
}
If you are using Spring JDBC, you can use Spring's GeneratedKeyHolder class to get the inserted ID.
See this answer...
How to get inserted id using Spring Jdbctemplate.update(String sql, obj...args)
If you are using JDBC (tested with MySQL) and you just want the last inserted ID, there is an easy way to get it. The method I'm using is the following:
public static Integer insert(ConnectionImpl connection, String insertQuery){
Integer lastInsertId = -1;
try{
final PreparedStatement ps = connection.prepareStatement(insertQuery);
ps.executeUpdate(insertQuery);
final com.mysql.jdbc.PreparedStatement psFinal = (com.mysql.jdbc.PreparedStatement) ps;
lastInsertId = (int) psFinal.getLastInsertID();
connection.close();
} catch(SQLException ex){
System.err.println("Error: "+ex);
}
return lastInsertId;
}
Also, (and just in case) the method to get the ConnectionImpl is the following:
public static ConnectionImpl getConnectionImpl(){
ConnectionImpl conexion = null;
final String dbName = "database_name";
final String dbPort = "3306";
final String dbIPAddress = "127.0.0.1";
final String connectionPath = "jdbc:mysql://"+dbIPAddress+":"+dbPort+"/"+dbName+"?autoReconnect=true&useSSL=false";
final String dbUser = "database_user";
final String dbPassword = "database_password";
try{
conexion = (ConnectionImpl) DriverManager.getConnection(connectionPath, dbUser, dbPassword);
}catch(SQLException e){
System.err.println(e);
}
return conexion;
}
Remember to add the connector/J to the project referenced libraries.
In my case, the connector/J version is the 5.1.42. Maybe you will have to apply some changes to the connectionPath if you want to use a more modern version of the connector/J such as with the version 8.0.28.
In the file, remember to import the following resources:
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import com.mysql.jdbc.ConnectionImpl;
Hope this will be helpful.
Connection cn = DriverManager.getConnection("Host","user","pass");
Statement st = cn.createStatement("Ur Requet Sql");
int ret = st.execute();

Exception while executing insert query having some special characters in the value

I have a column with character varying datatype. We are using Npgsql -Version 4.0.4 and PostgreSQL version 11.3
String value = "Shaun Hollis_002\\Top of Personal Folders\002\\� 35% Off
Harlem Globetrotters - 3 Events in Denver!.msg";
try
{
StringBuilder sb = new StringBuilder();
sb.AppendLine("Insert INTo tbltemptest1 (c_value) values(#value);");
using (NpgsqlConnection con = new NpgsqlConnection(DBConnectionString))
{
con.Open();
using (NpgsqlCommand command = new NpgsqlCommand(sb.ToString(), con))
{
command.Parameters.Add("#value", NpgsqlTypes.NpgsqlDbType.Varchar).Value = value;
command.ExecuteNonQuery();
}
}
}
catch (Exception ex)
{
throw;
}
Performing the insert query is throwing an exception
invalid byte sequence for encoding "UTF8": 0x00
The error pretty much say it: PostgreSQL refuses to accept strings with 0x00 (the null character) inside them. You will need to clean up your input before sending it to PostgreSQL.

Unexpected end of JSON input: asp.net core

when select query
select * from Permissions where AppId='yZVwUoxKQCu' FOR JSON PATH,INCLUDE_NULL_VALUES
executes in sql server direct it return full data with array wrapper without unexpected end.when the query read with DataReader in C# it the same query result have unexpected end. the sql execute using below the method.
Query like this
var sql = $"select * from Permissions where AppId='{AppId}' FOR JSON PATH,INCLUDE_NULL_VALUES";
var res = Connection.ExecuteScalarCommand(sql);
public static String ExecuteScalarCommand(string sql)
{
string CS = DbConnectionString;
SqlConnection con = new SqlConnection(CS);
string val = "";
try
{
con.Open();
SqlCommand cmd = new SqlCommand(sql, con);
val = cmd.ExecuteScalar().ToString();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message.ToString());
}
finally
{
con.Close();
}
return val;
}
The output given by sql server direct is given.its expected output.
[{"Id":49,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Season","UserId":60,"PermissionTitleId":1,"AppId":"yZVwUoxKQCu"},{"Id":50,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Categories","UserId":60,"PermissionTitleId":2,"AppId":"yZVwUoxKQCu"},{"Id":51,"IsCreatable":2,"IsViewable":2,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Tabs","UserId":60,"PermissionTitleId":3,"AppId":"yZVwUoxKQCu"},{"Id":52,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Fields","UserId":60,"PermissionTitleId":4,"AppId":"yZVwUoxKQCu"},{"Id":53,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"ContentBlock","UserId":60,"PermissionTitleId":5,"AppId":"yZVwUoxKQCu"},{"Id":54,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"User","UserId":60,"PermissionTitleId":6,"AppId":"yZVwUoxKQCu"},{"Id":55,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Roles","UserId":60,"PermissionTitleId":7,"AppId":"yZVwUoxKQCu"},{"Id":56,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Season","UserId":60,"PermissionTitleId":1,"AppId":"yZVwUoxKQCu"},{"Id":57,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Categories","UserId":60,"PermissionTitleId":2,"AppId":"yZVwUoxKQCu"},{"Id":58,"IsCreatable":2,"IsViewable":2,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Tabs","UserId":60,"PermissionTitleId":3,"AppId":"yZVwUoxKQCu"},{"Id":59,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Fields","UserId":60,"PermissionTitleId":4,"AppId":"yZVwUoxKQCu"},{"Id":60,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"ContentBlock","UserId":60,"PermissionTitleId":5,"AppId":"yZVwUoxKQCu"},{"Id":61,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"User","UserId":60,"PermissionTitleId":6,"AppId":"yZVwUoxKQCu"},{"Id":62,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Roles","UserId":60,"PermissionTitleId":7,"AppId":"yZVwUoxKQCu"},{"Id":42,"IsCreatable":0,"IsViewable":0,"IsDeletable":1,"IsUpdatable":0,"RoleId":28,"Title":"Season","UserId":60,"PermissionTitleId":1,"AppId":"yZVwUoxKQCu"},{"Id":43,"IsCreatable":1,"IsViewable":0,"IsDeletable":1,"IsUpdatable":1,"RoleId":28,"Title":"Categories","UserId":60,"PermissionTitleId":2,"AppId":"yZVwUoxKQCu"},{"Id":44,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":28,"Title":"Tabs","UserId":60,"PermissionTitleId":3,"AppId":"yZVwUoxKQCu"},{"Id":45,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":28,"Title":"Fields","UserId":60,"PermissionTitleId":4,"AppId":"yZVwUoxKQCu"},{"Id":46,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":28,"Title":"ContentBlock","UserId":60,"PermissionTitleId":5,"AppId":"yZVwUoxKQCu"},{"Id":47,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":28,"Title":"User","UserId":60,"PermissionTitleId":6,"AppId":"yZVwUoxKQCu"},{"Id":48,"IsCreatable":0,"IsViewable":0,"IsDeletable":1,"IsUpdatable":0,"RoleId":28,"Title":"Roles","UserId":60,"PermissionTitleId":7,"AppId":"yZVwUoxKQCu"}]
but out put from asp.net core is
[{"Id":49,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Season","UserId":60,"PermissionTitleId":1,"AppId":"yZVwUoxKQCu"},{"Id":50,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Categories","UserId":60,"PermissionTitleId":2,"AppId":"yZVwUoxKQCu"},{"Id":51,"IsCreatable":2,"IsViewable":2,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Tabs","UserId":60,"PermissionTitleId":3,"AppId":"yZVwUoxKQCu"},{"Id":52,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Fields","UserId":60,"PermissionTitleId":4,"AppId":"yZVwUoxKQCu"},{"Id":53,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"ContentBlock","UserId":60,"PermissionTitleId":5,"AppId":"yZVwUoxKQCu"},{"Id":54,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"User","UserId":60,"PermissionTitleId":6,"AppId":"yZVwUoxKQCu"},{"Id":55,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Roles","UserId":60,"PermissionTitleId":7,"AppId":"yZVwUoxKQCu"},{"Id":56,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Season","UserId":60,"PermissionTitleId":1,"AppId":"yZVwUoxKQCu"},{"Id":57,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Categories","UserId":60,"PermissionTitleId":2,"AppId":"yZVwUoxKQCu"},{"Id":58,"IsCreatable":2,"IsViewable":2,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Tabs","UserId":60,"PermissionTitleId":3,"AppId":"yZVwUoxKQCu"},{"Id":59,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Fields","UserId":60,"PermissionTitleId":4,"AppId":"yZVwUoxKQCu"},{"Id":60,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"ContentBlock","UserId":60,"PermissionTitleId":5,"AppId":"yZVwUoxKQCu"},{"Id":61,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"User","UserId":60,"PermissionT
The ExecuteScalar has a character limit:
The first column of the first row in the result set, or a null
reference (Nothing in Visual Basic) if the result set is empty.
Returns a maximum of 2033 characters.
To overcome that you will need to use a different function, like ExecuteReader.

Passing Table Type Input Parameter To Stored Procedure in Execute SQL Task in SSIS

I am trying to call a stored procedure from Execute SQL Task in SSIS. The stored procedure accepts two table type variable which I am constructing in another Execute SQL Task and storing in object type of variable.
Since I need to pass object type of variable I am using ADO.NET connection manager.
Also since I am using ADO.NET connection manager, I cannot pass the input parameter as ? mark in direct input and I need to write expressions.
But expressions do not support object type variable.
What is the way out?
Thanks
You won't be able to use the native ADO/OLE source components. You'll have to use a Script Component acting as a Source and use .NET to create
Relevant bits from my SQL TVP post
string connectionString = #"Data Source=localhost;Initial Catalog=master;Integrated Security=True";
System.Data.SqlClient.SqlConnection conn = new System.Data.SqlClient.SqlConnection(connectionString);
System.Data.DataTable dataTable = null;
// note that my data table does not have to be the same
// as my UDTT, nor do my columns have to have the same name
// Types-yes. Ordinal, probably so
dataTable = new System.Data.DataTable("Sample");
dataTable.Columns.Add("tweep", System.Type.GetType("System.String"));
dataTable.Columns.Add("have_met", System.Type.GetType("System.Boolean"));
// add rows to my data table but really, this could be any source
dataTable.Rows.Add(new object[] { "billinkc", true });
dataTable.Rows.Add(new object[] { "BrentO", true });
dataTable.Rows.Add(new object[] { "buckwoody", false });
// Hooray for #sqlsat35 and meeting Jen & Sean
dataTable.Rows.Add(new object[] { "MidnightDBA", true });
System.Data.SqlClient.SqlConnection connection = null;
System.Data.DataSet results = null;
System.Data.SqlClient.SqlCommand command = null;
System.Data.SqlClient.SqlDataReader dataReader = null;
connection = new System.Data.SqlClient.SqlConnection(connectionString);
try
{
connection.Open();
command = new System.Data.SqlClient.SqlCommand("TwitterAdd");
command.CommandType = System.Data.CommandType.StoredProcedure;
command.Connection = connection;
// Assigning a table valued parameter looks much like any other parameter
System.Data.SqlClient.SqlParameter tvp = command.Parameters.AddWithValue("#tvp", dataTable);
// this is the only special sauce (not required but helpful)
tvp.SqlDbType = System.Data.SqlDbType.Structured;
tvp.TypeName = "dbo.CONTRIVED_EXAMPLE";
dataReader = command.ExecuteReader();
if (dataReader.HasRows)
{
// pseudo logic here
while(dataReader.Read())
{
MyBuffer.AddRow();
MyBuffer.Col1 = dataReader["Col1"].ToString();
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex);
}

Different ways of performing bulk insert into database from a java application

I am looking for different ways of performing bulk insert into database (e.g. SQL Server 2012) from a Java application. I need to insert lot of entities into database very efficiently without making as many calls to database as there are entities.
My requirement is to perform a bulk insert of entities, where an insert of entity in database could involve inserting data into one or more tables. The following are the two ways which I can think of:
Dynamically generate a batch of SQL statements and execute it against the database by making use of native JDBC support.
Construct XML representation of all the entities and then invoke a stored procedure by passing the generated XML. The stored procedure takes care of parsing the XML and inserting the entities to database.
I am new to Java and not having enough knowledge of available frameworks. IMO, the above two approaches seems to be very naive and not leveraging the available frameworks. I am requesting experts to share different ways of achieving bulk insert along with its pros and cons. I am open to MyBatis, Spring-MyBatis, Spring-JDBC, JDBC, etc which solves the problem in an efficient manner.
Thanks.
I have a demo ,JDBC batch processing
file:demo.txt
The content
1899942 ,demo1
1899944 ,demo2
1899946 ,demo3
1899948 ,demo4
Insert the data reads the file content
my code:
public class Test2 {
public static void main(String[] args) {
long start = System.currentTimeMillis();
String sql = "insert into mobile_place(number,place) values(?,?)";
int count=0;
PreparedStatement pstmt = null;
Connection conn = JDBCUtil.getConnection();
try {
pstmt = conn.prepareStatement(sql);
InputStreamReader is = new InputStreamReader(new FileInputStream(new File("D:/CC.txt")),"utf-8");
BufferedReader br = new BufferedReader(is);
conn.setAutoCommit(false);
String s1 = null;
String s2 = null;
while(br.readLine() != null){
count++;
String str = br.readLine().toString().trim();
s1 = str.substring(0, str.indexOf(","));
s2 = str.substring(str.indexOf(",")+1,str.length());
pstmt.setString(1, s1);
pstmt.setString(2, s2);
pstmt.addBatch();
if(count%1000==0){
pstmt.executeBatch();
conn.commit();
conn.close();
conn = JDBCUtil.getConnection();
conn.setAutoCommit(false);
pstmt = conn.prepareStatement(sql);
}
System.out.println("insert "+count+"line");
}
if(count%1000!=0){
pstmt.executeBatch();
conn.commit();
}
long end = System.currentTimeMillis();
System.out.println("Total time spent:"+(end-start));
} catch (Exception e) {
e.printStackTrace();
}finally{
try {
pstmt.close();
conn.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
}
//getConnection()//get jdbc Connection
public static Connection getConnection(){
try {
Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
try {
conn = DriverManager.getConnection(url, userName, password);
} catch (SQLException e) {
e.printStackTrace();
}
return conn;
}
Speak for the first time, I hope I can help
I am the demo above use PreparedStatement [Read data calls a PreparedStatement one-off inserted]
JDBC batch There are 3 ways
1.use PreparedStatement
demo:
try {
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(o_url, userName, password);
conn.setAutoCommit(false);
String sql = "INSERT adlogs(ip,website,yyyymmdd,hour,object_id) VALUES(?,?,?,?,?)";
PreparedStatement prest = conn.prepareStatement(sql,ResultSet.TYPE_SCROLL_SENSITIVE,ResultSet.CONCUR_READ_ONLY);
for(int x = 0; x < size; x++){
prest.setString(1, "192.168.1.1");
prest.setString(2, "localhost");
prest.setString(3, "20081009");
prest.setInt(4, 8);
prest.setString(5, "11111111");
prest.addBatch();
}
prest.executeBatch();
conn.commit();
conn.close();
} catch (SQLException ex) {
Logger.getLogger(MyLogger.class.getName()).log(Level.SEVERE, null, ex);
}
2.use Statement.addBatch methods
demo:
conn.setAutoCommit(false);
Statement stmt = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_READ_ONLY);
for(int x = 0; x < size; x++){
stmt.addBatch("INSERT INTO adlogs(ip,website,yyyymmdd,hour,object_id) VALUES('192.168.1.3', 'localhost','20081009',8,'23123')");
}
stmt.executeBatch();
conn.commit();
3.Direct use of the Statement
demo:
conn.setAutoCommit(false);
Statement stmt = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE,
ResultSet.CONCUR_READ_ONLY);
for(int x = 0; x < size; x++){
stmt.execute("INSERT INTO adlogs(ip,website,yyyymmdd,hour,object_id) VALUES('192.168.1.3', 'localhost','20081009',8,'23123')");
}
conn.commit();
Using the above method Insert the 100000 pieces of data Time consuming:
method 1:17.844s
method 2:18.421s
method 3:16.359s
MS JDBC versions later than 4.1 have SQLServerBulkCopy class that I assume is equivalent to one available in .Net and theoretically it should work as fast as bcp command line utility.
https://msdn.microsoft.com/en-us/library/mt221490%28v=sql.110%29.aspx
you can custom your code with JDBC,there is no framework support your requirement

Resources