Exception while executing insert query having some special characters in the value - npgsql

I have a column with character varying datatype. We are using Npgsql -Version 4.0.4 and PostgreSQL version 11.3
String value = "Shaun Hollis_002\\Top of Personal Folders\002\\� 35% Off
Harlem Globetrotters - 3 Events in Denver!.msg";
try
{
StringBuilder sb = new StringBuilder();
sb.AppendLine("Insert INTo tbltemptest1 (c_value) values(#value);");
using (NpgsqlConnection con = new NpgsqlConnection(DBConnectionString))
{
con.Open();
using (NpgsqlCommand command = new NpgsqlCommand(sb.ToString(), con))
{
command.Parameters.Add("#value", NpgsqlTypes.NpgsqlDbType.Varchar).Value = value;
command.ExecuteNonQuery();
}
}
}
catch (Exception ex)
{
throw;
}
Performing the insert query is throwing an exception
invalid byte sequence for encoding "UTF8": 0x00


The error pretty much say it: PostgreSQL refuses to accept strings with 0x00 (the null character) inside them. You will need to clean up your input before sending it to PostgreSQL.

Related

How to use scope_identity in jdbc [duplicate]

I want to INSERT a record in a database (which is Microsoft SQL Server in my case) using JDBC in Java. At the same time, I want to obtain the insert ID. How can I achieve this using JDBC API?

If it is an auto generated key, then you can use Statement#getGeneratedKeys() for this. You need to call it on the same Statement as the one being used for the INSERT. You first need to create the statement using Statement.RETURN_GENERATED_KEYS to notify the JDBC driver to return the keys.
Here's a basic example:
public void create(User user) throws SQLException {
try (
Connection connection = dataSource.getConnection();
PreparedStatement statement = connection.prepareStatement(SQL_INSERT,
Statement.RETURN_GENERATED_KEYS);
) {
statement.setString(1, user.getName());
statement.setString(2, user.getPassword());
statement.setString(3, user.getEmail());
// ...
int affectedRows = statement.executeUpdate();
if (affectedRows == 0) {
throw new SQLException("Creating user failed, no rows affected.");
}
try (ResultSet generatedKeys = statement.getGeneratedKeys()) {
if (generatedKeys.next()) {
user.setId(generatedKeys.getLong(1));
}
else {
throw new SQLException("Creating user failed, no ID obtained.");
}
}
}
}
Note that you're dependent on the JDBC driver as to whether it works. Currently, most of the last versions will work, but if I am correct, Oracle JDBC driver is still somewhat troublesome with this. MySQL and DB2 already supported it for ages. PostgreSQL started to support it not long ago. I can't comment about MSSQL as I've never used it.
For Oracle, you can invoke a CallableStatement with a RETURNING clause or a SELECT CURRVAL(sequencename) (or whatever DB-specific syntax to do so) directly after the INSERT in the same transaction to obtain the last generated key. See also this answer.

Create Generated Column
String generatedColumns[] = { "ID" };
Pass this geneated Column to your statement
PreparedStatement stmtInsert = conn.prepareStatement(insertSQL, generatedColumns);
Use ResultSet object to fetch the GeneratedKeys on Statement
ResultSet rs = stmtInsert.getGeneratedKeys();
if (rs.next()) {
long id = rs.getLong(1);
System.out.println("Inserted ID -" + id); // display inserted record
}

When encountering an 'Unsupported feature' error while using Statement.RETURN_GENERATED_KEYS, try this:
String[] returnId = { "BATCHID" };
String sql = "INSERT INTO BATCH (BATCHNAME) VALUES ('aaaaaaa')";
PreparedStatement statement = connection.prepareStatement(sql, returnId);
int affectedRows = statement.executeUpdate();
if (affectedRows == 0) {
throw new SQLException("Creating user failed, no rows affected.");
}
try (ResultSet rs = statement.getGeneratedKeys()) {
if (rs.next()) {
System.out.println(rs.getInt(1));
}
rs.close();
}
Where BATCHID is the auto generated id.

I'm hitting Microsoft SQL Server 2008 R2 from a single-threaded JDBC-based application and pulling back the last ID without using the RETURN_GENERATED_KEYS property or any PreparedStatement. Looks something like this:
private int insertQueryReturnInt(String SQLQy) {
ResultSet generatedKeys = null;
int generatedKey = -1;
try {
Statement statement = conn.createStatement();
statement.execute(SQLQy);
} catch (Exception e) {
errorDescription = "Failed to insert SQL query: " + SQLQy + "( " + e.toString() + ")";
return -1;
}
try {
generatedKey = Integer.parseInt(readOneValue("SELECT ##IDENTITY"));
} catch (Exception e) {
errorDescription = "Failed to get ID of just-inserted SQL query: " + SQLQy + "( " + e.toString() + ")";
return -1;
}
return generatedKey;
}
This blog post nicely isolates three main SQL Server "last ID" options:
http://msjawahar.wordpress.com/2008/01/25/how-to-find-the-last-identity-value-inserted-in-the-sql-server/ - haven't needed the other two yet.

Instead of a comment, I just want to answer post.
Interface java.sql.PreparedStatement
columnIndexes « You can use prepareStatement function that accepts columnIndexes and SQL statement.
Where columnIndexes allowed constant flags are Statement.RETURN_GENERATED_KEYS1 or Statement.NO_GENERATED_KEYS[2], SQL statement that may contain one or more '?' IN parameter placeholders.
SYNTAX «
Connection.prepareStatement(String sql, int autoGeneratedKeys)
Connection.prepareStatement(String sql, int[] columnIndexes)
Example:
PreparedStatement pstmt =
conn.prepareStatement( insertSQL, Statement.RETURN_GENERATED_KEYS );
columnNames « List out the columnNames like 'id', 'uniqueID', .... in the target table that contain the auto-generated keys that should be returned. The driver will ignore them if the SQL statement is not an INSERT statement.
SYNTAX «
Connection.prepareStatement(String sql, String[] columnNames)
Example:
String columnNames[] = new String[] { "id" };
PreparedStatement pstmt = conn.prepareStatement( insertSQL, columnNames );
Full Example:
public static void insertAutoIncrement_SQL(String UserName, String Language, String Message) {
String DB_URL = "jdbc:mysql://localhost:3306/test", DB_User = "root", DB_Password = "";
String insertSQL = "INSERT INTO `unicodeinfo`( `UserName`, `Language`, `Message`) VALUES (?,?,?)";
//"INSERT INTO `unicodeinfo`(`id`, `UserName`, `Language`, `Message`) VALUES (?,?,?,?)";
int primkey = 0 ;
try {
Class.forName("com.mysql.jdbc.Driver").newInstance();
Connection conn = DriverManager.getConnection(DB_URL, DB_User, DB_Password);
String columnNames[] = new String[] { "id" };
PreparedStatement pstmt = conn.prepareStatement( insertSQL, columnNames );
pstmt.setString(1, UserName );
pstmt.setString(2, Language );
pstmt.setString(3, Message );
if (pstmt.executeUpdate() > 0) {
// Retrieves any auto-generated keys created as a result of executing this Statement object
java.sql.ResultSet generatedKeys = pstmt.getGeneratedKeys();
if ( generatedKeys.next() ) {
primkey = generatedKeys.getInt(1);
}
}
System.out.println("Record updated with id = "+primkey);
} catch (InstantiationException | IllegalAccessException | ClassNotFoundException | SQLException e) {
e.printStackTrace();
}
}

I'm using SQLServer 2008, but I have a development limitation: I cannot use a new driver for it, I have to use "com.microsoft.jdbc.sqlserver.SQLServerDriver" (I cannot use "com.microsoft.sqlserver.jdbc.SQLServerDriver").
That's why the solution conn.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS) threw a java.lang.AbstractMethodError for me.
In this situation, a possible solution I found is the old one suggested by Microsoft:
How To Retrieve ##IDENTITY Value Using JDBC
import java.sql.*;
import java.io.*;
public class IdentitySample
{
public static void main(String args[])
{
try
{
String URL = "jdbc:microsoft:sqlserver://yourServer:1433;databasename=pubs";
String userName = "yourUser";
String password = "yourPassword";
System.out.println( "Trying to connect to: " + URL);
//Register JDBC Driver
Class.forName("com.microsoft.jdbc.sqlserver.SQLServerDriver").newInstance();
//Connect to SQL Server
Connection con = null;
con = DriverManager.getConnection(URL,userName,password);
System.out.println("Successfully connected to server");
//Create statement and Execute using either a stored procecure or batch statement
CallableStatement callstmt = null;
callstmt = con.prepareCall("INSERT INTO myIdentTable (col2) VALUES (?);SELECT ##IDENTITY");
callstmt.setString(1, "testInputBatch");
System.out.println("Batch statement successfully executed");
callstmt.execute();
int iUpdCount = callstmt.getUpdateCount();
boolean bMoreResults = true;
ResultSet rs = null;
int myIdentVal = -1; //to store the ##IDENTITY
//While there are still more results or update counts
//available, continue processing resultsets
while (bMoreResults || iUpdCount!=-1)
{
//NOTE: in order for output parameters to be available,
//all resultsets must be processed
rs = callstmt.getResultSet();
//if rs is not null, we know we can get the results from the SELECT ##IDENTITY
if (rs != null)
{
rs.next();
myIdentVal = rs.getInt(1);
}
//Do something with the results here (not shown)
//get the next resultset, if there is one
//this call also implicitly closes the previously obtained ResultSet
bMoreResults = callstmt.getMoreResults();
iUpdCount = callstmt.getUpdateCount();
}
System.out.println( "##IDENTITY is: " + myIdentVal);
//Close statement and connection
callstmt.close();
con.close();
}
catch (Exception ex)
{
ex.printStackTrace();
}
try
{
System.out.println("Press any key to quit...");
System.in.read();
}
catch (Exception e)
{
}
}
}
This solution worked for me!
I hope this helps!

You can use following java code to get new inserted id.
ps = con.prepareStatement(query, Statement.RETURN_GENERATED_KEYS);
ps.setInt(1, quizid);
ps.setInt(2, userid);
ps.executeUpdate();
ResultSet rs = ps.getGeneratedKeys();
if (rs.next()) {
lastInsertId = rs.getInt(1);
}

It is possible to use it with normal Statement's as well (not just PreparedStatement)
Statement statement = conn.createStatement();
int updateCount = statement.executeUpdate("insert into x...)", Statement.RETURN_GENERATED_KEYS);
try (ResultSet generatedKeys = statement.getGeneratedKeys()) {
if (generatedKeys.next()) {
return generatedKeys.getLong(1);
}
else {
throw new SQLException("Creating failed, no ID obtained.");
}
}

Most others have suggested to use JDBC API for this, but personally, I find it quite painful to do with most drivers. When in fact, you can just use a native T-SQL feature, the OUTPUT clause:
try (
Statement s = c.createStatement();
ResultSet rs = s.executeQuery(
"""
INSERT INTO t (a, b)
OUTPUT id
VALUES (1, 2)
"""
);
) {
while (rs.next())
System.out.println("ID = " + rs.getLong(1));
}
This is the simplest solution for SQL Server as well as a few other SQL dialects (e.g. Firebird, MariaDB, PostgreSQL, where you'd use RETURNING instead of OUTPUT).
I've blogged about this topic more in detail here.

With Hibernate's NativeQuery, you need to return a ResultList instead of a SingleResult, because Hibernate modifies a native query
INSERT INTO bla (a,b) VALUES (2,3) RETURNING id
like
INSERT INTO bla (a,b) VALUES (2,3) RETURNING id LIMIT 1
if you try to get a single result, which causes most databases (at least PostgreSQL) to throw a syntax error. Afterwards, you may fetch the resulting id from the list (which usually contains exactly one item).

In my case ->
ConnectionClass objConnectionClass=new ConnectionClass();
con=objConnectionClass.getDataBaseConnection();
pstmtGetAdd=con.prepareStatement(SQL_INSERT_ADDRESS_QUERY,Statement.RETURN_GENERATED_KEYS);
pstmtGetAdd.setString(1, objRegisterVO.getAddress());
pstmtGetAdd.setInt(2, Integer.parseInt(objRegisterVO.getCityId()));
int addId=pstmtGetAdd.executeUpdate();
if(addId>0)
{
ResultSet rsVal=pstmtGetAdd.getGeneratedKeys();
rsVal.next();
addId=rsVal.getInt(1);
}

If you are using Spring JDBC, you can use Spring's GeneratedKeyHolder class to get the inserted ID.
See this answer...
How to get inserted id using Spring Jdbctemplate.update(String sql, obj...args)

If you are using JDBC (tested with MySQL) and you just want the last inserted ID, there is an easy way to get it. The method I'm using is the following:
public static Integer insert(ConnectionImpl connection, String insertQuery){
Integer lastInsertId = -1;
try{
final PreparedStatement ps = connection.prepareStatement(insertQuery);
ps.executeUpdate(insertQuery);
final com.mysql.jdbc.PreparedStatement psFinal = (com.mysql.jdbc.PreparedStatement) ps;
lastInsertId = (int) psFinal.getLastInsertID();
connection.close();
} catch(SQLException ex){
System.err.println("Error: "+ex);
}
return lastInsertId;
}
Also, (and just in case) the method to get the ConnectionImpl is the following:
public static ConnectionImpl getConnectionImpl(){
ConnectionImpl conexion = null;
final String dbName = "database_name";
final String dbPort = "3306";
final String dbIPAddress = "127.0.0.1";
final String connectionPath = "jdbc:mysql://"+dbIPAddress+":"+dbPort+"/"+dbName+"?autoReconnect=true&useSSL=false";
final String dbUser = "database_user";
final String dbPassword = "database_password";
try{
conexion = (ConnectionImpl) DriverManager.getConnection(connectionPath, dbUser, dbPassword);
}catch(SQLException e){
System.err.println(e);
}
return conexion;
}
Remember to add the connector/J to the project referenced libraries.
In my case, the connector/J version is the 5.1.42. Maybe you will have to apply some changes to the connectionPath if you want to use a more modern version of the connector/J such as with the version 8.0.28.
In the file, remember to import the following resources:
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import com.mysql.jdbc.ConnectionImpl;
Hope this will be helpful.

Connection cn = DriverManager.getConnection("Host","user","pass");
Statement st = cn.createStatement("Ur Requet Sql");
int ret = st.execute();

Unexpected end of JSON input: asp.net core

when select query
select * from Permissions where AppId='yZVwUoxKQCu' FOR JSON PATH,INCLUDE_NULL_VALUES
executes in sql server direct it return full data with array wrapper without unexpected end.when the query read with DataReader in C# it the same query result have unexpected end. the sql execute using below the method.
Query like this
var sql = $"select * from Permissions where AppId='{AppId}' FOR JSON PATH,INCLUDE_NULL_VALUES";
var res = Connection.ExecuteScalarCommand(sql);
public static String ExecuteScalarCommand(string sql)
{
string CS = DbConnectionString;
SqlConnection con = new SqlConnection(CS);
string val = "";
try
{
con.Open();
SqlCommand cmd = new SqlCommand(sql, con);
val = cmd.ExecuteScalar().ToString();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message.ToString());
}
finally
{
con.Close();
}
return val;
}
The output given by sql server direct is given.its expected output.
[{"Id":49,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Season","UserId":60,"PermissionTitleId":1,"AppId":"yZVwUoxKQCu"},{"Id":50,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Categories","UserId":60,"PermissionTitleId":2,"AppId":"yZVwUoxKQCu"},{"Id":51,"IsCreatable":2,"IsViewable":2,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Tabs","UserId":60,"PermissionTitleId":3,"AppId":"yZVwUoxKQCu"},{"Id":52,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Fields","UserId":60,"PermissionTitleId":4,"AppId":"yZVwUoxKQCu"},{"Id":53,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"ContentBlock","UserId":60,"PermissionTitleId":5,"AppId":"yZVwUoxKQCu"},{"Id":54,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"User","UserId":60,"PermissionTitleId":6,"AppId":"yZVwUoxKQCu"},{"Id":55,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Roles","UserId":60,"PermissionTitleId":7,"AppId":"yZVwUoxKQCu"},{"Id":56,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Season","UserId":60,"PermissionTitleId":1,"AppId":"yZVwUoxKQCu"},{"Id":57,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Categories","UserId":60,"PermissionTitleId":2,"AppId":"yZVwUoxKQCu"},{"Id":58,"IsCreatable":2,"IsViewable":2,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Tabs","UserId":60,"PermissionTitleId":3,"AppId":"yZVwUoxKQCu"},{"Id":59,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Fields","UserId":60,"PermissionTitleId":4,"AppId":"yZVwUoxKQCu"},{"Id":60,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"ContentBlock","UserId":60,"PermissionTitleId":5,"AppId":"yZVwUoxKQCu"},{"Id":61,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"User","UserId":60,"PermissionTitleId":6,"AppId":"yZVwUoxKQCu"},{"Id":62,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Roles","UserId":60,"PermissionTitleId":7,"AppId":"yZVwUoxKQCu"},{"Id":42,"IsCreatable":0,"IsViewable":0,"IsDeletable":1,"IsUpdatable":0,"RoleId":28,"Title":"Season","UserId":60,"PermissionTitleId":1,"AppId":"yZVwUoxKQCu"},{"Id":43,"IsCreatable":1,"IsViewable":0,"IsDeletable":1,"IsUpdatable":1,"RoleId":28,"Title":"Categories","UserId":60,"PermissionTitleId":2,"AppId":"yZVwUoxKQCu"},{"Id":44,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":28,"Title":"Tabs","UserId":60,"PermissionTitleId":3,"AppId":"yZVwUoxKQCu"},{"Id":45,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":28,"Title":"Fields","UserId":60,"PermissionTitleId":4,"AppId":"yZVwUoxKQCu"},{"Id":46,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":28,"Title":"ContentBlock","UserId":60,"PermissionTitleId":5,"AppId":"yZVwUoxKQCu"},{"Id":47,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":28,"Title":"User","UserId":60,"PermissionTitleId":6,"AppId":"yZVwUoxKQCu"},{"Id":48,"IsCreatable":0,"IsViewable":0,"IsDeletable":1,"IsUpdatable":0,"RoleId":28,"Title":"Roles","UserId":60,"PermissionTitleId":7,"AppId":"yZVwUoxKQCu"}]
but out put from asp.net core is
[{"Id":49,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Season","UserId":60,"PermissionTitleId":1,"AppId":"yZVwUoxKQCu"},{"Id":50,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Categories","UserId":60,"PermissionTitleId":2,"AppId":"yZVwUoxKQCu"},{"Id":51,"IsCreatable":2,"IsViewable":2,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Tabs","UserId":60,"PermissionTitleId":3,"AppId":"yZVwUoxKQCu"},{"Id":52,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Fields","UserId":60,"PermissionTitleId":4,"AppId":"yZVwUoxKQCu"},{"Id":53,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"ContentBlock","UserId":60,"PermissionTitleId":5,"AppId":"yZVwUoxKQCu"},{"Id":54,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"User","UserId":60,"PermissionTitleId":6,"AppId":"yZVwUoxKQCu"},{"Id":55,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":29,"Title":"Roles","UserId":60,"PermissionTitleId":7,"AppId":"yZVwUoxKQCu"},{"Id":56,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Season","UserId":60,"PermissionTitleId":1,"AppId":"yZVwUoxKQCu"},{"Id":57,"IsCreatable":0,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Categories","UserId":60,"PermissionTitleId":2,"AppId":"yZVwUoxKQCu"},{"Id":58,"IsCreatable":2,"IsViewable":2,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Tabs","UserId":60,"PermissionTitleId":3,"AppId":"yZVwUoxKQCu"},{"Id":59,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"Fields","UserId":60,"PermissionTitleId":4,"AppId":"yZVwUoxKQCu"},{"Id":60,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"ContentBlock","UserId":60,"PermissionTitleId":5,"AppId":"yZVwUoxKQCu"},{"Id":61,"IsCreatable":1,"IsViewable":1,"IsDeletable":1,"IsUpdatable":1,"RoleId":30,"Title":"User","UserId":60,"PermissionT

The ExecuteScalar has a character limit:
The first column of the first row in the result set, or a null
reference (Nothing in Visual Basic) if the result set is empty.
Returns a maximum of 2033 characters.
To overcome that you will need to use a different function, like ExecuteReader.

SqlServer Converting XML to varbinary and parsing it in .NET (C#)

Consider the following code:
[Test]
public void StackOverflowQuestionTest()
{
const string connectionString = "enter your connection string if you wanna test this code";
byte[] result = null;
using (var connection = new SqlConnection(connectionString))
{
connection.Open();
using (var sqlCommand = new SqlCommand("declare #xml as xml = '<xml/>' SELECT convert(varbinary(max), #xml) as value"))
//using (var sqlCommand = new SqlCommand("SELECT convert(varbinary(max), N'<xml/>') as value"))
{
sqlCommand.Connection = connection;
using (SqlDataReader reader = sqlCommand.ExecuteReader())
{
while (reader.Read())
{
result = (byte[])reader["value"];
}
reader.Close();
}
}
}
string decodedString = new UnicodeEncoding(false, true).GetString(result);
var document = XElement.Parse(decodedString);
}
If I run this test I get an XmlException with message : "Data at the root level is invalid. Line 1, position 1." As it turns out the problem is "0xFFFE" preamble which is considered as invalid character.
Note that if I use commented string instead, everything works just fine, which is strange as per me. Looks like SqlServer stores XML strings in UCS-2 with a BOM, and at the same time it stores nvarchar values without it.
The main question is: how can I decode this byte array to string which will not contain this preamble (BOM)?

In case anyone will need this in future, the following code works:
using(var ms = new MemoryStream(result))
{
using (var sr = new StreamReader(ms, Encoding.Unicode, true))
{
decodedString = sr.ReadToEnd();
}
}

.NET: How to insert XML document into SQL Server

I want to insert arbitrary XML into SQL Server. The XML is contained in an XmlDocument object.
The column I want to insert into is either nvarchar, ntext, or xml column (If it makes your life easier then you can pick which type it is. Really it's an xml column.)
Prototype
void SaveXmlToDatabase(DbConnection connection,
XmlDocument xmlToSave,
String tableName, String columnName);
{
}
The reason I ask is because I'm trying to find the proper way to turn the XmlDocument into something the database can take - being sure to keep the encodings right:
I have to make sure the encoding using during insert matches what the database takes
I have to synchronize the <?xml version="1.0" encoding="windows-1252"?> element
I know ntext, nvarchar, or xml are stored as UTF-16 inside SQL Server. So I have to be sure to give the data to SQL Server as UTF-16. This isn't a problem for Strings in .NET, since they are unicode UTF-16.
The 2nd problem, synchronizing the encoding attribute, is a tougher nut to crack. I have to figure out how to find the declaration element through the XmlDocument object:
<?xml version="1.0" encoding="windows-1252"?> (or whatever the encoding may be)
and adjust it to UTF-16
<?xml version="1.0" encoding="UTF-16"?>
My naive attempt (that fails)
Ignoring the encoding in the XML declaration, and just figuring out how to save anything into SQL Server:
void SaveXmlToDatabase(DbConnection connection,
XmlDocument xmlToSave,
String tableName, String columnName);
{
String sql = "INSERT INTO "+tableName+" ("+columnName+")
VALUES ('"+xmlToSave.ToString()+"')";
using (DbCommand command = connection.CreateCommand())
{
command.CommandText = sql;
DbTransaction trans = connection.BeginTransaction();
try
{
command.ExecuteNonQuery();
trans.Commit();
}
catch (Exception)
{
trans.Rollback();
throw;
}
}
}
This fails because the sql I try to run is:
INSERT INTO LiveData (RawXML)
VALUES ('System.Xml.XmlDocument')
This is because XmlDocument.ToString() returns "System.Xml.XmlDocument". Peeking at the implementation, it see that it literally is calling:
this.GetType().ToString();
Aside: Microsoft seems to have gone out of their way to prevent you
from getting the Xml as a string -
presumably because it leads to bugs
(But they don't tell us what bugs, why
they're bugs, or the right way to
convert an XmlDocument into a
String!)
See also
C#/SQL - What’s wrong with SqlDbType.Xml in procedures ?
How to convert XmlDocument to XmlReader for SqlXml data type.

You have to use an SqlParameter.
I would recommend doing it like that:
command.Parameters.Add(
new SqlParameter("#xml", SqlDbType.Xml)
{Value = new SqlXml(new XmlTextReader(xmlToSave.InnerXml
, XmlNodeType.Document, null)) })
and the SQL should look like:
String sql = "INSERT INTO "+tableName+" ("+columnName+") VALUES (#xml)";
And since the first child is always the xml node, you can replace the encoding with the following statement.
xmlToSave.FirstChild.InnerText = "version=\"1.0\" encoding=\"UTF-16\"";
All in all it would look like that:
void SaveXmlToDatabase(DbConnection connection,
XmlDocument xmlToSave,
String tableName, String columnName);
{
String sql = "INSERT INTO "+tableName+" ("+columnName+") VALUES (#xml)";
using (DbCommand command = connection.CreateCommand())
{
xmlToSave.FirstChild.InnerText = "version=\"1.0\" encoding=\"UTF-16\"";
command.CommandText = sql;
command.Parameters.Add(
new SqlParameter("#xml", SqlDbType.Xml)
{Value = new SqlXml(new XmlTextReader(xmlToSave.InnerXml
, XmlNodeType.Document, null)) });
DbTransaction trans = connection.BeginTransaction();
try
{
command.ExecuteNonQuery();
trans.Commit();
}
catch (Exception)
{
trans.Rollback();
throw;
}
}
}

The simplest solution is to use the OuterXml attribute of the document. I came across your question while trying to solve the problem under the condition that I could not executed a stored procedure. OuterXml returns the string of text you were expecting to get from .Value or .ToString()
void SaveXmlToDatabase(DbConnection connection,
XmlDocument xmlToSave,
String tableName, String columnName);
{
String sql = "INSERT INTO "+tableName+" ("+columnName+")
VALUES ('"+xmlToSave.OuterXml+"')";
using (DbCommand command = connection.CreateCommand())
{
command.CommandText = sql;
DbTransaction trans = connection.BeginTransaction();
try
{
command.ExecuteNonQuery();
trans.Commit();
}
catch (Exception)
{
trans.Rollback();
throw;
}
}
}

Better late than never... I think you're looking for something like this:
void SaveXmlToDatabase(DbConnection connection,
XmlDocument xmlToSave,
String tableName, String columnName)
{
String sql = "INSERT INTO "+tableName+" ("+columnName+")
VALUES (#XmlVal)";
using (DbCommand command = connection.CreateCommand())
{
command.CommandText = sql;
command.Parameters.AddWithValue("XmlVal", new SqlXml(new XmlNodeReader(xmlToSave)));
DbTransaction trans = connection.BeginTransaction();
try
{
command.ExecuteNonQuery();
trans.Commit();
}
catch (Exception)
{
trans.Rollback();
throw;
}
}
}
The XmlNodeReader object traverses and properly encodes the XmlDocument (or any other XmlNode), and the SqlXml object encapsulates that into the SqlDbType suitable for use with the parameter. This is safer and probably more efficient than using a string intermediary.

Just wanted to add an SQLConnection equivalent to Jonathan Overholt's solution:
private void StoreXMLInDatabase(XmlDocument doc)
{
// xmlvalue is the name of the database column you want to insert into
String sql = "INSERT INTO [tablename] (xmlvalue) VALUES (#xmlvalue)";
// In order to read out you connection string from app.config, remember first to add a reference to System.Configuration in references,
// and set using System.Configuration; in the top of the file
string connString = ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString;
using (SqlConnection conn = new SqlConnection(connString))
{
if (conn.State == ConnectionState.Closed)
{
conn.Open();
}
SqlTransaction transaction = conn.BeginTransaction();
try
{
// Remember to specify the SqlTransaction *name* (transaction)
SqlCommand cobj = new SqlCommand(sql, conn, transaction);
cobj.CommandText = sql;
cobj.Parameters.AddWithValue("xmlvalue", new SqlXml(new XmlNodeReader(doc)));
cobj.ExecuteNonQuery();
transaction.Commit();
}
catch (SqlException ex)
{
//2627 = inserting duplicate key fail. Lets the procedure continue although an identity insert fail occurs
if (ex.Number == 2627)
{
transaction.Rollback();
}
}
} // end using
}
Tested OK in Visual studio 2017 and 2013 with .net up to version 4.6.2, and against MS SQL Server 2008 R2, 2012 and 2016
If an identity insert fail and you want to stop the procedure then, just insert a line after transaction.Rollback() like this:
throw;
Or you can stop the procedure in any circumstance by putting the transaction.Rollback(); and throw; lines just outside the conditional (the if)

Java Unicode problem

My question would be what's wrong with the next code? I'm trying with j2ee to read some unicode from a database and some characters are returned as the famous question mark.
try
{
Class.forName("com.mysql.jdbc.Driver");
String connectionUrl = "jdbc:mysql://localhost/hortimart?" +
"user=webservices&password=stipjeservers";
Connection con = DriverManager.getConnection(connectionUrl);
Statement stmt = null;
ResultSet rs = null;
String SQL = "SELECT FirstName,LastName FROM users";
stmt = con.createStatement();
rs = stmt.executeQuery(SQL);
while (rs.next())
{
byte[] firstNameBytes = rs.getBytes(1);
String FirstName =new String(firstNameBytes,"UTF-8");
byte [] lastNameBytes = rs.getBytes(2);
String LastName =new String(lastNameBytes,"UTF-8");
System.out.println(FirstName+" "+LastName);
}
}
catch (SQLException e)
{
System.out.println("SQL Exception: "+ e.toString());
}
catch (ClassNotFoundException cE)
{
System.out.println("Class Not Found Exception: "+ cE.toString());
}
Now i've tried this code with j2Se as well and it works. So is it j2EE or i missed something in my code?
Thanks

It's most likely your console that's misconfigured. Have you examined the bytes of the result to see if it's UTF-8?

try to append characterEncoding=utf-8 to your connection url.
ie:
String connectionUrl = "jdbc:mysql://localhost/hortimart?" + "user=webservices&password=stipjeservers&characterEncoding=utf-8

Do you know if the strings have been encoded in UTF-8 (Unicode encompasses many encodings, UTF-8 being only one of them). Have you tried using a simple rs.getString(1) ?

It could be that either i) you have data in the database that is not unicode (how are your tables defined?) or ii) that your driver is not unicode-enabled.
EDIT: Have you tried setting both useUnicode and characterEncoding in your connection string?
...useUnicode=true&characterEncoding=utf8

Resources