SqlServer better to batch statements or foreach? - sql-server

Hypothetically, is it better to send N statements to Sql Server (2008), or is it better to send 1 command comprising N statements to Sql Server? In either case, I am running the same statement over a list of objects, and in both cases I would be using named parameters. Suppose my use case is dumping a cache of log items every few hours.
foreach example
var sql = "update blah blah blah where id = #id";
using(var conn = GetConnection())
{
foreach(var obj in myList)
{
var cmd = new SqlCommand()
{CommandText = sql, Connection = conn};
//add params from obj
cmd.ExecuteNonQuery();
}
}
batch example
var sql = #"
update blah blah blah where id = #id1
update blah blah blah where id = #id2
update blah blah blah where id = #id3
-etc";
using (var conn = GetConnection())
{
var cmd = new SqlCommand
{ CommandText = sql, Connection = conn};
for(int i=0; i<myList.Count; i++)
{
//add params: "id" + i from myList[i]
}
cmd.ExecuteNonQuery();
}
In time tests, the batch version took 15% longer than the foreach version for large inputs. I figure the batch version takes longer to execute because the server has to parse a huge statement and bind up to 2000 parameters. Supposing Sql Server is on the LAN, is there any advantage to using the batch method?

Your tests would seem to have given you the answer however let me add another. It is preferrable to encapsulate the update into a separate function and call that using a foreach:
private function UpdateFoo( int id )
{
const sql = "Update Foo Where Id = #Id";
using ( var conn = GetConnection() )
{
using ( var cmd = new SqlCommand( sql, conn ) )
{
cmd.AddParameterWithValue( "#Id", id )
cmd.ExecuteNonQuery();
}
}
}
private function UpdateLotsOfFoo()
{
foreach( var foo in myList )
{
UpdateFoo( foo.Id );
}
}
In this setup you are leveraging connection pooling which mitgates the cost of opening and closing connections.

#Thomas - this design can increase overhead of opening / closing connections in a loop. This is not a preferred practice and should be avoided. The code below allows the iteration of the statements while using one connection and will be easier on resources (both client and server side).
private void UpdateFoo(int id)
{
const string sql = "Update Foo Where Id = #Id";
using (var conn = GetConnection())
{
conn.Open();
foreach (var foo in myList)
{
UpdateFoo(foo.Id);
using (var cmd = new SqlCommand(sql, conn))
{
cmd.AddParameterWithValue("#Id", id);
cmd.ExecuteNonQuery();
}
}
conn.Close();
}
}

Related

Preparing statements and batching in npgsql

The Simple Preparation example in the docs (https://www.npgsql.org/doc/prepare.html#simple-preparation) shows an example where parameters are set after the command is prepared.
var cmd = new NpgsqlCommand(...);
cmd.Parameters.Add("param", NpgsqlDbType.Integer);
cmd.Prepare();
// Set parameters
cmd.ExecuteNonQuery();
// And so on
Questions
How are the parameters set?
Is it possible to use AddWithValue instead of Add if the AddWithValue(String, NpgsqlDbType, Object) method which specifies NpgsqlDbType is used -- docs say "setting the value isn't support"?
How does this work if multiple statements exist in the same command?
This answer (https://stackoverflow.com/a/53268090/10984827) shows that multiple commands in a single string can be prepared together but it's not clear how this CommandText string is created.
Edit: I think I'm almost there but I'm not sure how to create and execute the batched the query string. Here's my naive attempt at building a batched query using a StringBuilder. This doesn't work. How do I do this correctly?
using System;
using System.Collections.Generic;
using System.Text;
using Npgsql;
using NpgsqlTypes;
class Model
{
public int value1 { get; }
public int value2 { get; }
public Model(int value1, int value2)
{
this.value1 = value1;
this.value2 = value2;
}
}
class Program
{
static void Main(string[] args)
{
var dataRows = new List<Model>();
dataRows.Add(new Model(3,2));
dataRows.Add(new Model(27,-10));
dataRows.Add(new Model(11,-11));
var connString = "Host=127.0.0.1;Port=5432;Username=postgres;Database=dbtest1";
// tabletest1
// ----------
// id SERIAL PRIMARY KEY
// , value1 INT NOT NULL
// , value2 INT NOT NULL
using (var conn = new NpgsqlConnection(connString))
{
conn.Open();
var cmd = new NpgsqlCommand();
cmd.Connection = conn;
cmd.CommandText = $"INSERT INTO tabletest1 (value1,value2) VALUES (#value1,#value2)";
var parameterValue1 = cmd.Parameters.Add("value1", NpgsqlDbType.Integer);
var parameterValue2 = cmd.Parameters.Add("value2", NpgsqlDbType.Integer);
cmd.Prepare();
var batchCommand = new StringBuilder();
foreach (var d in dataRows)
{
parameterValue1.Value = d.value1;
parameterValue2.Value = d.value2;
batchCommand.Append(cmd.CommandText);
batchCommand.Append(";");
}
Console.WriteLine(batchCommand.ToString());
// conn.ExecuteNonQuery(batchCommand.ToString());
}
}
}
1) Simply capture the NpgsqlParameter returned from Add(), and then set its Value property:
var p = cmd.Parameters.Add("p", NpgsqlDbType.Integer);
cmd.Prepare();
p.Value = 8;
cmd.ExecuteNonQuery();
2) You can use AddWithValue() in the same way, but if you're preparing the command in order to reuse it several times, that makes less sense. The idea is that you first add the parameter without a value, then prepare, then execute it several times, setting the value each time.
3) You can prepare a multi-statement command. As things work now, all statements in the command will share the same parameter list (which lives on NpgsqlCommand). So the same pattern holds: create your command with your SQL and parameters, prepare it, and then set parameter values and execute. Each individual statement within your command will run prepared, benefiting from the perf increase.
Here's an example with two-statement batching:
cmd.CommandText = "INSERT INTO tabletest1 (value1,value2) VALUES (#v1,#v2); INSERT INTO tabletest1 (value1, value2) VALUES (#v3,#v4)";
var v1 = cmd.Parameters.Add("v1", NpgsqlDbType.Integer);
var v2 = cmd.Parameters.Add("v2", NpgsqlDbType.Integer);
var v3 = cmd.Parameters.Add("v3", NpgsqlDbType.Integer);
var v4 = cmd.Parameters.Add("v4", NpgsqlDbType.Integer);
cmd.Prepare();
while (...) {
v1.Value = ...;
v2.Value = ...;
v3.Value = ...;
v4.Value = ...;
cmd.ExecuteNonQuery();
}
However, if the objective is to efficiently insert lots of data, consider using COPY instead - it will be faster than even batched inserts.
Finally, to complete the picture, for INSERT statements specifically you can include more than one row in a single statement:
INSERT INTO tabletest1 (value1, value2) VALUES (1,2), (3,4)
You can also again parameterize the actual values, and prepare this command. This is similar to batching two INSERT statements, and should be faster (although still slower than COPY).
In NpgSQL 6.0 there has been the addition of batching/pipelining.
Here is an updated example:
await using var connection = new NpgsqlConnection(connString);
await connection.OpenAsync();
var batch = new NpgsqlBatch(connection);
const int count = 10;
const string parameterName = "parameter";
for (int i = 0; i < count; i++)
{
var batchCommand = new NpgsqlBatchCommand($"SELECT #{parameterName} as value");
batchCommand.Parameters.Add(new NpgsqlParameter(parameterName, i));
batch.BatchCommands.Add(batchCommand);
}
await batch.PrepareAsync();
var results = new List<int>(count);
await using (var reader = await batch.ExecuteReaderAsync())
{
do
{
while (await reader.ReadAsync())
{
results.Add(await reader.GetFieldValueAsync<int>("value"));
}
} while (await reader.NextResultAsync());
}
Console.WriteLine(string.Join(", ", results));

best solution for multiple insert update solution

Struggle with understanding C# & Npgsql as a beginner. Following code examples:
// Insert some data
using (var cmd = new NpgsqlCommand())
{ cmd.Connection = conn;
cmd.CommandText = "INSERT INTO data (some_field) VALUES (#p)";
cmd.Parameters.AddWithValue("p", "Hello world");
cmd.ExecuteNonQuery();
}
The syntax for more than one insert & update statement like this is clear so far:
cmd.CommandText = "INSERT INTO data (some_field) VALUES (#p);INSERT INTO data1...;INSERT into data2... and so on";
But what is the right solution for a loop which should handle one statement within.
This works not:
// Insert some data
using (var cmd = new NpgsqlCommand())
{
foreach(s in SomeStringCollectionOrWhatever)
{
cmd.Connection = conn;
cmd.CommandText = "INSERT INTO data (some_field) VALUES (#p)";
cmd.Parameters.AddWithValue("p", s);
cmd.ExecuteNonQuery();
}
}
It seems the values will be "concatenated" or remembered. I cannot see any possibility to "clear" the existing cmd-object.
My second solution would be to wrap the whole "using" block into the loop. But every cycle would create a new object. That seems ugly to me.
So what is the best solution for my problem?
To insert lots of rows efficiently, take a look at Npgsql's bulk copy feature - the API is more suitable (and more efficient) for inserting large numbers of rows than concatenating INSERT statements into a batch like you're trying to do.
If you want to rerun the same SQL with changing parameter values, you can do the following:
using (var cmd = new NpgsqlCommand("INSERT INTO data (some_field) VALUES (#p)", conn))
{
var p = new NpgsqlParameter("p", DbType.String); // Adjust DbType according to type
cmd.Parameters.Add(p);
cmd.Prepare(); // This is optional but will optimize the statement for repeated use
foreach(var s in SomeStringCollectionOrWhatever)
{
p.Value = s;
cmd.ExecuteNonQuery();
}
}
If you need lots of rows and performance is key then i would recommend Npgsql's bulk copy capability as #Shay mentioned. But if you are looking for quick way to do this without the bulk copy i would recommend to use Dapper.
Consider the example below.
Lets say you have a class called Event and a list of events to add.
List<Event> eventsToInsert = new List<Event>
{
new Event() { EventId = 1, EventName = "Bday1" },
new Event() { EventId = 2, EventName = "Bday2" },
new Event() { EventId = 3, EventName = "Bday3" }
};
The snippet that would add the list to the DB shown below.
var sqlInsert = "Insert into events( eventid, eventname ) values (#EventId, #EventName)";
using (IDbConnection conn = new NpgsqlConnection(cs))
{
conn.Open();
// Execute is an extension method supplied by Dapper
// This code will add all the entries in the eventsToInsert List and match up the values based on property name. Only caveat is that the property names of the POCO should match the placeholder names in the SQL Statement.
conn.Execute(sqlInsert, eventsToInsert);
// If we want to retrieve the data back into the list
List<Event> eventsAdded;
// This Dapper extension will return an Ienumerable, so i cast it to a List.
eventsAdded = conn.Query<Event>("Select * from events").ToList();
foreach( var row in eventsAdded)
{
Console.WriteLine($"{row.EventId} {row.EventName} was added");
}
}
-HTH

Using lock in Hangfire executed ASP.Net code

I am using hangfire in an ASP.Net MVC project to manage longrunning background job.
I am trying to use lock statement block for database operation. Here is my lock statement code-
public class LockedTransaction
{
private Object thisLock = new Object();
public LockedTransaction() { }
public void UpdateCustomerBalance(long CustomerId, decimal AmountToDeduct, string ConnectionString)
{
lock (thisLock)
{
using (SqlConnection connection = new SqlConnection(ConnectionString))
{
connection.Open();
using (SqlTransaction transaction = connection.BeginTransaction(System.Data.IsolationLevel.ReadCommitted))
{
using (SqlCommand command = new SqlCommand())
{
command.Connection = connection;
command.Transaction = transaction;
command.CommandText = "SELECT Balance FROM Customer WHERE Id=" + CustomerId;
var userBalance = Convert.ToDecimal(command.ExecuteScalar());
userBalance = userBalance - AmountToDeduct;
command.CommandText = "UPDATE Customer SET Balance=" + userBalance + " WHERE Id=" + CustomerId;
command.ExecuteNonQuery();
transaction.Commit();
}
}
}
}
}
}
Here is how I'm calling the above code-
foreach (var queue in queues)
{
queue.Send();
LockedTransaction lockedTransaction = new LockedTransaction();
lockedTransaction.UpdateCustomerBalance(queue.CustomerId, queue.cost, "ConnectionString");
}
The problem is, database value is not updated as expected. For example, I have 5 queues as follows -
queue[0].cost = 0.50;
queue[1].cost = 0.50;
queue[2].cost = 0.50;
queue[3].cost = 0.50;
queue[4].cost = 0.50;
Database value should be deducted 2.5 (cost total) after completing the loop. But it's not happening. Sometimes deducted value is 2.00, sometimes 1.5, etc.
Any suggestion?
Your lock object (thisLock) is an instance property. And cause in foreach loop you create a new instance of LockedTransaction for each element in the queue, lock doesn't prevent concurrent executions (each calling of UpdateCustomerBalance method uses own lock object).
Changing thisLock to static property should help you:
private static Object thisLock = new Object();

SqlServer Converting XML to varbinary and parsing it in .NET (C#)

Consider the following code:
[Test]
public void StackOverflowQuestionTest()
{
const string connectionString = "enter your connection string if you wanna test this code";
byte[] result = null;
using (var connection = new SqlConnection(connectionString))
{
connection.Open();
using (var sqlCommand = new SqlCommand("declare #xml as xml = '<xml/>' SELECT convert(varbinary(max), #xml) as value"))
//using (var sqlCommand = new SqlCommand("SELECT convert(varbinary(max), N'<xml/>') as value"))
{
sqlCommand.Connection = connection;
using (SqlDataReader reader = sqlCommand.ExecuteReader())
{
while (reader.Read())
{
result = (byte[])reader["value"];
}
reader.Close();
}
}
}
string decodedString = new UnicodeEncoding(false, true).GetString(result);
var document = XElement.Parse(decodedString);
}
If I run this test I get an XmlException with message : "Data at the root level is invalid. Line 1, position 1." As it turns out the problem is "0xFFFE" preamble which is considered as invalid character.
Note that if I use commented string instead, everything works just fine, which is strange as per me. Looks like SqlServer stores XML strings in UCS-2 with a BOM, and at the same time it stores nvarchar values without it.
The main question is: how can I decode this byte array to string which will not contain this preamble (BOM)?
In case anyone will need this in future, the following code works:
using(var ms = new MemoryStream(result))
{
using (var sr = new StreamReader(ms, Encoding.Unicode, true))
{
decodedString = sr.ReadToEnd();
}
}

Is it possible to use `SqlDbType.Structured` to pass Table-Valued Parameters in NHibernate?

I want to pass a collection of ids to a stored procedure that will be mapped using NHibernate. This technique was introduced in Sql Server 2008 ( more info here => Table-Valued Parameters ). I just don't want to pass multiple ids within an nvarchar parameter and then chop its value on the SQL Server side.
My first, ad hoc, idea was to implement my own IType.
public class Sql2008Structured : IType {
private static readonly SqlType[] x = new[] { new SqlType(DbType.Object) };
public SqlType[] SqlTypes(NHibernate.Engine.IMapping mapping) {
return x;
}
public bool IsCollectionType {
get { return true; }
}
public int GetColumnSpan(NHibernate.Engine.IMapping mapping) {
return 1;
}
public void NullSafeSet(DbCommand st, object value, int index, NHibernate.Engine.ISessionImplementor session) {
var s = st as SqlCommand;
if (s != null) {
s.Parameters[index].SqlDbType = SqlDbType.Structured;
s.Parameters[index].TypeName = "IntTable";
s.Parameters[index].Value = value;
}
else {
throw new NotImplementedException();
}
}
#region IType Members...
#region ICacheAssembler Members...
}
No more methods are implemented; a throw new NotImplementedException(); is in all the rest. Next, I created a simple extension for IQuery.
public static class StructuredExtensions {
private static readonly Sql2008Structured structured = new Sql2008Structured();
public static IQuery SetStructured(this IQuery query, string name, DataTable dt) {
return query.SetParameter(name, dt, structured);
}
}
Typical usage for me is
DataTable dt = ...;
ISession s = ...;
var l = s.CreateSQLQuery("EXEC some_sp #id = :id, #par1 = :par1")
.SetStructured("id", dt)
.SetParameter("par1", ...)
.SetResultTransformer(Transformers.AliasToBean<SomeEntity>())
.List<SomeEntity>();
Ok, but what is an "IntTable"? It's the name of SQL type created to pass table value arguments.
CREATE TYPE IntTable AS TABLE
(
ID INT
);
And some_sp could be like
CREATE PROCEDURE some_sp
#id IntTable READONLY,
#par1 ...
AS
BEGIN
...
END
It only works with Sql Server 2008 of course and in this particular implementation with a single-column DataTable.
var dt = new DataTable();
dt.Columns.Add("ID", typeof(int));
It's POC only, not a complete solution, but it works and might be useful when customized. If someone knows a better/shorter solution let us know.
A simpler solution than the accepted answer would be to use ADO.NET. NHibernate allows users to enlist IDbCommands into NHibernate transactions.
DataTable myIntsDataTable = new DataTable();
myIntsDataTable.Columns.Add("ID", typeof(int));
// ... Add rows to DataTable
ISession session = sessionFactory.GetSession();
using(ITransaction transaction = session.BeginTransaction())
{
IDbCommand command = new SqlCommand("StoredProcedureName");
command.Connection = session.Connection;
command.CommandType = CommandType.StoredProcedure;
var parameter = new SqlParameter();
parameter.ParameterName = "IntTable";
parameter.SqlDbType = SqlDbType.Structured;
parameter.Value = myIntsDataTable;
command.Parameters.Add(parameter);
session.Transaction.Enlist(command);
command.ExecuteNonQuery();
}
For my case, my stored procedure needs to be called in the middle of an open transaction.
If there is an open transaction, this code works because it is automatically reusing the existing transaction of the NHibernate session:
NHibernateSession.GetNamedQuery("SaveStoredProc")
.SetInt64("spData", 500)
.ExecuteUpdate();
However, for my new Stored Procedure, the parameter is not as simple as an Int64. It's a table-valued-parameter (User Defined Table Type)
My problem is that I cannot find the proper Set function.
I tried SetParameter("spData", tvpObj), but it's returning this error:
Could not determine a type for class: …
Anyways, after some trial and error, this approach below seems to work.
The Enlist() function is the key in this approach. It basically tells the SQLCommand to use the existing transaction. Without it, there will be an error saying
ExecuteNonQuery requires the command to have a transaction when the
connection assigned to the command is in a pending local transaction…
using (SqlCommand cmd = NHibernateSession.Connection.CreateCommand() as SqlCommand)
{
cmd.CommandText = "MyStoredProc";
NHibernateSession.Transaction.Enlist(cmd); // Because there is a pending transaction
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(new SqlParameter("#wiData", SqlDbType.Structured) { Value = wiSnSqlList });
int affected = cmd.ExecuteNonQuery();
}
Since I am using the SqlParameter class with this approach, SqlDbType.Structured is available.
This is the function where wiSnList gets assigned:
private IEnumerable<SqlDataRecord> TransformWiSnListToSql(IList<SHWorkInstructionSnapshot> wiSnList)
{
if (wiSnList == null)
{
yield break;
}
var schema = new[]
{
new SqlMetaData("OriginalId", SqlDbType.BigInt), //0
new SqlMetaData("ReportId", SqlDbType.BigInt), //1
new SqlMetaData("Description", SqlDbType.DateTime), //2
};
SqlDataRecord row = new SqlDataRecord(schema);
foreach (var wi in wiSnList)
{
row.SetSqlInt64(0, wi.OriginalId);
row.SetSqlInt64(1, wi.ShiftHandoverReportId);
if (wi.Description == null)
{
row.SetDBNull(2);
}
else
{
row.SetSqlString(2, wi.Description);
}
yield return row;
}
}
You can pass collections of values without the hassle.
Example:
var ids = new[] {1, 2, 3};
var query = session.CreateQuery("from Foo where id in (:ids)");
query.SetParameterList("ids", ids);
NHibernate will create a parameter for each element.

Resources