Could i increase speed of executing statements? - sql-server

Question

My function: Removes the unit in a specific row of a given Table
private static void removeUnits(String connectionString, String tableName, String columnID, String columnToFix)
{
List<String> rowsToEdit = new List<String>();
using (SqlConnection connection = new SqlConnection(connectionString))
{
using (var command = connection.CreateCommand())
{
command.CommandText = "SELECT " + columnID + "," + columnToFix + " FROM " + tableName;
connection.Open();
using (var reader = command.ExecuteReader())
{
var indexOfR_MEASUREDVALUEID = reader.GetOrdinal(columnID);
var indexOfT_VALUE = reader.GetOrdinal(columnToFix);
while (reader.Read())
{
var t_value = reader.GetValue(indexOfT_VALUE);
var t_id = reader.GetValue(indexOfR_MEASUREDVALUEID);
String newValue = getWithoutUnit(t_value.ToString());
if (newValue != null)
{
String sql = "UPDATE " + tableName + " SET " + columnToFix + "='" +
newValue + "' WHERE " + columnID + "='" + t_id + "';";
rowsToEdit.Add(sql);
}
}
connection.Close();
}
}
Console.WriteLine("start writing " + rowsToEdit.Count + " entries?");
Console.ReadLine();
SqlCommand sqlCmd;
sqlCmd = new SqlCommand("", connection);
sqlCmd.Connection.Open();
foreach (String command in rowsToEdit)
{
sqlCmd.CommandText = command;
sqlCmd.ExecuteNonQuery();
}
}
Console.WriteLine(rowsToEdit.Count + " commands executed");
}
I am using C# in Visual Studio 2010 and SQL-Server 2012.
It works fine, but executing of 200000 lines takes really long.
Is its possible to do this faster?

Answer 1

My suggestion involves looking into parallelism:
Note: The code isn't tested, just typed it out in Notepad++
private static void removeUnits(String connectionString, String tableName, String columnID, String columnToFix)
{
List<new Tuple<object, object>> rowsToEdit = new List<new Tuple<object, object>>();
using (SqlConnection connection = new SqlConnection(connectionString))
{
using (var command = connection.CreateCommand())
{
command.CommandText = "SELECT " + columnID + "," + columnToFix + " FROM " + tableName;
connection.Open();
using (var reader = command.ExecuteReader())
{
var indexOfR_MEASUREDVALUEID = reader.GetOrdinal(columnID);
var indexOfT_VALUE = reader.GetOrdinal(columnToFix);
while (reader.Read())
{
rowsToEdit.Add(new Tuple<object, object>(reader.GetValue(indexOfT_VALUE),reader.GetValue(indexOfR_MEASUREDVALUEID)));
}
connection.Close();
}
}
}
// Use parallelism here
Parallel.Foreach(rowsToEdit, currentRow =>
{
String newValue = getWithoutUnit(currentRow.Value1.ToString());
if (newValue != null)
{
// reopen connection
// use parameters here, and call SP
}
}
Console.WriteLine("start writing " + rowsToEdit.Count + " entries?");
Console.ReadLine();
}
Stored Proc:
Create StoredProcedure MySP
(
#tablename varchar(50),
#columnToFix varchar(50),
#newValue varchar(50),
#columnID varchar(50),
#tID varhcar(varchar(50)
)
As
Begin
Declare #MySQL varchar(500)
Set #MySQL = 'Update ' + #tableName + ' Set ' + #columnToFix + ' = '' + #newValue + ''' + ' Where ' + #columnID + ' = '' + #tID + '''
sp_executesql #MySQL
End
It's important to note that dynamic sql is generally frowned down upon. Here's a link discussing dynamic sql (pros/cons):http://www.sommarskog.se/dynamic_sql.html
Also it's possible you may run into some locking/contention issues b/c you're hitting one table(based on the input parameters of the function).

Answer 2

First try to use Parameters as already mentioned in the comments. Second, use the SqlCommand.Prepare() statement. The actual speed also depends on your machine capacities (HD Write Speed/RAM). Here is the example (you may have to edit the SqlDbTypes according to your Colum Types)
private static void RemoveUnits(string connectionString, string tableName, string columnID, string columnToFix)
{
Dictionary<int, string> rowsToEdit = new Dictionary<int, string>();
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT " + columnID + "," + columnToFix + " FROM " + tableName;
using (var reader = command.ExecuteReader())
{
var indexOfR_MEASUREDVALUEID = reader.GetOrdinal(columnID);
var indexOfT_VALUE = reader.GetOrdinal(columnToFix);
while (reader.Read())
{
string t_value = reader.GetString(indexOfT_VALUE);
int t_id = reader.GetInt32(indexOfR_MEASUREDVALUEID);
string newValue = getWithoutUnit(t_value);
if (newValue != null)
{
// save values in dictionary
rowsToEdit[t_id] = newValue;
}
}
}
}
Console.WriteLine("start writing " + rowsToEdit.Count + " entries?");
Console.ReadLine();
string updateCmd = "UPDATE " + tableName + " SET " + columnToFix + "= #Value WHERE " + columnID + "= #Key;";
using (SqlTransaction transaction = connection.BeginTransaction())
{
SqlCommand sqlCmd = connection.CreateCommand();
sqlCmd.CommandText = updateCmd;
sqlCmd.Parameters.Add("#Value", System.Data.SqlDbType.NVarChar, -1);
sqlCmd.Parameters.Add("#Key", System.Data.SqlDbType.Int);
// important for performance
sqlCmd.Prepare();
foreach (var update in rowsToEdit)
{
// change values of parameters
sqlCmd.Parameters["#Key"].Value = update.Key;
sqlCmd.Parameters["#Value"].Value = update.Value;
// and execute it
sqlCmd.ExecuteNonQuery();
}
transaction.Commit();
}
}
Console.WriteLine(rowsToEdit.Count + " commands executed");
}
Edit: if getWithoutUnit(string) does something not very complex, you could probably do your operation with a single statement using TSQL, which comes with various string manipulation capabilities. Like UPDATE TableName SET StringColumn = <string manipulation TSQL here>, see http://msdn.microsoft.com/en-us/library/ms181984.aspx
Edit2: added transaction from comment hint (indeed should be faster as well), strongly typed columns