NpgSqlCopyIn recommended error handling - npgsql

Question

I have following code for PostgreSQL bulk inserts using npgsql from .Net code.
try
{
var items = GetSourceData(task);
connection.Open();
var command = new NpgsqlCommand(null, connection);
BeforeDestinationCommandExecution(task, command);
command.CommandText = string.Format("COPY {0} FROM STDIN", task.DestinationTable);
command.CommandTimeout = 3600;
var cin = new NpgsqlCopyIn(command, connection);
var rowCount = 0;
try
{
cin.Start();
foreach (var item in items)
{
var b = StreamEncoding.GetBytes(ConvertSourceData(item));
cin.CopyStream.Write(b, 0, b.Length);
++rowCount;
}
cin.End();
log.Debug(string.Format("Table {0} contained {1:N0} records", task.DestinationTable, rowCount));
}
catch (Exception e)
{
log.ErrorException("Exception caught in inner try block - MigrateWithCopyMode", e);
try
{
// send CopyFail to server
cin.Cancel("Undo copy");
}
catch (Exception cancelException)
{
// we should get an error in response to our cancel request:
if (!cancelException.ToString().Contains("Undo copy"))
{
throw new Exception("Failed to cancel COPY: " + cancelException + " upon failure: " + e);
}
}
throw;
}
finally
{
_migrationCounts.Add(task.DestinationTable, rowCount);
}
For last 2 days I've experienced unhandled exception while executing the code. After some investigation and attaching code to UnhandledException event.
System.AppDomain.CurrentDomain.UnhandledException += unhandledException;
I've found that the problem was in data being processed.
Error [16] [HubAdapterMsSqlPostgres] Unhandled exception Npgsql.NpgsqlException:
null value in column "name_ru" violates not-null constraint
Severity: ERROR
Code: 23502
at Npgsql.NpgsqlState.<ProcessBackendResponses_Ver_3>d__a.MoveNext()
at Npgsql.NpgsqlState.IterateThroughAllResponses(IEnumerable`1 ienum)
at Npgsql.NpgsqlConnector.NpgsqlContextHolder.ProcessServerMessages()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
This error was generated on different thread. I guess after calling cin.CopyStream.Write(b, 0, b.Length); from previous code example.
My question is
What is recommended way to handle such error to be able omit rows with wrong values and continue in bulk insert operation
Thank you

Answer 1

You're trying to import data containing rows which are incompatible with your table definition - they lack data for a non-nullable column. COPY is an all-or-nothing process: either the entire process succeeds, or it fails entirely. Therefore, you have to either:
* Clean up your input first, removing the offending rows (this would involve parsing your input, which may be difficult), or
* Temporarily remove the non-null constraint on your table, perform the COPY, delete the problematic rows from the table and reinstate the constraint. This may or may not be appropriate depending on your usage scenario.
Regardless, you're using Npgsql 2.x, which is very old by now and unmaintained. It's highly recommended you upgrade to the latest version of Npgsql.