Serilog is not writing to the database consistently - sql-server

I am using the below code and every time I run the code, I do see the break point hitting in LogErrorToDatabase method. However sometimes I see the log entry for the ErrorGuid in the database and sometimes I do not see any log entries. The ErrorGuid gets returned to the client side but missing DB entry.
public async Task<ActionResult<IEnumerable<Location>>> Get()
{
try {
Log.Information("Inside Locations Get");
int a = int.Parse("?"); //Uncomment to test error handling
return await Repo.Get<Location>().ToListAsync();
}
catch (Exception ex)
{
//Log error
var id = Guid.NewGuid();
await LogErrorToDatabase(ex, id);
return BadRequest(new ErrorDetail()
{
ErrorGuid = id
});
}
}
public async Task LogErrorToDatabase(Exception ex, Guid id)
{
await Task.Run(() => Log.ForContext("ErrorGuid", id)
.Error(ex, "Error occured with getting the Location List"));
}
I have no clue on this randomness. Any help will be appreciated.

There are several steps you can take troubleshoot why messages are not being written to the Serilog Sink. You can see a number of them on another answer here in SO:
Serilog MSSQL Sink doesn't write logs to database
Update: String or binary data would be truncated is a common SQL Server exception that happens when you're trying to store a value that is larger than the table field can store. For example, if you have an nvarchar(4) field and try to store abcde (5 characters) you'll get the same kind of exception.
You either have to increase the size of the fields in the database you're using to store information, or you have to limit the number of characters you're logging in your C# code.

Thanks Calo. Your response was helpful. I turned the Audit on by switching from WriteTo to AuditTo, set "period": 1 and added Log.CloseAndFlush(); and get the below error logged to my SQL database. Help will be appreciated.
System.AggregateException: Failed to emit a log event. (String or binary data would be truncated.
The statement has been terminated.)
---> System.Data.SqlClient.SqlException (0x80131904): String or binary data would be truncated.
The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite, String method)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite, String methodName)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Serilog.Sinks.MSSqlServer.MSSqlServerAuditSink.Emit(LogEvent logEvent)
at Serilog.Core.Sinks.AggregateSink.Emit(LogEvent logEvent)
ClientConnectionId:c0e1f1a4-b655-4be1-b655-c8954f44d299
Error Number:8152,State:2,Class:16
--- End of inner exception stack trace ---
at Serilog.Core.Sinks.AggregateSink.Emit(LogEvent logEvent)
at Serilog.Core.Logger.Dispatch(LogEvent logEvent)
at Serilog.Core.Logger.Serilog.Core.ILogEventSink.Emit(LogEvent logEvent)
at Serilog.Core.Logger.Dispatch(LogEvent logEvent)
at Serilog.Core.Logger.Serilog.Core.ILogEventSink.Emit(LogEvent logEvent)
at Serilog.Core.Logger.Dispatch(LogEvent logEvent)
at Serilog.Core.Logger.Write(LogEventLevel level, Exception exception, String messageTemplate, Object[] propertyValues)
at Serilog.Core.Logger.Write(LogEventLevel level, String messageTemplate, Object[] propertyValues)
at Serilog.Core.Logger.Write[T0,T1,T2](LogEventLevel level, String messageTemplate, T0 propertyValue0, T1 propertyValue1, T2 propertyValue2)
at Serilog.Core.Logger.Error[T0,T1,T2](String messageTemplate, T0 propertyValue0, T1 propertyValue1, T2 propertyValue2)
at MCF.API.ExtensionHelpers.ExceptionMiddleware.InvokeAsync(HttpContext httpContext) in

Related

IIS, SQL Server & Umbraco

I wonder if anyone can help me out, we have a Umbraco installation running:
OS: Microsoft Windows Server 2016 Standard, v10.0.14393 Build 14393
SQL Server: SQL Server 2016 Standard Edition, v13.0.1728.2
Umbraco: v7.3.4
The server had a scheduled reboot this morning at 5 am, from then, we've can't get the application back online, every time we go to the site we get the following error:
[Win32Exception (0x80004005): The wait operation timed out]
[SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed before completion of the operation or the server is not responding.]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) +2442598
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) +5766516
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +285
System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) +4162
System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() +58
System.Data.SqlClient.SqlDataReader.get_MetaData() +89
System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption) +409
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) +2031
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) +911
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) +64
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) +240
System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior) +41
System.Data.Common.DbCommand.ExecuteReader(CommandBehavior behavior) +12
StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteDbDataReader(CommandBehavior behavior) +68
System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader() +14
Umbraco.Core.Persistence.<Query>d__71.MoveNext() +265
I can't see any updates have run and looked at the windows update logs there isn't anything there neither.
I've tried
Restarting the server
Restarting the application pool.
Restarting IIS
Increase timeout to 6000.
If anyone can shed any more light on this it would be great.
You need to increase the command timeout from the default of 30 seconds. It's a property of the DbCommand object.
See here:
https://learn.microsoft.com/en-us/dotnet/api/system.data.common.dbcommand.commandtimeout?view=netframework-4.8
I'd suggest you change it to the value 0 (i.e. unlimited)

SQL Server DbCommand Timeout with .Net Core container under load

I'm running a .Net Core container on Open Shift Enterprise V3 pointing to a SQL Server database.
I have a .Net Core REST API with a put method which adds or updates a record in the database.
The table I am adding/updating had 3000 records and has indexes.
This works fine locally using the same database and with the container. However when I start to put load through the the container (approximately 50 concurrent http connections with JMeter) I get random timeouts with the error message below. I don't get the problem running locally.
The local machine is a lot more powerful than the container and I have increased the CPU power on the container but this doesn't appear to have made any difference.
Any suggestions on things to try would be appreciated.
[10:45:36 ERR] Failed executing DbCommand (35,001ms) [Parameters=[#__get_Item_0='?' (Size = 255) (DbType = AnsiString)], CommandType='Text', CommandTimeout='30']
SELECT [e].[host_name], [e].[data_centre], [e].[Environment], [e].[is_physical_machine], [e].[mac_address], [e].[number_of_cores], [e].[number_of_sockets], [e].[number_of_v_cores], [e].[number_ofcpus], [e].[operating_system], [e].[operating_system_version], [e].[processor], [e].[uuid]
FROM [EntitlementServer].[host] AS [e]
WHERE [e].[host_name] = #__get_Item_0
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (258): Unknown error 258
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.ExecuteReader()
at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.Execute(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues)
ClientConnectionId:6ca037fc-9671-4d43-bebe-35879203c682
Error Number:-2,State:0,Class:11
Update 1
Whilst I found that scaling up the CPU and memory on the container did not help with performance. When I increased the number of running containers, that did increase throughput and the timeout issue went away. I am not sure why this would be the case.
The problem actually turned out to be a threading issue. The db timeout was a symptom. This is the reason why increasing the number of containers stopped the problem because the number of http threads also increased.
By making code asynchronous so that threads are released I have been able to fix the problem without having to increase the number of containers.

The MARS TDS header contained errors - ASP.NET Core + EF Core 2.1.4 + Azure SQL Server

I have an ASP.NET Core app [Microsoft.AspNetCore.App 2.1.4] with EF Core 2.1.4 [DbContext pooling enabled] and data stored on an Azure SQL database.
Occasinaly [once in 1-2 days] I get unrecoverable error stating System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
Once I restart the app, everything resumes working until it happens again.
Probably import to note is that I don't have MultipleActiveResultSets enabled in my connection string at all, which makes the issue even more strange.
Has anyone else noticed something similar? Are there any ways I can trace the problem?
Stack trace:
System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.Execute(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues)
at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteReader(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(DbContext _, Boolean buffer)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[TState,TResult](Func`3 operation, Func`3 verifySucceeded, TState state)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Inviton.Web.Portal.Controllers.EventListController.GetEvents(BL bl, Int32 venueId, Int32 promoterId)
---snip---
For all the ones who are still facing this problem and ends up on this page.
There's a open issue on SqlCLient https://github.com/dotnet/SqlClient/issues/85
As of today is not fixed.
But there's a hackaroud which mitigate it https://github.com/dotnet/SqlClient/issues/85#issuecomment-469432914
This worked for me:
Right Click your Project -> Manage nuget packages -> Updates.
Update Microsoft.Data.SqlClient to last version.

sql server express Permissions

I have made a Web application with the Data Base, in sql server express
I open the server and built the tables in visual studio (I could not find another way)
When I went Data Base website (in online) is saying that I have no permissions,
Does anyone know how I add permissions through visual studio?
error:
Server Error in '/' Application.
The SELECT permission was denied on the object 'UsersTable', database
'master', schema 'dbo'.
Description: An unhandled exception occurred during the execution of
the current web request. Please review the stack trace for more
information about the error and where it originated in the code.
Exception Details: System.Data.SqlClient.SqlException: The SELECT
permission was denied on the object 'UsersTable', database 'master',
schema 'dbo'.
Source Error:
An unhandled exception was generated during the execution of the
current web request. Information regarding the origin and location of
the exception can be identified using the exception stack trace below.
Stack Trace:
[SqlException (0x80131904): The SELECT permission was denied on the
object 'UsersTable', database 'master', schema 'dbo'.]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception,
Boolean breakConnection, Action1 wrapCloseInAction) +3279580
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject
stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +791
System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior,
SqlCommand cmdHandler, SqlDataReader dataStream,
BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
stateObj, Boolean& dataReady) +4927
System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() +92
System.Data.SqlClient.SqlDataReader.get_MetaData() +102
System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds,
RunBehavior runBehavior, String resetOptionsString) +552
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean
async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader
ds, Boolean describeParameterEncryptionRequest) +3114
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String
method, TaskCompletionSource1 completion, Int32 timeout, Task& task,
Boolean asyncWrite) +554
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String
method) +98
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior
behavior, String method) +307
System.Data.SqlClient.SqlCommand.ExecuteReader() +132
KIP_FP.KIPDB.chackUser(String userN) in
C:\Users\Administrator\Documents\KIP-FP\KIP-FP\KIPDB.cs:31
KIP_FP.KIPBL.chackUser(String userN) in
C:\Users\Administrator\Documents\KIP-FP\KIP-FP\KIPBL.cs:37
KIP_FP.loginUserPage.Button1_Click(Object sender, EventArgs e) in
C:\Users\Administrator\Documents\KIP-FP\KIP-FP\Login.aspx.cs:34
System.Web.UI.WebControls.Button.OnClick(EventArgs e) +11750641
System.Web.UI.WebControls.Button.RaisePostBackEvent(String
eventArgument) +150 System.Web.UI.Page.ProcessRequestMain(Boolean
includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
+6016
You have permission to make SELECT on the UsersTable table but Application Pool of your web application does not have it, because it is running under another user account which has no permission.
To add permission you have to run something like this:
USE master;
GRANT SELECT ON OBJECT::dbo.UsersTable TO PoolUser;
GO
But first you should find out which user is on App. Pool.

SSIS ETL timeout error

So I am just gonna try out my luck here with a problem that I have been facing with my prod ETL project. It is showing a pattern where it fails every time with a weird error below:
Build Dimensions Failed on - BRCSQL-NA1-01.PROD with following Error Messages:
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior)
at Microsoft.SqlServer.Dts.Pipeline.DataReaderSourceAdapter.PreExecute()
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPreExecute(IDTSManagedComponentWrapper100 wrapper)
ClientConnectionId:e4461e3d-2a35-42a2-b4da-517602d6c1f3
Enrollment Misc Source failed the pre-execute phase and returned error code 0x80131904.
The strange thing is that it just works the second time I would try to run it. So that has been our workaround, I would just go in and rerun it manually whenever I get this error.
It could be something stupid happening with our server maintenance, but I just wanted to make sure with you intelligent people of internet before complaining my server maintenance guys.

Resources