We have this problem in our system. We're using System.Transactions and work with TransactionScope (mostly, and definitely in this case). We've just migrated from EnterpriseServices to SystemTransactions recently, and we got this error several times, during heavy multi-threaded work:
System.Transactions.TransactionInDoubtException: The transaction is in doubt. ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning() at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error) at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket() at System.Data.SqlClient.TdsParserStateObject.ReadByte() at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest) at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest) at System.Data.SqlClient.SqlDelegatedTransaction.SinglePhaseCommit(SinglePhaseEnlistment enlistment)
It was apparently thrown during the Scope.Complete().
I've read a bit about TransactionInDoubt, that it might happen in the promotion to DTC, and maybe there's a slight connectivity issue with the SQL, or the MSDTC. but then - what is with the Timeout Expired which is a SqlException?? where is the timeout exactly? in the commit???
I've been scanning the web for the past few days and some occurrences here and there, but no good answers.
help anyone??
Related
At Flink 1.16 version , StreamTableEnvironment.fromDataStream(DataStream dataStream, Expression... fields) method is depricated.
At previous versions, it can be defined by using expressions event time and process time by methods
.rowtime() ,
.proctime() ,
as defined in https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/time_attributes/
What is the right methodology in newly releases?
At previous version we can define table from stream as
Table transactionTable = tableEnv.fromDataStream(transactionDataStream,$("Field1"),$("field2"),$("field3")
,$("transactionTime").rowtime(),$("ts").proctime());
but the method StreamTableEnvironment.fromDataStream(DataStream dataStream, Expression... fields) is depricated after version 1.15
You should now use <T> Table fromDataStream(DataStream<T> dataStream, Schema schema) instead, where the Schema will be something like this:
Schema.newBuilder()
.column("field1", DataTypes.INT())
.column("transactionTime", DataTypes.TIMESTAMP_LTZ(3))
.columnByExpression("ts", "PROCTIME()")
.watermark("transactionTime", "transactionTime - INTERVAL '10' SECOND"")
.build();
There's more information, and more examples, in the documentation.
I run some SQL queries on Azure from a .NET Core 2.2 console application, using Dapper in this way:
using (IDbConnection db = new SqlConnection(_appSettings.ConnectionStrings))
db.Execute("delete from MyTable where Id=#id", new { id = myObject.Id });
Sometimes (but very few time during the day) I get this error:
A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error) at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync() at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite, String methodName)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action2 paramReader) in C:\projects\dapper\Dapper\SqlMapper.cs:line 2827 at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command) in C:\projects\dapper\Dapper\SqlMapper.cs:line 570 at Dapper.SqlMapper.Execute(IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Nullable1 commandTimeout, Nullable`1 commandType) in C:\projects\dapper\Dapper\SqlMapper.cs:line 443
at ProjectQueue.Engine.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\userX\source\repos\myProject\ProjectQueue\Engine.cs:line 75
I searched in SO, and some tips are regard setting to false AUTO CLOSE on SQL (which I already have):
What can it be? How can investigate more and resolve the problem?
Azure SQL is known for having "transient errors" occasionally when connecting or issuing commands. Of course there could be other causes also for your specific error (network devices issues, hardware issues), but since they only happen occasionally and are not reproducible, the transient error handling solutions still apply.
In Azure, there can be a number of causes for these transient issues, often related to some auto-scaling or an infrastructure issue that is being mitigated behind the scenes by Azure engineers or by some Azure auto-healing, for which you will normally not have any visibility.
The main solution and the one recommended strongly by Microsoft is to implement retry handling to handle these "transient" issues. You could use a .NET library like Polly (https://github.com/App-vNext/Polly), or build your own with a while loop (https://learn.microsoft.com/en-us/sql/connect/ado-net/step-4-connect-resiliently-sql-ado-net?view=sql-server-ver15#step-2b-copy-and-paste-sample-code). They often recommend a retry back-off strategy that includes retrying quickly for the first retry, and if it keeps failing, then increase your retry intervals on subsequent retries.
Make sure you log the exceptions when retrying, so that you can determine if the issue becomes more persistent.
More info about Azure SQL transient errors:
https://learn.microsoft.com/en-us/azure/azure-sql/database/troubleshoot-common-connectivity-issues
And transient error handling: https://learn.microsoft.com/en-us/azure/architecture/best-practices/transient-faults
Hi all, this might be a shot in the dark, but I'm looking for help on sorting out an error that resides in either a report data definition or the Sequel Server Reporting Services (SSRS) report form. The error is preventing a particular form (customized packing slip) from printing when we have multiple lines.
Just to note, the report form is designed using Microsoft Report Builder, and gets it data from Epicor which has an associated "Report Data Definition".
The consultant who customized the packing slip is no longer reachable, and without his notes I'm trying to figure ou[enter image description here][1]t what was changed from the original data definition or report form that could have caused this error.
Okay, now on to the problem:
Upon trying to print a multiple line item packing slip, we get this error:
Program Ice.Services.Lib.RunTask raised an unexpected exception with the following message:
RunTask: CREATE VIEW failed because column 'OTSCity' in view 'RptLabels_8B03042B0E8248588E5DECCC9D76BB89' exceeds the maximum of 1024 columns.
Some information I've gathered during my hunting and pecking:
OTSCITY is a field in the OrderHED table (Order Header). The OrderHed table is one of the (25) report tables in the data definition, however it doesn't look like it's linked to any other table, and the field 'OTSCITY' is excluded (both label and column) in both the original and custom report
Not sure what 'RptLabels' refers to, but the long number that follows is the Table GUID. Everytime a report gets printed, a unique GUID is assigned to that particular instance. I'm guessing when a report gets printed it pulls the data from the data definition, and that snapshot of data is assigned with the GUID. So essentially it's referring to the report labels (RptLabels) in the pack slip instance I created when I hit the print button. That's just my guess.
I'm pretty sure the report forms have a limit to how many columns they can show, which is set to 1024, so perhaps some data table is returning too many columns when we try to make a packing slip form with multiple line items.
I'm leaving the rest of the Stack Trace at the bottom of this post in case you're a Epicor/SSRS/SQL wizard that can read that stuff. Pictures of the Report data definition are there as well.
Stack Trace:
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action 1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Ice.Core.RptBase.RptLabelsSqlTableBuilder.BuildView(SqlObjectsCreated sqlObjectsCreated) in c:\_Releases\ICE\3.1.400.0\source\Server\Internal\Lib\TaskLib\RptBase\RptLabelsSqlTableBuilder.cs:line 100
at Ice.Core.RptBase.ReportDatabaseBuilder 1.BuildSchemaAndWriteData(Func 2 executeCommand, SqlObjectsCreated sqlObjectsCreated) in c:\_Releases\ICE\3.1.400.0\source\Server\Internal\Lib\TaskLib\RptBase\ReportDatabaseBuilder.cs:line 165
at Ice.Core.RptBase.ReportDatabaseBuilder`1.GenerateSqlObjectsAndProcessReport(SqlConnection connection) in c:\_Releases\ICE\3.1.400.0\source\Server\Internal\Lib\TaskLib\RptBase\ReportDatabaseBuilder.cs:line 191
at Ice.Core.RptBase.ReportDatabaseBuilder`1.XMLClose() in c:\_Releases\ICE\3.1.400.0\source\Server\Internal\Lib\TaskLib\RptBase\ReportDatabaseBuilder.cs:line 132
at Ice.Core.RptTaskBase`1.XMLClose() in c:\_Releases\ICE\3.1.400.0\source\Server\Internal\Lib\TaskLib\RptBase\RptTaskBase.cs:line 134
at Erp.Internal.SR.PackingSlipPrint.RunProcess(Int64 Instance_TaskNum, String OutputFile) in c:\_Releases\ERP\RL10.1.400.0\Source\Server\Internal\SR\PackingSlipPrint\PackingSlipPrint.cs:line 919
at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in c:\_Releases\ICE\3.1.400.7\source\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 78
at Ice.Hosting.TaskCaller.ExecuteTask(Boolean suppressTransaction) in c:\_Releases\ICE\3.1.400.7\source\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 31
at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in c:\_Releases\ICE\3.1.400.7\source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 63
at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in c:\_Releases\ICE\3.1.400.7\source\Server\Services\Lib\RunTask\RunTask.cs:line 477
at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in c:\_Releases\ICE\3.1.400.7\source\Server\Services\Lib\RunTask\RunTask.cs:line 477
at Ice.Services.Lib.RunTaskSvcFacade.RunTask(Int64 ipTaskNum) in c:\_Releases\ICE\3.1.400.7\source\Server\Services\Lib\RunTask\RunTaskSvcFacade.cs:line 97
at SyncInvokeRunTask(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at Epicor.Hosting.OperationBoundInvoker.InnerInvoke(Object instance, Func`2 func) in c:\_Releases\ICE\3.1.400.7\source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs:line 59
at Epicor.Hosting.OperationBoundInvoker.Invoke(Object instance, Func`2 func) in c:\_Releases\ICE\3.1.400.7\source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs:line 28
at Epicor.Hosting.Wcf.EpiOperationInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) in c:\_Releases\ICE\3.1.400.7\source\Framework\Epicor.System\Hosting\Wcf\EpiOperationInvoker.cs:line 23
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
at System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.SecurityChannelListener`1.ReceiveItemAndVerifySecurityAsyncResult`2.InnerTryReceiveCompletedCallback(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object state)
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)
at System.Net.AsyncProtocolRequest.CompleteRequest(Int32 result)
at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.ConnectionStream.IOAsyncResult.OnAsyncIOComplete(Object state)
at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs e)
at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
[RDD OrderHed Table Exclusions][1]
It appears that you may have too many column labels in your customized report. Each column will have an optional label (Prompt/Heading) and Epicor creates a view with all these labels in it for use in the report. The standard Pack Slip in E10.0 has 904 columns in it which is far more than are actually used on the report. Have they added a lot more fields to the report? Admittedly an extra 120 columns is a lot if no additional tables were added.
You can check the number of columns in the EpicorSSRS database in SQL by using the Object Explorer Details and sorting the views by the create date/time to find the temp view you just created (after running the report). Run this query in the EpicorSSRS database to count the number of columns:
select count(*),c.table_name
from information_schema.COLUMNS c
JOIN information_schema.tables t ON c.TABLE_NAME = t.TABLE_NAME
AND c.TABLE_Schema = t.TABLE_Schema
WHERE TABLE_TYPE = 'view'
and t.Table_name = 'RptLabels_710812BD643A4097900608B397D0779A'
GROUP BY c.table_name
Obviously if it is failing, then the view won't be created, but try unchecking some of the unused labels in the RDD until you get to a point where it will run.
I don't know why having multiple lines on your pack slip would increase the number of fields. Your general understanding of how the reporting works is correct.
Here is a rundown on SQL Server's Sparce Columns. I bet the problem you are having is because the table does not define a COLUMN_SET FOR ALL_SPARSE_COLUMNS. See this article by Pinal Dave for more details.
We have a large web application in asp.net mvc using EF6 and SQL server 2012.
We have two environments: staging and production.
Every time we have a release, we first deploy the exact same code to staging and restore production's db on the server for testing purposes. That has always worked for us.
Now, when we restore production's db to the staging server we are getting some timeouts on some commands. What's strange is that after a few hours pass, we stop getting timeouts.
This is the stacktrace of one of the timeouts:
[Win32Exception (0x80004005): The wait operation timed out]
[SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The statement has been terminated.]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) +388
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +688
System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) +4403
System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() +82
System.Data.SqlClient.SqlDataReader.get_MetaData() +135
System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) +6664141
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite) +6666008
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite) +577
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) +107
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) +288
System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior) +180
Glimpse.Ado.AlternateType.GlimpseDbCommand.ExecuteDbDataReader(CommandBehavior behavior) +847
System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch(Func`1 operation, TInterceptionContext interceptionContext, Action`1 executing, Action`1 executed) +72
System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext) +306
System.Data.Entity.Core.Mapping.Update.Internal.DynamicUpdateCommand.Execute(Dictionary`2 identifierValues, List`1 generatedValues) +417
System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.Update() +218
[UpdateException: An error occurred while updating the entries. See the inner exception for details.]
System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.Update() +537
System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction(Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess) +627
System.Data.Entity.Core.Objects.ObjectContext.SaveChangesToStore(SaveOptions options, IDbExecutionStrategy executionStrategy) +212
System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute(Func`1 operation) +263
System.Data.Entity.Core.Objects.ObjectContext.SaveChanges(SaveOptions options) +262
System.Data.Entity.Internal.InternalContext.SaveChanges() +218
[DbUpdateException: An error occurred while saving entities that do not expose foreign key properties for their relationships. The EntityEntries property will return null because a single entity cannot be identified as the source of the exception. Handling of exceptions while saving can be made easier by exposing foreign key properties in your entity types. See the InnerException for details.]
System.Data.Entity.Internal.InternalContext.SaveChanges() +291
*******.Services.DbTransaction.TransactionManager.SaveChanges() in c:\BuildAgent\work\c357fed3de014622\Source\Services\*******.Services\*******.Services\DbTransaction\TransactionManager.cs:28
System.Web.Mvc.Async.<>c__DisplayClass48.<InvokeActionMethodFilterAsynchronouslyRecursive>b__41() +112
System.Web.Mvc.Async.<>c__DisplayClass48.<InvokeActionMethodFilterAsynchronouslyRecursive>b__41() +452
System.Web.Mvc.Async.<>c__DisplayClass48.<InvokeActionMethodFilterAsynchronouslyRecursive>b__41() +452
System.Web.Mvc.Async.<>c__DisplayClass48.<InvokeActionMethodFilterAsynchronouslyRecursive>b__41() +452
System.Web.Mvc.Async.<>c__DisplayClass33.<BeginInvokeActionMethodWithFilters>b__32(IAsyncResult asyncResult) +15
System.Web.Mvc.Async.<>c__DisplayClass2b.<BeginInvokeAction>b__1c() +37
System.Web.Mvc.Async.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult) +241
System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult asyncResult, ExecuteCoreState innerState) +29
System.Web.Mvc.Async.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) +111
System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult) +53
System.Web.Mvc.Async.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) +19
System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState) +51
System.Web.Mvc.Async.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) +111
System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +606
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +288
We have an action filter that calls the context.SaveChanges() method if everything went well on the request (so we don't have to do that manually in every action). That's why there is no action on the stacktrace, it times out on the SaveChanges() call when the action finished.
There is a lot happening on that request, that's why I'm not showing code, we are doing the usual stuff: updating some rows, inserting new ones, etc. There are some complex commands in there, but not complex enough for the db to timeout. And when we run the code on our dev machines with the same production db it runs ok.
The weird thing is that it stops timing out after a few hours. But we can't risk going to production like that.
Now, I know it's unlikely that you can give me a solution just with the information provided. But I'd like to know some tips, pointers, advice, where to look, what to check, etc.
Edit
Our server has 12GB of RAM and an Intel Xeon E5-2680 # 2.80GHz.
Our site is the only thing running there.
Our database is less than 4GB
And we don't get timeout on our dev machines.
We finally found what was happening:
We have a seed method that was executed after the migration, that method opened a connection to the database and was also querying some external services that took forever to respond, it kept running for hours and those queries were blocking the insert command.
Thanks to Zdravko's suggestion we were able to find the issue.
We ran exec sp_who2 and found that the insert was being blocked by the seed method's select query.
So, if anyone gets here with a similar problem: what helped us was using the sql profiler and filtering for queries taking longer than 30 sec to find the offending queries and using exec sp_who2 to check if someone is blocking them.
Timeouts are most likely due to SQL server rebuilding indexes and other maintenance jobs running after the restore. Those can take significant amount of time on a larger database. Full text search indexes for example take particularly long time to build...
I have a weird situation at certain times I get these errors sent to me via email. This last one lasted 10 minutes. My heavy queries start to timeout. The event, database and IIS logs look just fine so I am baffled. Could this be a disk issue, and if so how can I find out?
System.ComponentModel.Win32Exception: The wait operation timed out
Generated: Wed, 16 Apr 2014 23:05:20 GMT
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
Use PerfMon to analyze disk bottleneck.
Then memory pressure.
And then issues on the network.