Slow Save in Entity Framework, very specific entity - sql-server

Question

I have a very interesting (and confusing) issue for quite a while, but cannot get to the bottom of it
I have a WPF Project built on Entity Framework, with about 100 different entities/tables, which all work smoothly. except for one entity type, named SaleProduct
Saving a SaleProduct takes about 5 minutes, sometimes even more (other entities take under 2 seconds usually). And only in the window.
For testing purposes, I put this code in app startup:
Using d = GetSystemContext(True)
Dim sp = d.Find(Of SaleProduct)(66246)
sp.DiscountAmount += 1
d.SaveChanges()
End Using
And saving there is OK(11 seconds, bearable).
But when I open a window with the SaleProduct as the DataContext and save there, it takes several minutes.
So, I would like to debug this. Maybe I have some bad code somewhere. But I'm not managing with this either. If I press pause in Visual Studio while the app is stuck saving, it freezes on the SaveChanges line, meaning not some custom code. So what difference should it make if we are in the app startup or in the window?
But lets take this a step further, I disabled Just My Code to get a better picture of whats freezing. And it gets stuck here:
InternalDispatcher.cs
try
{
interceptionContext.MutableData.SetExecuted(operation(target, interceptionContext));
}
Where the target is a pretty simple SQL statement
UPDATE [dbo].[Sales]
SET [LastEditedOn] = #0
WHERE ([ID] = #1)
SELECT [AutoPurchaseDate], [GrantingDate], [ReceivedOn], [Name], [TotalVatRate]
FROM [dbo].[Sales]
WHERE ##ROWCOUNT > 0 AND [ID] = #1
Running this in SSMS takes <1 seconds.
But looking at the complete call stack I see that actually its stuck someplace in System.Data.dll. Though try as I might I cant find its .pdb.
So I tried to use decompile, but I get an error
Unable to compile the module
Even though, in the modules window the status is "Symbols loaded".
So I cannot pinpoint precisely.
For kicks, I tried upgrading to .net 4.8 (from 4.7). same problem.
What can I do? How to proceed? Where more can I Investigate?
NOTES
Tables Sales and SaleProducts both have Clustered Primary Keys.
Running SELECT * FROM sys.sysprocesses WHERE open_tran = 1 in SSMS while the code is stuck, returns 0 rows.
InSaveChanges, ChangeTracker.Entries returns precisely 2 entities. The SaleProduct, and its parent Sale.
I've posted a Profiler Trace here https://pastebin.com/enLgE0Ln
Here's the full StackTrace if it might help someone
[Managed to Native Transition]
System.Data.dll!SNINativeMethodWrapper.SNIReadSyncOverAsync(System.Runtime.InteropServices.SafeHandle pConn, ref System.IntPtr packet, int timeout) Unknown
System.Data.dll!System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync() Unknown
System.Data.dll!System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() Unknown
System.Data.dll!System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() Unknown
System.Data.dll!System.Data.SqlClient.TdsParserStateObject.TryReadByte(out byte value) Unknown
System.Data.dll!System.Data.SqlClient.TdsParser.TryRun(System.Data.SqlClient.RunBehavior runBehavior, System.Data.SqlClient.SqlCommand cmdHandler, System.Data.SqlClient.SqlDataReader dataStream, System.Data.SqlClient.BulkCopySimpleResultSet bulkCopyHandler, System.Data.SqlClient.TdsParserStateObject stateObj, out bool dataReady) Unknown
System.Data.dll!System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() Unknown
System.Data.dll!System.Data.SqlClient.SqlDataReader.MetaData.get() Unknown
System.Data.dll!System.Data.SqlClient.SqlCommand.FinishExecuteReader(System.Data.SqlClient.SqlDataReader ds, System.Data.SqlClient.RunBehavior runBehavior, string resetOptionsString, bool isInternal, bool forDescribeParameterEncryption, bool shouldCacheForAlwaysEncrypted) Unknown
System.Data.dll!System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(System.Data.CommandBehavior cmdBehavior, System.Data.SqlClient.RunBehavior runBehavior, bool returnStream, bool async, int timeout, out System.Threading.Tasks.Task task, bool asyncWrite, bool inRetry, System.Data.SqlClient.SqlDataReader ds, bool describeParameterEncryptionRequest) Unknown
System.Data.dll!System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior cmdBehavior, System.Data.SqlClient.RunBehavior runBehavior, bool returnStream, string method, System.Threading.Tasks.TaskCompletionSource<object> completion, int timeout, out System.Threading.Tasks.Task task, out bool usedCache, bool asyncWrite, bool inRetry) Unknown
System.Data.dll!System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior cmdBehavior, System.Data.SqlClient.RunBehavior runBehavior, bool returnStream, string method) Unknown
System.Data.dll!System.Data.SqlClient.SqlCommand.ExecuteReader(System.Data.CommandBehavior behavior, string method) Unknown
> EntityFramework.dll!System.Data.Entity.Infrastructure.Interception.InternalDispatcher<System.Data.Entity.Infrastructure.Interception.IDbCommandInterceptor>.Dispatch<System.Data.Common.DbCommand, System.Data.Entity.Infrastructure.Interception.DbCommandInterceptionContext<System.Data.Common.DbDataReader>, System.Data.Common.DbDataReader>(System.Data.Common.DbCommand target, System.Func<System.Data.Common.DbCommand, System.Data.Entity.Infrastructure.Interception.DbCommandInterceptionContext<System.Data.Common.DbDataReader>, System.Data.Common.DbDataReader> operation, System.Data.Entity.Infrastructure.Interception.DbCommandInterceptionContext<System.Data.Common.DbDataReader> interceptionContext, System.Action<System.Data.Entity.Infrastructure.Interception.IDbCommandInterceptor, System.Data.Common.DbCommand, System.Data.Entity.Infrastructure.Interception.DbCommandInterceptionContext<System.Data.Common.DbDataReader>> executing, System.Action<System.Data.Entity.Infrastructure.Interception.IDbCommandInterceptor, System.Data.Common.DbCommand, System.Data.Entity.Infrastructure.Interception.DbCommandInterceptionContext<System.Data.Common.DbDataReader>> executed) Line 138 C#
EntityFramework.dll!System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(System.Data.Common.DbCommand command, System.Data.Entity.Infrastructure.Interception.DbCommandInterceptionContext interceptionContext) Line 103 C#
EntityFramework.dll!System.Data.Entity.Core.Mapping.Update.Internal.DynamicUpdateCommand.Execute(System.Collections.Generic.Dictionary<int, object> identifierValues, System.Collections.Generic.List<System.Collections.Generic.KeyValuePair<System.Data.Entity.Core.Mapping.Update.Internal.PropagatorResult, object>> generatedValues) Line 118 C#
EntityFramework.dll!System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.Update() Line 532 C#
EntityFramework.dll!System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction<int>(System.Func<int> func, System.Data.Entity.Infrastructure.IDbExecutionStrategy executionStrategy, bool startLocalTransaction, bool releaseConnectionOnSuccess) Line 2203 C#
EntityFramework.dll!System.Data.Entity.Core.Objects.ObjectContext.SaveChangesToStore(System.Data.Entity.Core.Objects.SaveOptions options, System.Data.Entity.Infrastructure.IDbExecutionStrategy executionStrategy, bool startLocalTransaction) Line 2148 C#
EntityFramework.SqlServer.dll!System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute<int>(System.Func<int> operation) Unknown
EntityFramework.dll!System.Data.Entity.Core.Objects.ObjectContext.SaveChangesInternal(System.Data.Entity.Core.Objects.SaveOptions options, bool executeInExistingTransaction) Line 2046 C#
EntityFramework.dll!System.Data.Entity.Internal.InternalContext.SaveChanges() Line 483 C#
Machshevet.dll!Machshevet.Core.RecordContext.SaveChanges() Line 762 Basic
Machshevet.dll!Machshevet.Windows.RecordWindow.Save() Line 169 Basic
Machshevet.dll!Machshevet.Windows.RecordWindow.<closure>.<lambda35-0>() Line 44 Basic
PresentationCore.dll!System.Windows.Input.CommandBinding.OnExecuted(object sender, System.Windows.Input.ExecutedRoutedEventArgs e) Unknown
PresentationCore.dll!System.Windows.Input.CommandManager.ExecuteCommandBinding(object sender, System.Windows.Input.ExecutedRoutedEventArgs e, System.Windows.Input.CommandBinding commandBinding) Unknown
PresentationCore.dll!System.Windows.Input.CommandManager.FindCommandBinding(System.Windows.Input.CommandBindingCollection commandBindings, object sender, System.Windows.RoutedEventArgs e, System.Windows.Input.ICommand command, bool execute) Unknown
PresentationCore.dll!System.Windows.Input.CommandManager.FindCommandBinding(object sender, System.Windows.RoutedEventArgs e, System.Windows.Input.ICommand command, bool execute) Unknown
PresentationCore.dll!System.Windows.Input.CommandManager.OnExecuted(object sender, System.Windows.Input.ExecutedRoutedEventArgs e) Unknown
PresentationCore.dll!System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate handler, object target) Unknown
PresentationCore.dll!System.Windows.RoutedEventHandlerInfo.InvokeHandler(object target, System.Windows.RoutedEventArgs routedEventArgs) Unknown
PresentationCore.dll!System.Windows.EventRoute.InvokeHandlersImpl(object source, System.Windows.RoutedEventArgs args, bool reRaised) Unknown
PresentationCore.dll!System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject sender, System.Windows.RoutedEventArgs args) Unknown
PresentationCore.dll!System.Windows.Input.RoutedCommand.ExecuteImpl(object parameter, System.Windows.IInputElement target, bool userInitiated) Unknown
PresentationCore.dll!System.Windows.Input.CommandManager.TransferEvent(System.Windows.IInputElement newSource, System.Windows.Input.ExecutedRoutedEventArgs e) Unknown
PresentationCore.dll!System.Windows.Input.CommandManager.OnExecuted(object sender, System.Windows.Input.ExecutedRoutedEventArgs e) Unknown
PresentationCore.dll!System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate handler, object target) Unknown
PresentationCore.dll!System.Windows.RoutedEventHandlerInfo.InvokeHandler(object target, System.Windows.RoutedEventArgs routedEventArgs) Unknown
PresentationCore.dll!System.Windows.EventRoute.InvokeHandlersImpl(object source, System.Windows.RoutedEventArgs args, bool reRaised) Unknown
PresentationCore.dll!System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject sender, System.Windows.RoutedEventArgs args) Unknown
PresentationCore.dll!System.Windows.Input.RoutedCommand.ExecuteImpl(object parameter, System.Windows.IInputElement target, bool userInitiated) Unknown
PresentationFramework.dll!MS.Internal.Commands.CommandHelpers.CriticalExecuteCommandSource(System.Windows.Input.ICommandSource commandSource, bool userInitiated) Unknown
PresentationFramework.dll!System.Windows.Controls.Primitives.ButtonBase.OnClick() Unknown
PresentationFramework.dll!System.Windows.Controls.Button.OnClick() Unknown
System.Windows.Controls.Ribbon.dll!System.Windows.Controls.Ribbon.RibbonButton.OnClick() Unknown
PresentationFramework.dll!System.Windows.Controls.Primitives.ButtonBase.OnMouseLeftButtonUp(System.Windows.Input.MouseButtonEventArgs e) Unknown
PresentationCore.dll!System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate handler, object target) Unknown
PresentationCore.dll!System.Windows.RoutedEventHandlerInfo.InvokeHandler(object target, System.Windows.RoutedEventArgs routedEventArgs) Unknown
PresentationCore.dll!System.Windows.EventRoute.InvokeHandlersImpl(object source, System.Windows.RoutedEventArgs args, bool reRaised) Unknown
PresentationCore.dll!System.Windows.UIElement.ReRaiseEventAs(System.Windows.DependencyObject sender, System.Windows.RoutedEventArgs args, System.Windows.RoutedEvent newEvent) Unknown
PresentationCore.dll!System.Windows.UIElement.OnMouseUpThunk(object sender, System.Windows.Input.MouseButtonEventArgs e) Unknown
PresentationCore.dll!System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate handler, object target) Unknown
PresentationCore.dll!System.Windows.RoutedEventHandlerInfo.InvokeHandler(object target, System.Windows.RoutedEventArgs routedEventArgs) Unknown
PresentationCore.dll!System.Windows.EventRoute.InvokeHandlersImpl(object source, System.Windows.RoutedEventArgs args, bool reRaised) Unknown
PresentationCore.dll!System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject sender, System.Windows.RoutedEventArgs args) Unknown
PresentationCore.dll!System.Windows.UIElement.RaiseTrustedEvent(System.Windows.RoutedEventArgs args) Unknown
PresentationCore.dll!System.Windows.Input.InputManager.ProcessStagingArea() Unknown
PresentationCore.dll!System.Windows.Input.InputManager.ProcessInput(System.Windows.Input.InputEventArgs input) Unknown
PresentationCore.dll!System.Windows.Input.InputProviderSite.ReportInput(System.Windows.Input.InputReport inputReport) Unknown
PresentationCore.dll!System.Windows.Interop.HwndMouseInputProvider.ReportInput(System.IntPtr hwnd, System.Windows.Input.InputMode mode, int timestamp, System.Windows.Input.RawMouseActions actions, int x, int y, int wheel) Unknown
PresentationCore.dll!System.Windows.Interop.HwndMouseInputProvider.FilterMessage(System.IntPtr hwnd, MS.Internal.Interop.WindowMessage msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled) Unknown
PresentationCore.dll!System.Windows.Interop.HwndSource.InputFilterMessage(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled) Unknown
WindowsBase.dll!MS.Win32.HwndWrapper.WndProc(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled) Unknown
WindowsBase.dll!MS.Win32.HwndSubclass.DispatcherCallbackOperation(object o) Unknown
WindowsBase.dll!System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate callback, object args, int numArgs) Unknown
WindowsBase.dll!System.Windows.Threading.ExceptionWrapper.TryCatchWhen(object source, System.Delegate callback, object args, int numArgs, System.Delegate catchHandler) Unknown
WindowsBase.dll!System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority priority, System.TimeSpan timeout, System.Delegate method, object args, int numArgs) Unknown
WindowsBase.dll!MS.Win32.HwndSubclass.SubclassWndProc(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam) Unknown
[Native to Managed Transition]
[Managed to Native Transition]
WindowsBase.dll!System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame frame) Unknown
PresentationFramework.dll!System.Windows.Application.RunDispatcher(object ignore) Unknown
PresentationFramework.dll!System.Windows.Application.RunInternal(System.Windows.Window window) Unknown

Answer 1

The specific answer to this issue will require seeing the specific code responsible. It likely is not an issue with a single table beyond the relationships that table has defined within the DbContext and database schema, and the current state of what the DbContext is tracking. (And what the Database might be locking, though that is usually a more intermittent type of problem)
Some things to check:
In your App example, you appear to be getting a fresh instance of the DbContext and disposing it. (Good) Does the 11s time reflect the Find+Save, or just Save? The very first query that a DbContext executes in an application incurs a 1-off (think Static) spin-up cost to initialize the model. For a more accurate timing with a first run scenario, you could do something like:
Using d = GetSystemContext(True)
Dim dump = d.Sales.Any()
' Start timing here...
Dim sp = d.Find(Of SaleProduct)(66246)
sp.DiscountAmount += 1
d.SaveChanges()
' End timing here...
End Using
*Sorry, my VB.Net is sadly out of date.. :) In any case, using a quick operation against the DbContext like an Any() will incur the 1-off spin-up cost. Any operation after that should be a more pure representation of the time needed to load & save data.
In your problematic code, are DbContext's guaranteed to be short-lived like this? DbContexts that are alive longer accrue references to a number of entities and any time any entity is persisted, it will scour it's tree of entities to determine if any related ones may need to be updated. Short-lived, disposed DbContexts are the way to go.
Have you run a profiler against the database to capture the exact SQL being run? Your example SQL doesn't look like an EF query, and references a "Sales" table rather than "SalesProduct". Having a profiler running while debugging your EF application operations is invaluable for revealing stuff going on behind the scenes that you don't expect.
Other possibilities would really depend on the code in question, but a profiler combined with breakpoints will generally help spot issues. Is the delay confirmed to be just when DbContext.SaveChanges happens, or the whole save process? If it's something like from the time the user clicks a Save button, then a Profiler might reveal that your code is tripping a lot of lazy-load calls (loading data to check and update prior to saving) or loading unexpected data. (common issues I've seen are misplaced ToList() calls before filtering statements, usually because the devs want to do something that EF cannot translate and ToList() seems like a "fix") If the delay is just from SaveChanges then the profiler can confirm what Update statements and the like are being produced and how long they are taking to run.

Answer 2

In the end it seems that the error disappeared after a index rebuild. Even though I cant understand if it was an index issue, how come it worked flawlessly time and again via SSMS and EF without window. But for now its working just dandy. Thanks again for all posters, as you were extremely helpful in narrowing down and focusing.

Answer 3

Entity framework is slow , very slow
in production code you need to use c# SQL command