I am using a Silverlight DataGrid with a DomainDataSource and a DataPager and EF 4
When using MSSQL server profiler, I noticed 2 queries which we taking the bulk of the data retrieval time. One query gets the data for the given load size, and another which gets the total page count. The one getting the page count is very slow for large sets of data, much slower than getting the data itself!
So my question is this: Is it possible to suppress this query? I know the datapager needs to know how many pages there are but I think I can work around that if I have to
Thanks
setting 'IncludeTotalCount' on the query to false worked.
I did this in the override method of Load in the database context but iguess it can be done on a specific query.
Related
In an ASP.NET MVC application using EF 6 with SQL Server, when updating a table to change one particular row, it takes a very long time (10 minutes plus, and only sometimes the change ultimately gets through).
However, using the same web page to update any other row in the same table, it's immediate. Also, when I open SQL Server Management Studio and use an update query to update that specific row, it's immediate as well, and so is changing the row through the Edit Top 200 Records functionality.
The table in question holds various statuses used for keeping track record processing (there are 23 records in the table). It has an ID which is the primary key (only column referenced by other tables), and it has Name and Description columns. I'm changing the description in the example above.
As the row I'm changing is for the OK status, which is the most used one, the only thing I could come up with is that somehow all records referencing this status are also updated or at least checked, but besides the fact that this is not exactly how relational databases work, that would also still not explain why the update is immediate when I use a query in SSMS. Hence my assumption that this is somehow caused by EF ding or checking something in the background.
Unfortunately this is on a production environment where I have very limited access or debugging options. On the TEST and ACCEPTANCE environments it is working normally.
Any ideas what might cause this behavior?
Thanks, Patrick
Thanks all for taking the time to try and help me out here. I managed to get some debug messages in the controller code, and it turns out that the controller method called by the page submit is not even hit most of the time. I don't see any differences in the generated html between the view for the offending record and the views of any of the other records, so it still strikes me as weird that the same page seems to act differently with only 1 specific record, but at least now I know I have to look for the answer in ASP/MVC, and not EF or the db.
Thanks again!
My data source cannot be a single table as I need data that spans across 6 tables. For that I have created a view that does joins on these tables.
When I use this view as data source, indexing takes a lot of times and times out. I tried increasing timeout to 40 minutes and one more suggested change:
"disableOrderByHighWaterMarkColumn" : true
It timed-out. I also set Batch Size:1000. This time it populated the index but failed after few hours saying "connection lost". and "thanks" to disableOrderByHighWaterMarkColumn, If I rerun indexer again it will process all the rows again.
My question is what is the best way to approach a solution to this problem.
Follow-up Question: Since I am relying on a view, I cannot have auto change tracking. I am using a high watermark column (LastUpdatedTime) to track changes in my View. I only want to keep 6 months of data in my index so I am not sure how I can do that when I am using View. I have "where CreateDateTime > dateadd(month, -6, getdate())" clause already in my View but this will not enable Indexer to delete "out-of-time-window" rows(documents) from index. How can I achieve my goals here?
Should I write a processor task to periodically query all documents using C# SDK and delete documents based on date?
Sorry to hear the Azure SQL Database indexer is giving you trouble. I noticed a couple of things in your question that might be worth thinking about in terms of SQL performance:
My data source cannot be a single table as I need data that spans across 6 tables. For that I have created a view that does joins on these tables When I use this view as data source, indexing takes a lot of times and times out.
It's worth taking a look at the query performance troubleshooting guide and figure out what exactly is happening in your Azure SQL database that is causing problems. Assuming you want to use change tracking support, the default query the indexer uses against the SQL database looks like this:
SELECT * FROM c WHERE hwm_column > #hwmvalue ORDER BY hwm_column
We frequently see issues with performance here when there isn't an index on the hwm_column or if hwm_column is computed. You can read more about issues with the high water mark column here.
I tried increasing timeout to 40 minutes and one more suggested change: "disableOrderByHighWaterMarkColumn" : true It timed-out. I also set Batch Size:1000. This time it populated the index but failed after few hours saying "connection lost". and "thanks" to disableOrderByHighWaterMarkColumn, If I rerun indexer again it will process all the rows again.
disableOrderByHighWaterMarkColumn doesn't seem like it will work for your scenario, so I agree that you shouldn't set it. Decreasing the batch size seems to have had a positive effect, I would consider measuring the performance gain here using the troubleshooting guide referenced above
Follow-up Question: Since I am relying on a view, I cannot have auto change tracking. I am using a high watermark column (LastUpdatedTime) to track changes in my View. I only want to keep 6 months of data in my index so I am not sure how I can do that when I am using View. I have "where CreateDateTime > dateadd(month, -6, getdate())" clause already in my View but this will not enable Indexer to delete "out-of-time-window" rows(documents) from index. How can I achieve my goals here? Should I write a processor task to periodically query all documents using C# SDK and delete documents based on date?
Instead of filtering out data that is more than 6 months old, I would consider adding soft delete policy. The challenge here is that the indexer needs to pick up rows that should be deleted. The easiest way to accomplish this might updating your application logic to add a new column to your view indicating the row should be deleted. Once the value of this column changes, the LastUpdatedTime should also be updated so it shows up in the next indexer query.
You can write your own processor task, but querying all documents in Azure Cognitive Search and paging through them may have negative performance implications on your search performance. I would recommend trying to get it working with your indexer first.
I noticed even the simplest 'SELECT MAX(TIMESTAMP) FROM MYVIEW' is somewhat slow (taking minutes) in my environment, and found it's doing a TableScan of 100+GB across 80K micropartitions.
My expectation was this to finish in milliseconds using MIN/MAX/COUNT metadata in each micropartitions. In fact, I do see Snowflake finishing the job in milliseconds using metadata for almost same MIN/MAX value lookup in following article:
http://cloudsqale.com/2019/05/03/performance-of-min-max-functions-metadata-operations-and-partition-pruning-in-snowflake/
Is there any limitation in how Snowflake decides to use metadata? Could it be because I'm querying through a view, instead of querying a table directly?
=== Added for clarity ===
Thanks for answering! Regarding how the view is defined, it seems to adds a WHERE clause for additional filtering with a cluster key. So I believe it should still be possible to fully use metadata of miropartitions. But as posted, TableScan is being done in profilter output.
I'm bit concerned on your comment on SecureView. The view I'm querying is indeed a SECURE VIEW - does it affect how optimizer handles my query? Could that be a reason why TableScan is done?
It looks like you're running the query on a view. The metadata you're referring to will be used when you're running a simple MIN MAX etc on the table, however if you have some logic inside your view which requires filtering / joining of data then Snowflake cannot return results just based off the metadata.
So yes, you are correct when you say the following because your view is probably doing something other than a simple MAX on the table:
...Could it be because I'm querying through a view, instead of querying a table directly?
I have an angular 4 UI, web API and SQL server in the back-end. I use a third party grid from ag grid (https://www.ag-grid.com/).
Here are some info about my scenario :
I have total 1 million records in SQL server
I need to group the data based on user selection from the UI. Typically, after the grouping, the number of records to fetch from DB becomes something around 200 thousands.
3.User will do some filtering from the UI as well which makes the number of records to fetch will become something around 4000 to 5000.
I implemented paging (page size 100) at server side. I send offset and number of records to fetch from DB . The problem in this case is, since it is a huge data , my grouping at the database takes 15 seconds to finish which is quite a lot of time.
Another approach I tried is, get the paged data from DB without grouping and do the grouping in the API which is kind of finishing in milliseconds. The problem here is, I am showing some values in the UI based on groups such as count and sum of the some fields , the data shown in the currently fetched data might not be correct/complete as the the current group may span to the other pages in the DB which are not fetched yet.
Any recommendations to deal with this kind of scenario would be appreciated.
Thanks.
I have 1 application module, 1 connection to DB and two DataControls based on a single ViewObject. They are placed on the same form. Is it any possibility that ADF makes 2 sessions when I insert data to first DataControl and trying to re-execute query in second?
Yours is not practically a problem. It is the way it should work. Two users cannot update/change the same row in the same time. The first that Commits the change is OK whereas to the second an Error popup will be displayed telling him that the current row has been updated from someone else. If the users are not working (changing) the same row but different rows of the same ViewObject then you should consider this link:
http://radio-weblogs.com/0118231/stories/2004/03/24/whyDoIGetOraclejborowinconsistentexception.html
I suggest you take a look also to this book, you can find it free to download, just search a bit.
http://www.amazon.com/Quick-Start-Oracle-Fusion-Development/dp/0071744282
Have a nice day, tung.