We have a DMV query that executes every 10 mins and inserts usage statistics, like SESSION_CURRENT_DATABASE, SESSION_LAST_COMMAND_STARTTIME, etc.. and supposedly has been running fine for the last 2 years.
Today we were notified by data hyperingestion team that the last records shown were from 6/10. So we found out the job has been stuck for 14 days not executing new statistics since. We've immediately restarted the job and it's been executing successfully since the morning, but basically we've lost the data during this 14 days period. Is there a way for us to execute this DMV query between 6/10-6/24 on the $SYSTEM.DISCOVER to recover these past 14 days of data?
Or all hope's lost?
DMV query:
SELECT [SESSION_ID]
,[SESSION_SPID]
,[SESSION_CONNECTION_ID]
,[SESSION_USER_NAME]
,[SESSION_CURRENT_DATABASE]
,[SESSION_USED_MEMORY]
,[SESSION_PROPERTIES]
,[SESSION_START_TIME]
,[SESSION_ELAPSED_TIME_MS]
,[SESSION_LAST_COMMAND_START_TIME]
,[SESSION_LAST_COMMAND_END_TIME]
,[SESSION_LAST_COMMAND_ELAPSED_TIME_MS]
,[SESSION_IDLE_TIME_MS]
,[SESSION_CPU_TIME_MS]
,[SESSION_LAST_COMMAND_CPU_TIME_MS]
,[SESSION_READS]
,[SESSION_WRITES]
,[SESSION_READ_KB]
,[SESSION_WRITE_KB]
,[SESSION_COMMAND_COUNT]
FROM $SYSTEM.DISCOVER_SESSIONS
I wouldn't say it's "gone" unless the instance has been restarted or the db has been detached. For example, the dmv for procedure usage should still have data in it, but you won't be able to specifically recreate what it looked like 10 days ago.
You can get a rough idea by looking back through the 2 years of data you already have, and get a sense of if there are spikes or consistent usage. Then, grab a snapshot of the DMV today, and extrapolate it back 14 days to get a rough idea of what usage was like.
Related
I have a query that has around 160 million rows that would take me 4 hours to import into Power BI.
Ever since my company changed their server to Azure, I can never import this query successfully. It would start loading for 1 million rows-ish, after 1 minute or two this error always pops out.
I tried:
changing the command time out to 200 minutes, still errors out within loading for a minute or two, sometimes within 10 seconds
if I select top 1000 rows in my query, it will complete without error. But when I switch back to the original query, it always fails.
Attaching the error message. I have talked to the DE in my team and they don't seem to have a clue. Does anyone have any idea on how to fix this?
this is the error message
We had the same problem and went for a delta lake in combination with option 1.
You have to ask your self first why you are importing so much data. Always keep your model as small as possible. I can't imagine you are looking at every row. If this is needed you could you could use a combination of direct query for details and loading an aggregate for your reporting. But load aggregates instead of everything.
Maybe your can load less history, like the last two years.
You could look into loading incrementally, you could load per partition.
You can try to increase the DTU's for your server.
I have a MSSQL2005 database that has records dating back to 2004, there are currently just under 1,000,000 records in one particular table.
Thing is, if I run a report comparing 2009 data against 2010 data, 2008 against 2009, 2009 against 2009 or any combination of years before this year then results are returned in 1-5 seconds.
If however I run a report that includes 2011 data then the report takes ~6 minutes.
I've checked the data and it looks similar to previous years and is cross-referenced against the same data used in all of the reports.
It's as if the database has exceeded some limit; that data for this year has become fragmented and therefore harder to access. I'm not saying this is the case but it may be for all I know.
Anyone have any suggestions?
Shaun.
Update:Since posting the question I found DBCC DBREINDEX table_name which seems to have done the trick.
What do the execution plans look like? If different you might need to manually update statistics on the table as the newly inserted rows are likely to be disproportionately unrepresented in the statistics and it might thus choose a sub optimal plan.
See this blog post for an explanation of this issue Statistics, row estimations and the ascending date column
Additionally check that your 2011 query isn't encountering blocking due to concurrent inserts or updates that do not affect queries against historic data.
I am using the default SQL Server storage for Hangfire. Since the rate of job creation per-day can be quite high (in the count of hundreds of thousands), the size of the database also grows quite fast.
Initially I thought it will still be fine because Succeeded jobs would be removed after one day. To my surprise, even after several days, the job is not yet removed from Hangfire.Jobs table.
When I ran this query on 30 July 2015:
select min(ExpireAt) from Hangfire.Job
where StateName = 'Succeeded'
I got 2015-07-21 13:28:27.543, which is about 9 days ago.
I can manually delete the jobs myself or create a batch job that would periodically clean it up, but is there any scenario where the Succeeded job are still retained beyond their ExpireAt?
I see a similar question here from 2013, but it was not answered so I am posting my version.
We are using SQL Server 2008 (SP4) - 10.0.6000.29 (X64) and have a database that is about 70GB in size with about 350 tables. On a daily basis, there are only a small number of updates occurring, though a couple times a year we dump a fair amount of data into it. There are several Windows Services that constantly query the database, but rarely updated it. There are also several websites that use it, and desktop applications (again, minimal daily updates).
The problem we have is that every once in a while a query that hits certain records will take much longer than normal. The following is a bogus example:
This query against 2 tables with less than 600 total records might take 30+ seconds:
select *
from our_program_access bpa
join our_user u on u.user_id = bpa.user_id
where u.user_id = 50 and program_name = 'SomeApp'
But when you change the user_id value to another user record, it takes less than one second:
select *
from our_program_access bpa
join our_user u on u.user_id = bpa.user_id
where u.user_id = 51 and program_name = 'SomeApp'
The real queries that are being used are a little more complex, but the idea is the same: search ID 50 takes 30+ seconds, search ID 51 takes < 1 second, but both return only 1 record out of about 600 total.
We have found that the issue seems related to the statistics. When this problem occurs, we run sp_updatestats, and all the queries are equal and fast in time. So, we started to run sp_updatestats in a maintainenance plan every night. But the problem still pops up. We also tried setting AUTO_UPDATE_STATISTICS_ASYNC on, but the problem eventually popped up.
While the database is large, it doesn't really undergo tremendous changes, though it does face constant queries from different services.
There are several other databases on the same server such as a mail log, SharePoint, and web filtering. Overall, performance is very good until we run into this problem.
Does it make sense that on a database that undergoes relatively small changes daily would need to run sp_updatstats so frequently? What else can we do to resolve this issue?
I have run three queries in SSMS and cannot understand the difference in time it takes each of them to run. The first two queries extract data for a single month, the third extracts data for both months. All queries work correctly, but I cannot understand how the second one can take so long to run.
The queries, the number of records returned and the time taken for each query is as follows:
Can anyone explain this?
SELECT *
FROM [vw_Movement]
WHERE ShiftDateTime BETWEEN '1 Aug 14 6:00' AND '31 Aug 14 18:00'
(Rcds returned=16,342, time=0 secs)
SELECT *
FROM [vw_Movement]
WHERE ShiftDateTime BETWEEN '1 Sep 14 6:00' AND '30 Sep 14 18:00'
(Rcds returned=14,468, time=24 secs)
SELECT *
FROM [vw_Movement]
WHERE ShiftDateTime BETWEEN '1 Aug 14 6:00' AND '30 Sep 14 18:00'
(Rcds returned=30,810, time=0 secs)
Have you looked at the execution plan (CTL-L in SSMS)? You don't say whether or not the underlying data is in a view or table, but from the vw_ prefix, I'm going to go with view.
So you might also want to try looking at the view definition itself, copy it as a new query into SSMS, put the WHERE statement at the end, and analyze the entire query instead of just the view. Maybe by commenting out some of the joins you can narrow down the problem.
It likely is related to
(1) statistics - you can update the statistics of the underlying tables for thatview during your maintenance window. People warn of running update stats during normal business hours, I feel that depends on your workload. Update stats doesn't lock the tables from my experience should be okay but it really depends on how much the db server/host can handle.
(2) Bad query plan cached in procedure cache. If you search the internet you can find methods to specifically remove on your specific cached query plan. The bad plan is also related to bad old statistics.
FYI - By updating stats you automatically your query will be recompiled due to change in stats. Taking action of (2) should not be needed.