I am parameterizing my web app's ad hoc sql. As a result, I expect the query plan cache to reduce in size and have a higher hit ratio. Perhaps even other important metrics will be improved.
Could I use perfmon to track this? If so, what counters should I use? If not perfmon, how could I report on the impact of this change?
SQL Server, Plan Cache Object
Cache Hit Ratio Ratio between cache hits and lookups.
Cache Object Counts Number of cache objects in the cache.
Cache Pages Number of 8-kilobyte (KB) pages used by cache objects.
Cache Objects in use Number of cache objects in use.
Also sys.dm_os_memory_clerks and sys.dm_os_memory_cache_counters will give information about memory allocations (in general) and SQL caches (in general). You'll be interested in allocation for the plan cache memory clerk.
And finally there are the execution DMVs: sys.dm_exec_query_stats and sys.dm_exec_cached_plans.
These counters and DMVs should cover what you need, for more details see Execution Plan Caching and Reuse.
You can use SQL Server Profiler. Create a new trace, and capture the TSQL->Exec Prepared Sql and TSQL->Prepare Sql events. The former will tell you when it's reusing a query plan, the latter when it is regenerating the plan.
You can do the same for Stored Procedures as well, under the SP category of events.
Related
In AWS Redshift we can manage query priority using WLM. Do we have any such feature for Snowflake or is it done using multi warehouse strategy?
I think you've got the right idea that warehouses are typically the best approach this problem in Snowflake.
If you have a high priority query/process/account, it's entirely reasonable to provide that with a dedicated warehouse. That will guarantee that your query won't be competing with any resources on other warehouses.
You can also then size that warehouse appropriately. if it's a Small query, or file copy query, for example, and it's just really important that it runs right away, then you can give it a dedicated Small/X-Small warehouse. If it's a big query that doesn't run very frequently, you can give it a larger warehouse. If you set it to auto-suspend then you won't even incur much extra cost for the extra dedicated compute.
Background
We have an export functionality which allows the user to export measurements from many devices for a certain time range.
The database is continuously being updated with sensor data from many devices (at 10 to 50 measurements per second from each device). So since these CSV exports often contain millions of rows (each row containing data from several different devices, where each device is in a separate database), it was written to fetch data from these tables in chunks (I guess to avoid slowing down inserts, since exporting is a lower priority task).
Problem
When the first chunk is queried, SQL Server will create an execution plan which fits the passed parameters. However, at the beginning of the export, it's possible that ranges of data are missing due to low connectivity, or are flagged differently due to time sync errors, meaning the following queries reusing this cached plan will possibly not get the optimal plan for their parameters.
One option is to add OPTION (RECOMPILE) to each query, but many sources claim that this will impose unnecessary burden on the CPU and that merely updating statistics on a regular base should help SQL Server create a better plan.
However, this doesn't make too much sense, because even if the plan cache is invalidates, the next time a user creates the query, first chunk will again dictate the "shape" of the execution plan.
Is this reasoning correct?
Should I actually use OPTION (RECOMPILE) with each query, or just add an "Update Statistics" maintenance plan?
I have a stored procedure that takes up too much of server resources. I want to limit this query to only use no more than, say, 10% of the reosurces. I've heard of Resource governor, but it is only available for enterprise and developer editions. I have a standard edition.
Is there any other alternatives to this, except for buying a better version of sql server?
Define 'resources'. IO? CPU? Memory? Network? Contention? Log size? tempdb? And you cannot answer 'all'. You need to specify what resources are being consumed now by the procedure.
Your question is too generic to be answered. There is no generic mechanism to limit 'resources' on a query or procedure, even the Resource Governor only limits some resources. You describe your problem as is a high volume data manipulation for a long period of time like tens of thousands of inserts updates throughout the database which would indicate toward batching the processing. If the procedure does indeed what you describe then throttling its execution is probably not what you want, because you want to reduce the duration of the transactions (batches) not increase it.
And first and foremost: did you analyzed the procedure as procedure resource consumption as a performance problem, using a methodology like Waits and Queues? Only after you done so and the procedure runs optimally (meaning it consumes the least resources required to perform it's job) can you look into throttling the procedure (and most likely by that time the need to throttle has magically vanished).
You can use the MAXDOP query hint to restrict how many CPUs a given query can utilize. Barring the high-end bells and whistles you mentioned, I don't think there's anything that lets you put single-process-level constraints on memory allocation or disk usage.
If you run multiple DBs on the same SQL Server do they all fight for Procedure Cache? What I am trying to figure out is how does SQL Server determine how long to hold onto Procedure Cache? If other DBs are consuming memory will that impact the procedure cache for a given DB on that same server?
I am finding that on some initial loads of page within our application that it is slow, but once the queries are cachced it is obviously fast. Just not sure how long SQL Server keeps procedure cache and if other DBs will impact that amount of time.
The caching/compiling happens end to end
IIS will unload after 20 mins of not used by default.
.net compilation to CLR
SQL compilation
loading data into memory
This is why the initial calls take some time
Generally stuff stays in cache:
while still in use
no memory pressure
still valid (eg statistics updates will invalidate cached plans)
If you are concerned, add more RAM. Also note that each database will have different load patterns and SQL Server will juggle memory very well. Unless you don't have enough RAM...
From the documentation:
Execution plans remain in the procedure cache as long as there is enough memory to store them. When memory pressure exists, the Database Engine uses a cost-based approach to determine which execution plans to remove from the procedure cache. To make a cost-based decision, the Database Engine increases and decreases a current cost variable for each execution plan according to the following factors.
This link might also be of interest to you: Most Executed Stored Procedure?
My understanding is that some of the DMV's in SQL Server depend on query plans being cached. My questions are these. Are all query plans cached? If not, when is a query plan not cached? For ones that are cached, how long do they stay in the cache?
Thanks very much
Some of the SQL Server DMV's that capture tokens relating directly to the query plan cache, are at the mercy of the memory pressure placed on the query plan cache (due to adhoc queries, other memory usage and high activity, or through recompilation). The query plan cache is subject to plan aging (e.g. a plan with a cost of 10 that has been referenced 5 times has an "age" value of 50):
If the following criteria are met, the plan is removed from memory:
· More memory is required by the system
· The "age" of the plan has reached zero
· The plan isn't currently being referenced by an existing connection
Ref.
Those DMV's not directly relating to the query plan cache are flushed under 'general' memory pressure (cached data pages) or if the sql server service is restarted.
The factors affecting query plan caching have changed slightly since SQL Server 2000. The up-to-date reference for SQL Server 2008 is here: Plan Caching in SQL Server 2008
I just want to add some geek minutia: The Query plan cache leverages the general caching mechanism of SQL Server. These caches use the Clock algorithm for eviction, see Q and A: Clock Hands - what are they for. For query plan caches, the cost of the entry takes into consideration the time, IO and memory needed to create the cache entry.
For ones that are cached, how long do
they stay in the cache?
A valid object stays in cache until the clock hand decrements the cost to 0. See sys.dm_os_memory_cache_clock_hands. There is no absolute time answer to this question, the clock hand could decrement an entry to 0 in a second, in a hour, in a week or in a year. It all depends on the initial cost of the entry (query/schema complexity), on the frequency of reusing the plan, and the clock hands speed (memory pressure).
Cached object may be invalidated though. The various reasons why a Query plan gets invalidated are explained in great detail the white paper linked by Mitch: Plan Caching in SQL Server 2008.