low plan cache memory - sql-server

My question is: Why would an SQL server have a low amount of memory allocated to Plan Cache? And, if a correction is needed, what might be done to correct this?
We have an SQL server with an issue of Compilations per second being high indicating not enough of the execution plans are cached for use (first detected when we ran sp_AskBrent #ExpertMode=1, #Seconds=30 [from brentozar.com/askbrent/]).
We have run the SQL Live Monitor application (https://sqlmonitor.codeplex.com/) on the server and the Plan Cache results show a very low amount of memory (355.27 MB) allocated to caching execution plans and therefore a low Plan Cache Hit Ratio (varying between 5 and 50 percent).
My research shows that memory allocated to Plan Cache is not a configurable amount, but a calculation based on the memory allocated to the SQL instance. So, for this server, which has 48GB total and 40GB allocated to SQL, the calculation of (.75 * 4GB) + (.1 * 36GB) should allocate 6.6GB for Plan Cache. Did I calculate correctly?
Of note, this server has only one production database and that database is 50GB in size. We have Optimize for Ad hoc Workloads set to True and just set Parameterization at the database level to Forced.
Compared to another SQL server (that has 32GB total and 26GB allocated) the Plan Cache numbers look more reasonable (4GB in size and a Hit Ratio of above 80 percent.
Also, running the script below against both SQL servers consistently shows the problem server having a hit percentage in the mid 70% range and the other server showing a hit percentage in the high 90% range.
WITH cte1
AS ( SELECT [dopc].[object_name] ,
[dopc].[instance_name] ,
[dopc].[counter_name] ,
[dopc].[cntr_value] ,
[dopc].[cntr_type] ,
ROW_NUMBER() OVER ( PARTITION BY [dopc].[object_name], [dopc].[instance_name] ORDER BY [dopc].[counter_name] ) AS r_n
FROM [sys].[dm_os_performance_counters] AS dopc
WHERE [dopc].[counter_name] LIKE '%Cache Hit Ratio%'
AND ( [dopc].[object_name] LIKE '%Plan Cache%'
OR [dopc].[object_name] LIKE '%Buffer Cache%'
)
AND [dopc].[instance_name] LIKE '%_Total%'
)
SELECT CONVERT(DECIMAL(16, 2), ( [c].[cntr_value] * 1.0 / [c1].[cntr_value] ) * 100.0) AS [hit_pct]
FROM [cte1] AS c
INNER JOIN [cte1] AS c1
ON c.[object_name] = c1.[object_name]
AND c.[instance_name] = c1.[instance_name]
WHERE [c].[r_n] = 1
AND [c1].[r_n] = 2;

See:
... The maximum size for all caches is a function of the buffer pool size and cannot exceed the maximum server memory...
(https://technet.microsoft.com/en-us/library/ms181055%28v=sql.105%29.aspx)
I think optimize for ad hoc workloads option helps if you mostly run adhoc queries:
sp_configure 'show advanced options',1
GO
reconfigure
GO
sp_configure 'optimize for ad hoc workloads',1
GO
reconfigure
GO
DBCC FREEPROCCACHE
GO
Don't try it first on Prod servers, especialy FREEPROCCACHE.

Related

Snowflake query credit calculation

One of my user has asked if it is possible to calculate the credit burnt for executing a particular query in snowflake. Based on my understanding I think it is not possible because the credit burnt is at the warehouse level and not at query level. But I still thought if someone has a way to calculate the credit per query.
Thanks
I ended up writing a query as below
SELECT query_id
,warehouse_name
,start_time
,end_time
,total_elapsed_sec
,case
when total_elapsed_sec < 60 then 60
else total_elapsed_sec
end as total_elapsed_sec_1
,ROUND(unit_of_credit*total_elapsed_sec_1 / 60/60,2) total_credit
,total_credit*3.00 query_cost --change based on how much you are paying for a credit
FROM (
select query_id
,warehouse_name
,start_time
,end_time
,total_elapsed_time/1000 total_elapsed_sec
,CASE WHEN warehouse_size = 'X-Small' THEN 1
WHEN warehouse_size = 'Small' THEN 2
WHEN warehouse_size = 'Medium' THEN 4
WHEN warehouse_size = 'Large' THEN 8
WHEN warehouse_size = 'X-Large' THEN 16
WHEN warehouse_size = '2X-Large' THEN 32
WHEN warehouse_size = '3X-Large' THEN 64
WHEN warehouse_size = '4X-Large' THEN 128
ELSE 1
END unit_of_credit
from table(information_schema.QUERY_HISTORY_BY_USER
(user_name => 'USERNAME',
END_TIME_RANGE_START => dateadd('hours',-1,current_timestamp()), --you can manipulate this based on your need
END_TIME_RANGE_END => current_timestamp(),RESULT_LIMIT => 10000)));
If you are running sequential queries, like from the web UI using "run all", and nobody else is sharing the warehouse, then execution_time * warehouse_credits_per_time = cost.
If you have a warehouse that is always queued up/running, then the cost is prorate of total_warehouse_cost * sum(query_execution_time) / total_execution_time.
If you processing is in a loop, then any one query is "free", because without it the other code would run. But if you have a loop then you are caring about latency, or reducing your warehouse size, auto-scaling. Thus it's not really free..
So both the first to methods are actually the same thing, which you have to prorate the time.
For our processing most of it in a loop, so we are looking to reduce/manage latency, so we watch 'long running' or 'total time' of parts of our pipeline to find things to improve. As if the SQL is running by itself, the time is the cost, and if the warehouse is running many concurrent requests, then they are "slowed down" by the N concurrency, or they are not (a free lunch), and we discount that last bucket..
The actual credit burnt for a specific query would be little bit difficult to calculate because of various factors , you can reach some what closure with the elapsed time calculation
select sum(TOTAL_ELAPSED_TIME),WAREHOUSE_SIZE from query_history
where QUERY_TEXT='select * from query_history' -- Your Query
and WAREHOUSE_NAME='XXXXXX' -- replace Your WH name
and USER_NAME='XXXXXX'-- Replace your User Name
group by WAREHOUSE_SIZE
With this elapsed time and based on some assumption
Size of the warehouse was consistent during various execution
Warehouse credits also burnt based on the auto-suspend setting ( if execution time is 30 sec you have to pay for 5 minutes, if the auto-suspend is set as 300 sec)
As suggested above post, it will also shared the credit usage if multiple user is sharing the Warehouse at the same time for different query execution
During Query execution whether result is getting fetched from catch or remote storage
If above pointers are known to you calculate the total credit spent specific to the warehouse size, sum that up
Thanks
- Palash Chatterjee

How to check max-sql-memory and cache settings for an already running instance of cockroach db?

I have a cockroachdb instance running in production and would like to know the settings for the --max-sql-memory and --cache specified when the database was started. I am trying to enhance performance by following this production checklist but I am not able infer the setting either on dashboard or sql console.
Where can I check the values of max-sql-memory and cache value ?
Note: I am able to access the cockroachdb admin console and sql tables.
You can find this information in the logs, shortly after node startup:
I190626 10:22:47.714002 1 cli/start.go:1082 CockroachDB CCL v19.1.2 (x86_64-unknown-linux-gnu, built 2019/06/07 17:32:15, go1.11.6)
I190626 10:22:47.815277 1 server/status/recorder.go:610 available memory from cgroups (8.0 EiB) exceeds system memory 31 GiB, using system memory
I190626 10:22:47.815311 1 server/config.go:386 system total memory: 31 GiB
I190626 10:22:47.815411 1 server/config.go:388 server configuration:
max offset 500000000
cache size 7.8 GiB <====
SQL memory pool size 7.8 GiB <====
scan interval 10m0s
scan min idle time 10ms
scan max idle time 1s
event log enabled true
If the logs have been rotated, the value depends on the flags.
The defaults for v19.1 are 128MB, with recommended settings being 0.25 (a quarter of system memory).
The settings are not currently logged periodically or exported through metrics.

SQL query big duration increase after cpu+mainboard upgrade

After upgrading server hardware (cpu+mainboard) I'm having a big increase in query duration for really small and simple querys.
Software: Windows Server 2012 R2 + SQL Server 2014
Storage: Samsung SSD 850 EVO 2TB Disk
Old Hardware: i7-4790k 4.0Ghz 4core cpu + Asus H97M-E mainboard + 32 GB DDR3
New Hardware: i9-7900X 3.60Ghz 10core cpu + Asus Prime X299 mainboard + 32 GB DDR4
Query Sample:
UPDATE CLIE_PRECIOS_COMPRA SET [c_res_tr] = '0.0' WHERE eje ='18' AND mes =8 AND dia =27 AND hor =19 AND unipro='001'
SQL Profiler Result :
Old Hardware - CPU: 0, Reads 4, Writes 0, Duration 123
Old Hardware - CPU: 0, Reads 4, Writes 0, Duration 2852
I've checked network speed of both server to be the same but anyway I'm running the querys directly in the server throught Microsoft SQL Server Management console to avoid applicactions or network issues.
Checked Storage speed too being the same both at reading and writting in old and new hardware.
Also played with paralelism and tried diferent scenarios even disabling paralelism with the same result.
Of course the data is the same having the same copy of SQL database in both hardware.
I've set the duration to be showed in microseconds instead of miliseconds to appreciate better the diference.
The diference in duration for a single query is not really visible to user but the problem is that there are several thousands querys of this type and the time increase is important.
Any hint or thing to investigate would be really appreciated.
Current Execution Plan New Server: https://www.brentozar.com/pastetheplan/?id=HJYDtQQD7
Current Execution Plan Old Server: https://www.brentozar.com/pastetheplan/?id=SynyW4mPQ
Thanks in advance.

Almost empty plan cache

I am experiencing a strange situation - my plan cache is almost empty. I use the following query to see what's inside:
SELECT dec.plan_handle,qs.sql_handle, dec.usecounts, dec.refcounts, dec.objtype
, dec.cacheobjtype, des.dbid, des.text,deq.query_plan
FROM sys.dm_exec_cached_plans AS dec
join sys.dm_exec_query_stats AS qs on dec.plan_handle=qs.plan_handle
CROSS APPLY sys.dm_exec_sql_text(dec.plan_handle) AS des
CROSS APPLY sys.dm_exec_query_plan(dec.plan_handle) AS deq
WHERE cacheobjtype = N'Compiled Plan'
AND objtype IN (N'Adhoc', N'Prepared')
One moment it shows me 82 rows, the next one 50, then 40 then 55 and so on while an hour before I couldn't reach the end of the plan cache issuing the same command. The point is that SQL Server keeps the plan cache very-very small.
The main reason of my investigation is high CPU compared to our baselines without any high loads, under normal during-the day workload - constantly 65-80%
Perfmon counters show low values for Plan Cache Hit Ratio - around 30-50%, high compilations - 400 out of 2000 batch requests per second and high CPU - 73 avg. What could cause this behaviour?
The main purpose of the question is to learn the possible reasons for an empty plan cache.
Memory is OK - min: 0 max: 245000.
I also didn't notice any signs of memory pressure - PLE, lazy writes, free list stalls disk activity were just ok, logs did not tell me a thing.
I came here for possible causes of this so I could proceed with investigation.
EDIT: I have also considered this thread:
SQL Server 2008 plan cache is almost always empty
But none of the recommendations/possible reasons are relevant.
The main purpose of the question is to learn the possible reasons for an empty plan cache.
If it is to learn,the answer from Martin Smith,in the thread you referred will help you
If you want to know in particular,why plan is getting emptied,i recommend using extended events and try below extended event

Paginated searching... does performance degrade heavily after N records?

I just tried the following query on YouTube:
http://www.youtube.com/results?search_query=test&search=tag&page=100
and received the error message:
Sorry, YouTube does not serve more than 1000 results for any query.
(You asked for results starting from 2000.)
I also tried Google search for "test", and although it said there were about 3.44 billion results, I was only able to get to page 82 (or about 820 results).
This leads me to wonder, does performance start to degrade with paginated searches after N records (specifically wondering about with ROW_NUMBER() in SQL Server or similar feature in other DB systems), or are YouTube/Google doing this for other reasons? Granted, it's pretty unlikely that most people would need to go past the first 1000 results for a query, but I would imagine the limitation is specifically put in place for some technical reason.
Then again Stack Overflow lets you page through 47k results: https://stackoverflow.com/questions/tagged/c?page=955&sort=newest&pagesize=50
Yes. High offsets are slow and inefficient.
The only way to find the records at an offset, is to compute all records that came before and then discard them.
(I dont know ROW_NUMBER(), but would be LIMIT in standard SQL. So
SELECT * FROM table LIMIT 1999,20
)
.. in the above example, the first 2000 records have to be fetched first, and then discarded. Generally it can't skip ahead, or use indexes to jump right to the correct location in the data, because normally there would be a 'WHERE' clause filtering the results.
It is possible to cache the results, which is probably what SO does. So it doesn't actually have to compute the large offsets each and every time. (Most of SO's searches are a 'small' set of known tags, so its quite feasible to cache. A arbitrary search query is will have much versions to catch, making it impractical)
(Alternatively it might be using some other implementation that does allow arbitrary offsets)
Other places taking about similar things
http://sphinxsearch.com/docs/current.html#conf-max-matches
Back of the envolope test:
mysql> select gridimage_id from gridimage_search where moderation_status = "geograph" order by imagetaken limit 100999,3;
...
3 rows in set (11.32 sec)
mysql> select gridimage_id from gridimage_search where moderation_status = "geograph" order by imagetaken limit 3;
...
3 rows in set (4.59 sec)
(Arbitrary query choosen so as not to use indexes very well, if indexes can be used the difference is less pronounced and harder to see. But in a production system running lots of queries, 1 or 2ms difference is huge)
Update: (to show a indexed query)
mysql> select gridimage_id from gridimage_search order by imagetaken limit 10;
...
10 rows in set (0.00 sec)
mysql> select gridimage_id from gridimage_search order by imagetaken limit 100000,10;
...
10 rows in set (1.70 sec)
It's a TOP clause designed to limit the amount of physical reads that the database has to perform, which limits the amount of time that the query takes. Imagine you have 82 billion links to stories about "Japan" in your database. What if someone queries "Japan"? Are all 82 billion results really going to be clicked? No. The user needs the top 1000 most relevant results. When the search is generic, like "test", there is no way to determine relevance. In this case, YouTube/Google has to limit the volume returned so other users aren't affected by generic searches. What's faster, returning 1,000 results or 82,000,000,000 results?

Resources