EDIT: I think it's a problem on the subquery on the LINQ-generated query, it get all the records... But I don't know how could I fix it
I have made a simple ASP.NET MVC 2 application that does SELECT queries on a view, I get really poor performance, and while doing a simple benchmark with jMeter (10 conccurents connection) while disabling the cache (I don't want everything to rely on the non customizable/extreme OutputCache)
I see that the SQL Server get overloaded, consuming a LOT of CPU (up to 100%) and all its reserved memory space (512MB)
Here is the action code that cause the problems (manual transactions because it cause DeadLock with the other program that insert new data on the database) :
public ActionResult Index(int page = 0)
{
IronViperEntities db = new IronViperEntities();
db.Connection.Open();
DbTransaction transaction = db.Connection.BeginTransaction(IsolationLevel.ReadUncommitted);
var messages = (from globalView in db.GlobalViews orderby globalView.MessagePostDate descending select globalView).Skip(page*perPage).Take(perPage);
transaction.Commit();
db.Connection.Close();
ViewData["page"] = page;
ViewData["messages"] = messages;
return View();
}
Here is the query executed on the database :
SELECT TOP (100)
[Extent1].[MessageId] AS [MessageId],
[Extent1].[MessageUuid] AS [MessageUuid],
[Extent1].[MessageData] AS [MessageData],
[Extent1].[MessagePostDate] AS [MessagePostDate],
[Extent1].[ChannelName] AS [ChannelName],
[Extent1].[UserName] AS [UserName],
[Extent1].[UserUuid] AS [UserUuid],
[Extent1].[ChannelUuid] AS [ChannelUuid]
FROM ( SELECT [Extent1].[MessageId] AS [MessageId], [Extent1].[MessageUuid] AS [MessageUuid], [Extent1].[MessageData] AS [MessageData], [Extent1].[MessagePostDate] AS [MessagePostDate], [Extent1].[ChannelName] AS [ChannelName], [Extent1].[UserName] AS [UserName], [Extent1].[UserUuid] AS [UserUuid], [Extent1].[ChannelUuid] AS [ChannelUuid], row_number() OVER (ORDER BY [Extent1].[MessagePostDate] DESC) AS [row_number]
FROM (SELECT
[GlobalView].[MessageId] AS [MessageId],
[GlobalView].[MessageUuid] AS [MessageUuid],
[GlobalView].[MessageData] AS [MessageData],
[GlobalView].[MessagePostDate] AS [MessagePostDate],
[GlobalView].[ChannelName] AS [ChannelName],
[GlobalView].[UserName] AS [UserName],
[GlobalView].[UserUuid] AS [UserUuid],
[GlobalView].[ChannelUuid] AS [ChannelUuid]
FROM [dbo].[GlobalView] AS [GlobalView]) AS [Extent1]
) AS [Extent1]
WHERE [Extent1].[row_number] > 0
ORDER BY [Extent1].[MessagePostDate] DESC
View Code :
SELECT dbo.Messages.Id AS MessageId, dbo.Messages.Uuid AS MessageUuid, dbo.Messages.Data AS MessageData, dbo.Messages.PostDate AS MessagePostDate,
dbo.Channels.Name AS ChannelName, dbo.Users.Name AS UserName, dbo.Users.Uuid AS UserUuid, dbo.Channels.Uuid AS ChannelUuid
FROM dbo.Messages INNER JOIN
dbo.Users ON dbo.Messages.UserId = dbo.Users.Id INNER JOIN
dbo.Channels ON dbo.Messages.ChannelId = dbo.Channels.Id
I don't think the server hardware is a problem, I can run equivalent Rails/Grails application without any performance issue. (Dual Core, 3Gb of RAM)
A select count(*) on GlobalView returns ~270.000 lines, indexes are daily rebuilt and a explain show it uses all the clustered indexes.
I get an HTTP average response time of 8000ms, the SQL Server Management Studio shows an average CPU time for this SQL query of 866ms and an average logical IO of 7,592.03.
Database file size if ~180MB
I am using Windows Server 2008 R2 Enterprise Edition, ASP.NET MVC 2 with IIS 7.5 and SQL Server 2008 R2 Express Edition with Advanced Services. They are the only things running on this server.
What can I do ?
Thank you
I guess you got the query from SQL Server Profiler. Save the result, and pass it into the Database Engine Tuning Advisor. That might help you create additional indexes and statistics.
Just out of curiosity: wouldn't appending a .ToList() to the end of the var messages = ... line help?
I've found the probleme,
I replaced "orderby globalView.MessagePostDate descending" by "orderby globalView.MessageId descending", because there isn't any index on MessagePostDate, and that is muuuuch better !
Thank you
Related
I tested this query using clickhouse-benchmark
select pageUrl ,
pageTitle ,
sum(visits) as visits ,
count(DISTINCT visitorId) as visitors ,
any(featuredImage) as featuredImage ,
any(pageType) as pageType ,ssid
FROM PageViews FINAL where (createdAtDay >= toDate('2022-06-01') and createdAtDay <= toDate('2022-10-01')) GROUP BY pageUrl,pageTitle,ssid ORDER BY visits DESC
If executing the command from terminal on the server that's hosting clickhouse instance I get pretty fast response :
but when I try to execute same query remotely (which is the usual scenario, I'm not hosting my api in same server as clickhouse)
I get slow response :
I tried to get logs using this query :
SELECT
event_time, query_duration_ms / 1000 AS secs,
formatReadableSize(memory_usage) AS memory,
Settings['max_threads'] AS threads,
query
FROM system.query_log
WHERE (user = 'default') AND (type = 'QueryFinish') and (query like '%select pageUrl ...%')
ORDER BY event_time DESC
and it seems even in database logs the execution time is very big :
last 5 are the remote ones, first 5 are from the server, the time diff is very big.
is this normal? is there any type of config that I must setup ?
I have two BI reporting servers (one is Development, the other Production) each with their own own Azure SQL instance, I am using Pentaho to automate the ETL process and part of my process is to execute a query joining two tables of similar data in order to join a user's login row with their logout row.
with login_cte as
(
select
os.[k_actions]
,[k_hosts]
,[k_logger_names]
,[k_principals]
,[k_sites]
,[k_roles]
,[k_date] as k_date_logged_in
,[k_time] as k_time_logged_in
,[k_date_ct] as k_date_ct_logged_in
,[k_time_ct] as k_time_ct_logged_in
,[key] as key_logged_in
,[type]
,[id]
,[sort]
,[level]
,[timestamp_utc] as timestamp_utc_logged_in
,[timestamp_ct] as timestamp_ct_logged_in
,[service]
,[version]
,[message] as message_logged_in
,[thread_name]
,[db_name]
,[customer_id]
from other_stage os
left join dim_actions da on os.k_actions = da.k_actions
where [action_name] = 'SET_ROLE_DEVICE_USER'
order by [key] asc
offset 0 rows
),
logout_cte as
(
select
[k_actions]
,[k_hosts] as k_hosts_out
,[k_logger_names] as k_logger_names_out
,[k_principals] as k_principals_out
,[k_sites] as k_sites_out
,null as k_roles_out
,[k_date] as k_date_logged_out
,[k_time] as k_time_logged_out
,[k_date_ct] as k_date_ct_logged_out
,[k_time_ct] as k_time_ct_logged_out
,[key] as key_logged_out
,[type] as type_out
,[id] as id_out
,[sort] as sort_out
,[level] as level_out
,[timestamp_utc] as timestamp_utc_logged_out
,[timestamp_ct] as timestamp_ct_logged_out
,[service] as service_out
,[version] as version_out
,[message] as message_logged_out
,[thread_name] as thread_name_out
,[db_name]
,[customer_id]
from logout_stage
order by [key] asc
offset 0 rows
)
select
li.k_actions
,li.k_hosts
,li.k_logger_names
,li.k_principals
,li.k_roles
,li.k_sites
,li.k_date_logged_in
,li.k_time_logged_in
,lo.k_date_logged_out
,lo.k_time_logged_out
,li.k_date_ct_logged_in
,li.k_time_ct_logged_in
,lo.k_date_ct_logged_out
,lo.k_time_ct_logged_out
,li.key_logged_in
,lo.key_logged_out
,li.type
,li.id
,li.sort
,li.level
,li.timestamp_utc_logged_in
,li.timestamp_ct_logged_in
,lo.timestamp_utc_logged_out
,lo.timestamp_ct_logged_out
,li.service
,li.version
,li.message_logged_in
,lo.message_logged_out
,li.thread_name
,li.db_name
,li.customer_id
,datediff(second, li.timestamp_utc_logged_in, lo.timestamp_utc_logged_out) as login_duration
,row_number() over (partition by li.key_logged_in order by lo.key_logged_out asc) as rn
from login_cte li left join logout_cte lo
on li.k_principals = lo.k_principals_out
and li.db_name = lo.db_name
and lo.key_logged_out > li.key_logged_in
and lo.timestamp_utc_logged_out > li.timestamp_utc_logged_in
order by lo.key_logged_out asc
The issue I'm having is that this query runs just fine by itself in my production instance but when run on the development instance, which has fewer rows in each table (~70k vs ~124k), the query fails with this error.
SQLServerException: The database 'tempdb' has reached its size quota. Partition or delete data, drop indexes, or consult the documentation for possible resolutions.
Doing some digging into the tempdb size on each I ran this query:
SELECT SUM(unallocated_extent_page_count) AS [free pages],
(SUM(unallocated_extent_page_count)*1.0/128) AS [free space in MB]
FROM tempdb.sys.dm_db_file_space_usage;
With these results:
Production:
free pages
free space in MB
923,792
7,217.125
Development:
free pages
free space in MB
8,386,864
65,522.375
while the query is running I can run file space usage query again and watch the free space slow go done to zero then the query fails. Meanwhile the same query on Production shows the free space go down but never hits zero and will succeed.
I am fairly new to Azure SQL and have tried searching for an answer but can only find answers on how to find a query that is using up the tempdb (of which I already know...) but not any on what would cause the same query to fail on one and not the other. I do not have access to the Azure console but can connect to the DB with a power user through either DBeaver or Microsoft SQL Server Management Studio.
Edit 12-14-21:
Paste of the query execution plan
Hey guys the code below is taking a really long time. I've been looking at it for quite a while. Is there anything that stands out as the obvious cause of delay?
[SQLSRV-3-JB] is a linked server BTW
Select count(cast (Unique_ID as bigint)) as [Count],
T.[Region_Code],
T.[Region_Name],
U.[Region_Code],
Y.[Examination_Year],
case when [Subject] = 'MUSIC THEORY' THEN 'Theory'
else 'Practical'
end
from [SQLSRV-3-JB].[X].[dbo].[Exam_and_Candidate_Details] Y
left join [SQLSRV-3-JB].[X].[dbo].[UK_Exam_Centre_Info] T
on Y.Centre_Code = T.Centre_Code
left join [SQLSRV-3-JB].[X].[dbo].[UK_Exam_Centres] U
on Y.Centre_Code = U.Centre_Code
where Y.[Examination_Year] between 2010 and 2016
group by Y.[Examination_Year],
T.[Region_Code],
T.[Region_Name],
U.[Region_Code],
case when [Subject] = 'MUSIC THEORY' THEN 'Theory'
else 'Practical'
end
Yes, there is one very obvious problem - the remote server. When you use a linked server like this, SQL Server has a disturbing habit of pulling all of the data in the remote tables to the local server before performing JOINs or filtration.
The correct way to handle this is to make this query a view on the remote server, and query the view with your WHERE clause on your end. So, your remote code would look like this:
-- Remote Server
USE X
GO
CREATE VIEW dbo.ExamCenterInfo
AS
Select count(cast (Unique_ID as bigint)) as [Count],
T.[Region_Code],
T.[Region_Name],
U.[Region_Code],
Y.[Examination_Year],
case when [Subject] = 'MUSIC THEORY' THEN 'Theory'
else 'Practical'
end
from dbo.[Exam_and_Candidate_Details] Y
left join dbo.[UK_Exam_Centre_Info] T
on Y.Centre_Code = T.Centre_Code
left join dbo.[UK_Exam_Centres] U
on Y.Centre_Code = U.Centre_Code
group by Y.[Examination_Year],
T.[Region_Code],
T.[Region_Name],
U.[Region_Code],
case when [Subject] = 'MUSIC THEORY' THEN 'Theory'
else 'Practical'
end
Then your local code:
-- Local Server
SELECT *
FROM [SQLSRV-3-JB].[X].[dbo].ExamCenterInfo ECI
where ECI.[Examination_Year] between 2010 and 2016
NOTE: There is some possibility that this will pull all of the view output before filtering by year. If this is still a problem, you will have to create a more complex mechanism to execute the view with filtration on the remote server.
I've a server with 20 clients. If I don't use a client for 1 day, the result of query arrives after 2:30 minutes (over 1000 rows). After the execution of 5/6 queries, the result arrives after few seconds.
I think it's a scheduling problem of SQL Server. How can I resolve?
Thanks
UPDATE
this is the query
Select * from [WWALMDB].[dbo].[v_AlarmConsolidated]
Where Critico = 1 AND ApprovatoQA = 0
AND InAttesaDiRiconoscimento Like '%param1%'
AND (Tipo Like '%param2%') AND Area Like '%param3%'
AND Nome Like '%%param4%%' AND Descrizione Like '%%param5%%'
AND (([Dataora Scatto] >= CONVERT(DATETIME,'param6',105))
AND ([Dataora Scatto] <= CONVERT(DATETIME,'param7',105))
OR( ([Dataora Rientro] >= CONVERT(DATETIME,'param6',105))
AND ([Dataora Rientro] <= CONVERT(DATETIME,'param7',105)) )
OR( ([Dataora PresoInCarico] >= CONVERT(DATETIME,'param6',105))
AND ([Dataora PresoInCarico] <= CONVERT(DATETIME,'param7',105)) ))
ORDER BY AlarmID DESC
What you described is a normal SQL Server caching behavior.
More information about MSSQL Server caching mechanisms you can find here
It's impossible to tell you anything else without knowing the background (example sql queries, health status of your sql server machine and actual db data structure)
UPDATE:
Your query is probably so badly optimized, that is actually a miracle it's only 2:30
SELECT *
LIKE '%%*%%'
ORDER BY
I'm trying to build an XE in order to find out which of our internal apps (that don't have app names and thus show up as .Net SQLClient Data Provider) are hitting particular servers. Ideally, I'd like to get the name of the Client and Database , but not sure if I can do that in one XE.
I figured for ease of use, I'd use a histogram/asynchronous_bucketizer, and save counts of what's trying to hit and how often. However, I can't seem to get it work on 2012, much less 2008. If I use sqlserver.existing_connection it works, but only gives me the count when it connects. I want to get counts during the day and see how often it occurs from each server, so I tried preconnect_completed. Is this the right event?
Also, and part of the reason I'm using XE, is that those servers can get thousands of calls a minute.
Here's what I've come up with thus far, which works but only gives me current SSMS connections that match - obviously, I'll change that to the .Net SQLClient Data Provider.
CREATE EVENT SESSION UnknownAppHosts
ON SERVER
ADD EVENT sqlserver.existing_connection(
ACTION(sqlserver.client_hostname)
WHERE ([sqlserver].[client_app_name] LIKE 'Microsoft SQL Server Management%')
)
ADD TARGET package0.histogram
( SET slots = 50,
filtering_event_name='sqlserver.existing_connection',
source_type=1,
source='sqlserver.client_hostname'
)
WITH(MAX_DISPATCH_LATENCY =1SECONDS);
GO
Aha! It's login, not preconnect_starting or preconnect_completed.
CREATE EVENT SESSION UnknownAppHosts
ON SERVER
ADD EVENT sqlserver.login(
ACTION(sqlserver.client_hostname)
WHERE ([sqlserver].[client_app_name] LIKE 'Microsoft SQL Server Management%')
)
ADD TARGET package0.histogram
( SET slots = 50,
filtering_event_name='sqlserver.login',
source_type=1,
source='sqlserver.client_hostname'
)
WITH(MAX_DISPATCH_LATENCY =1SECONDS);
GO
Then to query it, some awesome code I made horrid:
-- Parse the session data to determine the databases being used.
SELECT slot.value('./#count', 'int') AS [Count] ,
slot.query('./value').value('.', 'varchar(20)')
FROM
(
SELECT CAST(target_data AS XML) AS target_data
FROM sys.dm_xe_session_targets AS t
INNER JOIN sys.dm_xe_sessions AS s
ON t.event_session_address = s.address
WHERE s.name = 'UnknownAppHosts'
AND t.target_name = 'Histogram') AS tgt(target_data)
CROSS APPLY target_data.nodes('/HistogramTarget/Slot') AS bucket(slot)
ORDER BY slot.value('./#count', 'int') DESC