Parameterized raw sql query much slower than query with actual values - sql-server

I am trying to execute a raw sql query with connections[].cursor() in a Django app that connects to SQL server. The query executes much faster (<1s) when I provide the actual vlaues in the query string.
from django.db import connections
with connections['default'].cursor() as cursor:
cursor.execute("""
select c.column1 as c1
, ve.column2 as c2
from view_example c
left join view_slow_view ve on c.k1 = ve.k2
where c.column_condition = value_1 and c.column_cd_2 = value2
""")
result = dictfetchall(cursor)
But when I provide the values as params in the cursor.execute() method, the query becomes much slower (2 minutes).
from django.db import connections
with connections['default'].cursor() as cursor:
cursor.execute("""
select c.column1 as c1
, ve.column2 as c2
from view_example c
left join view_slow_view ve on c.k1 = ve.k2
where c.column_condition = %s and c.column_condition_2 = %s
""", [value_1, value_2])
contracts_dict_lst = dictfetchall(cursor)
I should also mention that the query is actually slow when executed on SSMS ONLY IF a condition is NOT provided:
where c.column_condition = value_1 and c.column_cd_2 = value2
It is as if when Django sends the query, it is executed without the parameters (hence the long response time) and then the parameters are provided so the result is filtered.
The values in question are provided by the user, so they change and have to be passed as params and not directly in the query to avoid sql injection.
The query is also much more complex than the example given above and doesn't map cleanly to a model so I have to use connection[].cursor()

This is probably parameter sniffing issue. If that's the case, there are couple of solutions. The easiest solution is using query hint.
Option 1:
from django.db import connections
with connections['default'].cursor() as cursor:
cursor.execute("""
select c.column1 as c1
, ve.column2 as c2
from view_example c
left join view_slow_view ve on c.k1 = ve.k2
where c.column_condition = %s and c.column_condition_2 = %s
OPTION(RECOMPILE) -- add this line to your query
""", [value_1, value_2])
contracts_dict_lst = dictfetchall(cursor)
Option 2:
from django.db import connections
with connections['default'].cursor() as cursor:
cursor.execute("""
declare v1 varchar(100) = %s -- declare variable and use them
declare v2 varchar(100) = %s
select c.column1 as c1
, ve.column2 as c2
from view_example c
left join view_slow_view ve on c.k1 = ve.k2
where c.column_condition = v1 and c.column_condition_2 = v2
""", [value_1, value_2])
contracts_dict_lst = dictfetchall(cursor)
This is a good link for more reading.

Related

Using order by clause with skip and take fails

I am using EF Core for my .NET Core 2.1 project to interact with Azure SQL Server.
I have query which fetches around 1.5L rows by joining multiple tables. It's taking a long time to return the result and it is getting timed out (more than 30 seconds). So I am going with pagination using skip and take.
So my final call looks like
query.Skip((pageNumber - 1) * pageSize).Take(pageSize).ToList();
This call works absolutely fine and give the expected results.
But when I add order by clause to this query:
query.OrderBy(sortField)
.Skip((pageNumber - 1) * pageSize).Take(pageSize).ToList()
Below is the generated SQL
SELECT [p].[Id], [p].[CreatedById], [p].[CreatedOn], [p].[CurrencyId], [p].[CustomerId], [p].[DataMigrationLogId], [p].[FollowUp], [p].[IsActive], [p].[ProjectName], [p].[PlantCode], [p].[ShipToDistanceFromPlant], [p].[StatusId],
[p].[UpdatedById], [p].[UpdatedOn], [p.DataMigrationLog].[Id], [p.DataMigrationLog].[CreatedOn], [p.DataMigrationLog].[GeneratedOn], [p.DataMigrationLog].[HasEdsBid], [p.DataMigrationLog].[HasMbiBid], [p.DataMigrationLog].[Log],
[p.DataMigrationLog].[RequestXml], [p.DataMigrationLog].[Status], [p.DataMigrationLog].[Xq1ProjectId], [p.UpdatedBy].[Id], [p.UpdatedBy].[CellPhone], [p.UpdatedBy].[CreatedOn], [p.UpdatedBy].[Discriminator], [p.UpdatedBy].[Email],
[p.UpdatedBy].[FirstName], [p.UpdatedBy].[HasAccessToDoors], [p.UpdatedBy].[HasAccessToWindows], [p.UpdatedBy].[IsActive], [p.UpdatedBy].[LastName], [p.UpdatedBy].[Prefix], [p.UpdatedBy].[RecentProjectId], [p.UpdatedBy].[UpdatedOn],
[p.UpdatedBy].[WorkPhone], [p.UpdatedBy].[XQ1LoginName], [p.UpdatedBy].[IsSuperAdmin], [p.UpdatedBy].[HasAccessToRestrictedReports], [p.UpdatedBy].[HasEdsProgramAndDealerPriceAccess], [p.UpdatedBy].[IsIss],
[p.UpdatedBy].[ProjectVisibilityId], [p.UpdatedBy].[CustomerId], [p.UpdatedBy].[IsApiUser], [p.CreatedBy].[Id], [p.CreatedBy].[CellPhone], [p.CreatedBy].[CreatedOn], [p.CreatedBy].[Discriminator], [p.CreatedBy].[Email],
[p.CreatedBy].[FirstName], [p.CreatedBy].[HasAccessToDoors], [p.CreatedBy].[HasAccessToWindows], [p.CreatedBy].[IsActive], [p.CreatedBy].[LastName], [p.CreatedBy].[Prefix], [p.CreatedBy].[RecentProjectId], [p.CreatedBy].[UpdatedOn],
[p.CreatedBy].[WorkPhone], [p.CreatedBy].[XQ1LoginName], [p.CreatedBy].[IsSuperAdmin], [p.CreatedBy].[HasAccessToRestrictedReports], [p.CreatedBy].[HasEdsProgramAndDealerPriceAccess], [p.CreatedBy].[IsIss],
[p.CreatedBy].[ProjectVisibilityId], [p.CreatedBy].[CustomerId], [p.CreatedBy].[IsApiUser], [p.Status].[Id], [p.Status].[Code], [p.Status].[Name], [p.Customer].[Id], [p.Customer].[AxCustomerNumber], [p.Customer].[CreatedById],
[p.Customer].[CreatedOn], [p.Customer].[CreditTermId], [p.Customer].[CurrencyId], [p.Customer].[CustomerTypeId], [p.Customer].[DefaultPricing], [p.Customer].[FabricSystemFreight], [p.Customer].[IsActive], [p.Customer].[IsSpecialFreight],
[p.Customer].[LocalityRepId], [p.Customer].[MinFreightCharge], [p.Customer].[CompanyName], [p.Customer].[Notes], [p.Customer].[PrimaryBusinessId], [p.Customer].[ProgramAccountId], [p.Customer].[Prospect],
[p.Customer].[ShippingDollarsThreshold], [p.Customer].[ShippingMilesThreshold], [p.Customer].[SpecialNote], [p.Customer].[SpecialPlantInstruction], [p.Customer].[UpdatedById], [p.Customer].[UpdatedOn],
[p.Customer].[Xq1ProgGroupName], [p.Customer].[Xq1SrNo], [p.Customer].[XqCustomerNumber], [p.Customer.ProgramAccount].[Id], [p.Customer.ProgramAccount].[AxCustomerNumber], [p.Customer.ProgramAccount].[CreatedById],
[p.Customer.ProgramAccount].[CreatedOn], [p.Customer.ProgramAccount].[CreditTermId], [p.Customer.ProgramAccount].[CurrencyId], [p.Customer.ProgramAccount].[CustomerTypeId], [p.Customer.ProgramAccount].[DefaultPricing],
[p.Customer.ProgramAccount].[FabricSystemFreight], [p.Customer.ProgramAccount].[IsActive], [p.Customer.ProgramAccount].[IsSpecialFreight], [p.Customer.ProgramAccount].[LocalityRepId], [p.Customer.ProgramAccount].[MinFreightCharge],
[p.Customer.ProgramAccount].[CompanyName], [p.Customer.ProgramAccount].[Notes], [p.Customer.ProgramAccount].[PrimaryBusinessId], [p.Customer.ProgramAccount].[ProgramAccountId], [p.Customer.ProgramAccount].[Prospect],
[p.Customer.ProgramAccount].[ShippingDollarsThreshold], [p.Customer.ProgramAccount].[ShippingMilesThreshold], [p.Customer.ProgramAccount].[SpecialNote], [p.Customer.ProgramAccount].[SpecialPlantInstruction],
[p.Customer.ProgramAccount].[UpdatedById], [p.Customer.ProgramAccount].[UpdatedOn], [p.Customer.ProgramAccount].[Xq1ProgGroupName], [p.Customer.ProgramAccount].[Xq1SrNo], [p.Customer.ProgramAccount].[XqCustomerNumber],
[p.Customer.CustomerType].[Id], [p.Customer.CustomerType].[Code], [p.Customer.CustomerType].[Name]
FROM [Projects] AS [p]
LEFT JOIN [DataMigrationLogs] AS [p.DataMigrationLog] ON [p].[DataMigrationLogId] = [p.DataMigrationLog].[Id]
INNER JOIN [Security].[XQUsers] AS [p.UpdatedBy] ON [p].[UpdatedById] = [p.UpdatedBy].[Id]
INNER JOIN [Security].[XQUsers] AS [p.CreatedBy] ON [p].[CreatedById] = [p.CreatedBy].[Id]
INNER JOIN [ProjectStatuses] AS [p.Status] ON [p].[StatusId] = [p.Status].[Id]
LEFT JOIN [Customers] AS [p.Customer] ON [p].[CustomerId] = [p.Customer].[Id]
LEFT JOIN (
SELECT [p.Customer.LocalityRep].*
FROM [Security].[XQUsers] AS [p.Customer.LocalityRep]
WHERE [p.Customer.LocalityRep].[Discriminator] = N'SALES_SP'
) AS [t] ON [p.Customer].[LocalityRepId] = [t].[Id]
LEFT JOIN [Customers] AS [p.Customer.ProgramAccount] ON [p.Customer].[ProgramAccountId] = [p.Customer.ProgramAccount].[Id]
LEFT JOIN (
SELECT [p.Customer.ProgramAccount.LocalityRep].*
FROM [Security].[XQUsers] AS [p.Customer.ProgramAccount.LocalityRep]
WHERE [p.Customer.ProgramAccount.LocalityRep].[Discriminator] = N'SALES_SP'
) AS [t0] ON [p.Customer.ProgramAccount].[LocalityRepId] = [t0].[Id]
LEFT JOIN [CustomerTypes] AS [p.Customer.CustomerType] ON [p.Customer].[CustomerTypeId] = [p.Customer.CustomerType].[Id]
WHERE ([p.UpdatedBy].[Discriminator] IN (N'INT_SYSADMMIN', N'XqInternalUser', N'EXT', N'SALES_SP', N'XqUser') AND [p.CreatedBy].[Discriminator] IN (N'INT_SYSADMMIN', N'XqInternalUser', N'EXT', N'SALES_SP', N'XqUser')) AND (EXISTS (
SELECT 1
FROM [Security].[BusinessUserRegions] AS [r]
WHERE [r].[RegionId] IN (CAST(6 AS bigint), CAST(8 AS bigint), CAST(9 AS bigint)) AND ([t].[Id] = [r].[BusinessUserId])) OR ([p.Customer].[ProgramAccountId] IS NOT NULL AND EXISTS (
SELECT 1
FROM [Security].[BusinessUserRegions] AS [r0]
WHERE [r0].[RegionId] IN (CAST(6 AS bigint), CAST(8 AS bigint), CAST(9 AS bigint)) AND ([t0].[Id] = [r0].[BusinessUserId]))))
ORDER BY [p].[CreatedOn]
It takes more than 30 seconds to execute and Azure SQL Server has query timeout of maximum 30 second, so it results in exception
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
From my layman's understanding about SQL I think SQL us fetching the entire dataset for sorting and then trying to do pagination and hence it's taking more than 30 seconds. Please correct me if I am wrong.
I need to sort the result in query in self otherwise it will not provide the desired result set.
Is their any way to deal with this situation.
Any help is highly appreciated.

SQL Server : nested looping over two Selects

I have the following two queries that produce the results I need. Now the final output I truly need I would usually use python for after the results are returned, but unfortunately only SQL can be used.
Query A:
SELECT *
FROM openquery(PROD, 'SELECT `status`, computer_name, device_type
FROM assets
WHERE (device_type="SERVER")
AND (status="ACTIVE")')
Query B:
SELECT *
FROM openquery(AppMap, 'SELECT `t1`.`uaid` AS `uaid`, `t3`.`computer_name`,
FROM ((`applications` `t1`
JOIN `app_infrastructure` `t2` ON (((`t1`.`uaid` = `t2`.`uaid`))))
JOIN `infrastructure` `t3` ON ((`t2`.`infrastructure_id` = `t3`.`infrastructure_id`)));')
How I would want to process the results:
if a computer_name is in both A and B:
final_row = ['computer_name', 1]
elseif a computer_name is in A but not B:
final_row = ['computer_name', 0]
elseif a computer_name is in B but not A:
final_row = ['computer_name', 2]
So my final query results need to look like those rows, does that make sense?
In a stored procedure, use both queries to load table variables.
Then do a FULL OUTER JOIN query, joining the two table variables on computer_name, and use a CASE expression to get your final_row value for each computer name.

using case statement in a join with linq

I'm trying to convert this SQL query into a linq query but not having much luck.
SELECT DISTINCT gen.ID
, gen.Name
, Ssec.System
FROM dbo.Generic Gen
JOIN dbo.SystemsSelected SSel
ON Gen.RecordID = SSel.RecordID
JOIN dbo.Security SSec
ON (
SSel.SystemA = CASE WHEN Ssec.System = '1stSystem' THEN 1 ELSE NULL END
OR SSel.SystemB = CASE WHEN Ssec.System = '2ndSystem' THEN 1 ELSE NULL END
)
and SSec.Username = 'myUserName'
I've had a look at the following posts but cannot apply the examples to my code:
Link join with case condition
Linq query with table joins
Any help would be appreciated!
Thanks
I cannot answer my own question directly but I found a workaround.
I used linq to instead call a Stored Procedure and that worked fine.
To do this:
1) Save code as a Stored Procedure that accepts a parameter named UserName
2) add the SP into Entity Framework Model
3) call the SP using:
public JsonResult GetNewTeams(string userUserName)
{
using (YourDBNameEntities db = new YourDBNameEntities ())
{
var ret = db.SP_YourStoredProcedureName(userUserName).ToList();
return Json(ret, JsonRequestBehavior.AllowGet);
}
}
Thats it, done, no more scary linq!! :)
sauce: Using stored procedures in LINQ

SQL queries do not currently support returning aliases

I have this raw sql that I am trying to run on grails
final session = sessionFactory.currentSession;
final String query = 'select count(A.*) from Artifact as A LEFT JOIN classification as C on (A.id=C.artifact_id) where C.id IS NULL OR C.active=0';
final sqlQuery = session.createSQLQuery(query);
def totalunclass = sqlQuery.with {
addEntity(Artifact);
}
But I am getting the error Message: SQL queries do not currently support returning aliases
I tried another method but failed there too achieve the result click here
Try changing your query line to look like this
final String query = 'select count(*) from Artifact A LEFT JOIN classification C on (A.id=C.artifact_id) where C.id IS NULL OR C.active=0';
Try this and remove aliases:
final session = sessionFactory.currentSession;
final String query = 'select count(Artifact.*) from Artifact LEFT JOIN classification on (Artifact.id=classification.artifact_id) where classification.id IS NULL OR classification.active=0';
final sqlQuery = session.createSQLQuery(query);
def totalunclass = sqlQuery.with {
addEntity(Artifact);
}

Translating a QueryExpression into SQL: what's a Natural join?

I am trying to translate a QueryExpression that is in some existing code into a T-SQL select statement.
I've run across the following statement and I'm having trouble understanding what they mean by a Natural Join:
linkEntity1.JoinOperator = JoinOperator.Natural;
Would this be equivalent to an Inner Join in T-SQL? Googling has not been much help.
Here's the rest of the QueryExpression Code:
QueryExpression query = new QueryExpression();
query.EntityName = "showinfo";
ColumnSet columns = new ColumnSet();
columns.Attributes = new String[] { "company" };
query.ColumnSet = columns;
query.Criteria = new FilterExpression();
query.Criteria.FilterOperator = LogicalOperator.And;
ConditionExpression condition1 = new ConditionExpression();
condition1.AttributeName = "company";
condition1.Operator = ConditionOperator.NotNull;
query.Criteria.Conditions = new ConditionExpression[] { condition1 };
LinkEntity linkEntity1 = new LinkEntity();
linkEntity1.JoinOperator = JoinOperator.Natural;
linkEntity1.LinkFromEntityName = "show";
linkEntity1.LinkFromAttributeName = "showid";
linkEntity1.LinkToEntityName = "showintegration";
linkEntity1.LinkToAttributeName = "showcode";
linkEntity1.LinkCriteria = new FilterExpression();
linkEntity1.LinkCriteria.FilterOperator = LogicalOperator.And;
ConditionExpression condition2 = new ConditionExpression();
condition2.AttributeName = "showend";
condition2.Operator = ConditionOperator.Null;
linkEntity1.LinkCriteria.Conditions = new ConditionExpression[] { condition2 };
query.LinkEntities = new LinkEntity[] { linkEntity1 };
There is no equivalent in SQL Server of a natural join where table intersect is based on column names by the RDBMS.
I'm glad of that because it is at best ambiguous and at worst dangerous. JOINs should be explicit. Examples why:
having a InsertedBy column in both tables (quite common): should we have to prefix with the table name to remove ambiguity?
future DDL that add columns that change JOIN semantics
See
Natural join in SQL Server
SQL Server - lack of NATURAL JOIN / x JOIN y USING(field)
Edit:
It looks like natural join means "don't repeat the column in the output" (like USING in MySQL would do) according to the JoinOperator Enumeration.
If I understand this (debatable!) it's misleading. Especially when I read the "LeftOuter" narrative..
A natural join compares all columns in the two tables that have the same column names. It's equivalent to an inner join with the matching columns explicitly listed.
Yes - the natural join is inner join - so you can write:
select * from tab1, tab2 where tab1.col1 = tab2.col1
as
select * from tab1 inner join tab2 on tab1.col1 = tab2.col1

Resources