I am using EF Core for my .NET Core 2.1 project to interact with Azure SQL Server.
I have query which fetches around 1.5L rows by joining multiple tables. It's taking a long time to return the result and it is getting timed out (more than 30 seconds). So I am going with pagination using skip and take.
So my final call looks like
query.Skip((pageNumber - 1) * pageSize).Take(pageSize).ToList();
This call works absolutely fine and give the expected results.
But when I add order by clause to this query:
query.OrderBy(sortField)
.Skip((pageNumber - 1) * pageSize).Take(pageSize).ToList()
Below is the generated SQL
SELECT [p].[Id], [p].[CreatedById], [p].[CreatedOn], [p].[CurrencyId], [p].[CustomerId], [p].[DataMigrationLogId], [p].[FollowUp], [p].[IsActive], [p].[ProjectName], [p].[PlantCode], [p].[ShipToDistanceFromPlant], [p].[StatusId],
[p].[UpdatedById], [p].[UpdatedOn], [p.DataMigrationLog].[Id], [p.DataMigrationLog].[CreatedOn], [p.DataMigrationLog].[GeneratedOn], [p.DataMigrationLog].[HasEdsBid], [p.DataMigrationLog].[HasMbiBid], [p.DataMigrationLog].[Log],
[p.DataMigrationLog].[RequestXml], [p.DataMigrationLog].[Status], [p.DataMigrationLog].[Xq1ProjectId], [p.UpdatedBy].[Id], [p.UpdatedBy].[CellPhone], [p.UpdatedBy].[CreatedOn], [p.UpdatedBy].[Discriminator], [p.UpdatedBy].[Email],
[p.UpdatedBy].[FirstName], [p.UpdatedBy].[HasAccessToDoors], [p.UpdatedBy].[HasAccessToWindows], [p.UpdatedBy].[IsActive], [p.UpdatedBy].[LastName], [p.UpdatedBy].[Prefix], [p.UpdatedBy].[RecentProjectId], [p.UpdatedBy].[UpdatedOn],
[p.UpdatedBy].[WorkPhone], [p.UpdatedBy].[XQ1LoginName], [p.UpdatedBy].[IsSuperAdmin], [p.UpdatedBy].[HasAccessToRestrictedReports], [p.UpdatedBy].[HasEdsProgramAndDealerPriceAccess], [p.UpdatedBy].[IsIss],
[p.UpdatedBy].[ProjectVisibilityId], [p.UpdatedBy].[CustomerId], [p.UpdatedBy].[IsApiUser], [p.CreatedBy].[Id], [p.CreatedBy].[CellPhone], [p.CreatedBy].[CreatedOn], [p.CreatedBy].[Discriminator], [p.CreatedBy].[Email],
[p.CreatedBy].[FirstName], [p.CreatedBy].[HasAccessToDoors], [p.CreatedBy].[HasAccessToWindows], [p.CreatedBy].[IsActive], [p.CreatedBy].[LastName], [p.CreatedBy].[Prefix], [p.CreatedBy].[RecentProjectId], [p.CreatedBy].[UpdatedOn],
[p.CreatedBy].[WorkPhone], [p.CreatedBy].[XQ1LoginName], [p.CreatedBy].[IsSuperAdmin], [p.CreatedBy].[HasAccessToRestrictedReports], [p.CreatedBy].[HasEdsProgramAndDealerPriceAccess], [p.CreatedBy].[IsIss],
[p.CreatedBy].[ProjectVisibilityId], [p.CreatedBy].[CustomerId], [p.CreatedBy].[IsApiUser], [p.Status].[Id], [p.Status].[Code], [p.Status].[Name], [p.Customer].[Id], [p.Customer].[AxCustomerNumber], [p.Customer].[CreatedById],
[p.Customer].[CreatedOn], [p.Customer].[CreditTermId], [p.Customer].[CurrencyId], [p.Customer].[CustomerTypeId], [p.Customer].[DefaultPricing], [p.Customer].[FabricSystemFreight], [p.Customer].[IsActive], [p.Customer].[IsSpecialFreight],
[p.Customer].[LocalityRepId], [p.Customer].[MinFreightCharge], [p.Customer].[CompanyName], [p.Customer].[Notes], [p.Customer].[PrimaryBusinessId], [p.Customer].[ProgramAccountId], [p.Customer].[Prospect],
[p.Customer].[ShippingDollarsThreshold], [p.Customer].[ShippingMilesThreshold], [p.Customer].[SpecialNote], [p.Customer].[SpecialPlantInstruction], [p.Customer].[UpdatedById], [p.Customer].[UpdatedOn],
[p.Customer].[Xq1ProgGroupName], [p.Customer].[Xq1SrNo], [p.Customer].[XqCustomerNumber], [p.Customer.ProgramAccount].[Id], [p.Customer.ProgramAccount].[AxCustomerNumber], [p.Customer.ProgramAccount].[CreatedById],
[p.Customer.ProgramAccount].[CreatedOn], [p.Customer.ProgramAccount].[CreditTermId], [p.Customer.ProgramAccount].[CurrencyId], [p.Customer.ProgramAccount].[CustomerTypeId], [p.Customer.ProgramAccount].[DefaultPricing],
[p.Customer.ProgramAccount].[FabricSystemFreight], [p.Customer.ProgramAccount].[IsActive], [p.Customer.ProgramAccount].[IsSpecialFreight], [p.Customer.ProgramAccount].[LocalityRepId], [p.Customer.ProgramAccount].[MinFreightCharge],
[p.Customer.ProgramAccount].[CompanyName], [p.Customer.ProgramAccount].[Notes], [p.Customer.ProgramAccount].[PrimaryBusinessId], [p.Customer.ProgramAccount].[ProgramAccountId], [p.Customer.ProgramAccount].[Prospect],
[p.Customer.ProgramAccount].[ShippingDollarsThreshold], [p.Customer.ProgramAccount].[ShippingMilesThreshold], [p.Customer.ProgramAccount].[SpecialNote], [p.Customer.ProgramAccount].[SpecialPlantInstruction],
[p.Customer.ProgramAccount].[UpdatedById], [p.Customer.ProgramAccount].[UpdatedOn], [p.Customer.ProgramAccount].[Xq1ProgGroupName], [p.Customer.ProgramAccount].[Xq1SrNo], [p.Customer.ProgramAccount].[XqCustomerNumber],
[p.Customer.CustomerType].[Id], [p.Customer.CustomerType].[Code], [p.Customer.CustomerType].[Name]
FROM [Projects] AS [p]
LEFT JOIN [DataMigrationLogs] AS [p.DataMigrationLog] ON [p].[DataMigrationLogId] = [p.DataMigrationLog].[Id]
INNER JOIN [Security].[XQUsers] AS [p.UpdatedBy] ON [p].[UpdatedById] = [p.UpdatedBy].[Id]
INNER JOIN [Security].[XQUsers] AS [p.CreatedBy] ON [p].[CreatedById] = [p.CreatedBy].[Id]
INNER JOIN [ProjectStatuses] AS [p.Status] ON [p].[StatusId] = [p.Status].[Id]
LEFT JOIN [Customers] AS [p.Customer] ON [p].[CustomerId] = [p.Customer].[Id]
LEFT JOIN (
SELECT [p.Customer.LocalityRep].*
FROM [Security].[XQUsers] AS [p.Customer.LocalityRep]
WHERE [p.Customer.LocalityRep].[Discriminator] = N'SALES_SP'
) AS [t] ON [p.Customer].[LocalityRepId] = [t].[Id]
LEFT JOIN [Customers] AS [p.Customer.ProgramAccount] ON [p.Customer].[ProgramAccountId] = [p.Customer.ProgramAccount].[Id]
LEFT JOIN (
SELECT [p.Customer.ProgramAccount.LocalityRep].*
FROM [Security].[XQUsers] AS [p.Customer.ProgramAccount.LocalityRep]
WHERE [p.Customer.ProgramAccount.LocalityRep].[Discriminator] = N'SALES_SP'
) AS [t0] ON [p.Customer.ProgramAccount].[LocalityRepId] = [t0].[Id]
LEFT JOIN [CustomerTypes] AS [p.Customer.CustomerType] ON [p.Customer].[CustomerTypeId] = [p.Customer.CustomerType].[Id]
WHERE ([p.UpdatedBy].[Discriminator] IN (N'INT_SYSADMMIN', N'XqInternalUser', N'EXT', N'SALES_SP', N'XqUser') AND [p.CreatedBy].[Discriminator] IN (N'INT_SYSADMMIN', N'XqInternalUser', N'EXT', N'SALES_SP', N'XqUser')) AND (EXISTS (
SELECT 1
FROM [Security].[BusinessUserRegions] AS [r]
WHERE [r].[RegionId] IN (CAST(6 AS bigint), CAST(8 AS bigint), CAST(9 AS bigint)) AND ([t].[Id] = [r].[BusinessUserId])) OR ([p.Customer].[ProgramAccountId] IS NOT NULL AND EXISTS (
SELECT 1
FROM [Security].[BusinessUserRegions] AS [r0]
WHERE [r0].[RegionId] IN (CAST(6 AS bigint), CAST(8 AS bigint), CAST(9 AS bigint)) AND ([t0].[Id] = [r0].[BusinessUserId]))))
ORDER BY [p].[CreatedOn]
It takes more than 30 seconds to execute and Azure SQL Server has query timeout of maximum 30 second, so it results in exception
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
From my layman's understanding about SQL I think SQL us fetching the entire dataset for sorting and then trying to do pagination and hence it's taking more than 30 seconds. Please correct me if I am wrong.
I need to sort the result in query in self otherwise it will not provide the desired result set.
Is their any way to deal with this situation.
Any help is highly appreciated.
I have the following two queries that produce the results I need. Now the final output I truly need I would usually use python for after the results are returned, but unfortunately only SQL can be used.
Query A:
SELECT *
FROM openquery(PROD, 'SELECT `status`, computer_name, device_type
FROM assets
WHERE (device_type="SERVER")
AND (status="ACTIVE")')
Query B:
SELECT *
FROM openquery(AppMap, 'SELECT `t1`.`uaid` AS `uaid`, `t3`.`computer_name`,
FROM ((`applications` `t1`
JOIN `app_infrastructure` `t2` ON (((`t1`.`uaid` = `t2`.`uaid`))))
JOIN `infrastructure` `t3` ON ((`t2`.`infrastructure_id` = `t3`.`infrastructure_id`)));')
How I would want to process the results:
if a computer_name is in both A and B:
final_row = ['computer_name', 1]
elseif a computer_name is in A but not B:
final_row = ['computer_name', 0]
elseif a computer_name is in B but not A:
final_row = ['computer_name', 2]
So my final query results need to look like those rows, does that make sense?
In a stored procedure, use both queries to load table variables.
Then do a FULL OUTER JOIN query, joining the two table variables on computer_name, and use a CASE expression to get your final_row value for each computer name.
I'm trying to convert this SQL query into a linq query but not having much luck.
SELECT DISTINCT gen.ID
, gen.Name
, Ssec.System
FROM dbo.Generic Gen
JOIN dbo.SystemsSelected SSel
ON Gen.RecordID = SSel.RecordID
JOIN dbo.Security SSec
ON (
SSel.SystemA = CASE WHEN Ssec.System = '1stSystem' THEN 1 ELSE NULL END
OR SSel.SystemB = CASE WHEN Ssec.System = '2ndSystem' THEN 1 ELSE NULL END
)
and SSec.Username = 'myUserName'
I've had a look at the following posts but cannot apply the examples to my code:
Link join with case condition
Linq query with table joins
Any help would be appreciated!
Thanks
I cannot answer my own question directly but I found a workaround.
I used linq to instead call a Stored Procedure and that worked fine.
To do this:
1) Save code as a Stored Procedure that accepts a parameter named UserName
2) add the SP into Entity Framework Model
3) call the SP using:
public JsonResult GetNewTeams(string userUserName)
{
using (YourDBNameEntities db = new YourDBNameEntities ())
{
var ret = db.SP_YourStoredProcedureName(userUserName).ToList();
return Json(ret, JsonRequestBehavior.AllowGet);
}
}
Thats it, done, no more scary linq!! :)
sauce: Using stored procedures in LINQ
I have this raw sql that I am trying to run on grails
final session = sessionFactory.currentSession;
final String query = 'select count(A.*) from Artifact as A LEFT JOIN classification as C on (A.id=C.artifact_id) where C.id IS NULL OR C.active=0';
final sqlQuery = session.createSQLQuery(query);
def totalunclass = sqlQuery.with {
addEntity(Artifact);
}
But I am getting the error Message: SQL queries do not currently support returning aliases
I tried another method but failed there too achieve the result click here
Try changing your query line to look like this
final String query = 'select count(*) from Artifact A LEFT JOIN classification C on (A.id=C.artifact_id) where C.id IS NULL OR C.active=0';
Try this and remove aliases:
final session = sessionFactory.currentSession;
final String query = 'select count(Artifact.*) from Artifact LEFT JOIN classification on (Artifact.id=classification.artifact_id) where classification.id IS NULL OR classification.active=0';
final sqlQuery = session.createSQLQuery(query);
def totalunclass = sqlQuery.with {
addEntity(Artifact);
}
I am trying to translate a QueryExpression that is in some existing code into a T-SQL select statement.
I've run across the following statement and I'm having trouble understanding what they mean by a Natural Join:
linkEntity1.JoinOperator = JoinOperator.Natural;
Would this be equivalent to an Inner Join in T-SQL? Googling has not been much help.
Here's the rest of the QueryExpression Code:
QueryExpression query = new QueryExpression();
query.EntityName = "showinfo";
ColumnSet columns = new ColumnSet();
columns.Attributes = new String[] { "company" };
query.ColumnSet = columns;
query.Criteria = new FilterExpression();
query.Criteria.FilterOperator = LogicalOperator.And;
ConditionExpression condition1 = new ConditionExpression();
condition1.AttributeName = "company";
condition1.Operator = ConditionOperator.NotNull;
query.Criteria.Conditions = new ConditionExpression[] { condition1 };
LinkEntity linkEntity1 = new LinkEntity();
linkEntity1.JoinOperator = JoinOperator.Natural;
linkEntity1.LinkFromEntityName = "show";
linkEntity1.LinkFromAttributeName = "showid";
linkEntity1.LinkToEntityName = "showintegration";
linkEntity1.LinkToAttributeName = "showcode";
linkEntity1.LinkCriteria = new FilterExpression();
linkEntity1.LinkCriteria.FilterOperator = LogicalOperator.And;
ConditionExpression condition2 = new ConditionExpression();
condition2.AttributeName = "showend";
condition2.Operator = ConditionOperator.Null;
linkEntity1.LinkCriteria.Conditions = new ConditionExpression[] { condition2 };
query.LinkEntities = new LinkEntity[] { linkEntity1 };
There is no equivalent in SQL Server of a natural join where table intersect is based on column names by the RDBMS.
I'm glad of that because it is at best ambiguous and at worst dangerous. JOINs should be explicit. Examples why:
having a InsertedBy column in both tables (quite common): should we have to prefix with the table name to remove ambiguity?
future DDL that add columns that change JOIN semantics
See
Natural join in SQL Server
SQL Server - lack of NATURAL JOIN / x JOIN y USING(field)
Edit:
It looks like natural join means "don't repeat the column in the output" (like USING in MySQL would do) according to the JoinOperator Enumeration.
If I understand this (debatable!) it's misleading. Especially when I read the "LeftOuter" narrative..
A natural join compares all columns in the two tables that have the same column names. It's equivalent to an inner join with the matching columns explicitly listed.
Yes - the natural join is inner join - so you can write:
select * from tab1, tab2 where tab1.col1 = tab2.col1
as
select * from tab1 inner join tab2 on tab1.col1 = tab2.col1