I'm getting an issue during an ETL of client data where SQL chews up 100% CPU on my dev server. This only happens on occasion, and I have found the particular part of the SP that's causing it, but not sure why it's using so much CPU.
The LoadId and ClientId are both input variables for the SP. Basically, I am trying to find if any of the objects IDs in the Staging table (newly loaded data) match with existing objects (for a particular client), and also check in the Validation table (data gets a validation check before it gets processed) for any errors.
SELECT src.Id ,
o.Id ,
CASE WHEN o.Id IS NULL THEN 0
ELSE 1
END
FROM ObjectsStaging src
LEFT OUTER JOIN client.Objects o ON src.Id = o.UniqueId
WHERE src.LoadId = 22
AND ( o.ClientId IS NULL
OR o.ClientId = 3
)
AND NOT EXISTS ( SELECT 1
FROM dbo.ValidationLog v
WHERE v.LoadId = 22
AND v.RowId = src.RowId )
Maybe try this but change the v.PK to a Non nullible column in the V table.
SELECT src.Id ,
o.Id ,
CASE WHEN o.Id IS NULL THEN 0
ELSE 1
END
FROM ObjectsStaging src
LEFT OUTER JOIN client.Objects o ON src.Id = o.UniqueId
LEFT OUTER JOIN dbo.ValidationLog v on v.LoadId = 22 AND v.RowId = src.RowId
WHERE src.LoadId = 22
AND ( o.ClientId IS NULL
OR o.ClientId = 3
)
AND v.PK is null -- V.loadid is null ? --(same as not exists)
Related
I have a strange problem, if I execute the following query, Sql Server (2008 and 2016) needs about 35 seconds to complete.
SELECT DISTINCT F.FilterTitle, F.FilterSlug, F.FilterOrderList, F.FilterType
FROM products p
JOIN products_translations pt ON pt.ProductId = p.Id AND pt.Culture = 'it'
JOIN facets F ON F.ProductId = p.Id AND F.Culture = 'it' AND F.FilterType = 2
JOIN products_categories pc ON pc.productId = p.id
JOIN categories C ON pc.CategoryId = C.Id
JOIN categories_slugs cps ON cps.CategoryId = C.ParentCategoryId AND cps.Culture = 'it'
JOIN categories_slugs cs ON cs.CategoryId = C.Id AND cs.Culture = 'it'
WHERE cps.value = 'string1' AND cs.value = 'string2'
AND CONTAINS(pt.DescriptionForSearch, '"value*"', LANGUAGE 1040) --this is a full text
AND p.Visible = 1 AND p.Payments = 0 --both are bit fields
ORDER BY F.FilterType, F.FilterOrderList, F.FilterTitle;
If I comment the line
and p.Visible = 1 and p.Payments = 0
the query needs just a couple of ms. The same if I remove the line
and contains(pt.DescriptionForSearch, '"value*"' , language 1040)
and leave the other statement intact. I also created two indexes for both the bit fields without changes.
Can someone shed some light on why this is happening?
UPDATE
At this link the two execution plans that I forgot to include.
I have a table named SalesOrders. I merge data form multiple tables into it. Within this table, I have a column named Ack which will toggle to 'N' when the row is inserted or updated (I toggle 'Y' in C# code). the trouble I'm having is when I run my query, I only want Ack to change to 'N' when something changes. I tried adding conditions in the WHEN MATCHED statement but the table never updates when there is a change.
MERGE QA.dbo.SalesOrders AS TARGET USING(SELECT SOD.ORDNUM_28 +
LINNUM_28 + DELNUM_28 AS [SalesOrd], SOD.PRTNUM_28, PM.PMDES1_01,
SOD.CURDUE_28, SOD.DUEQTY_28, CPD.CUSTPRT_103, SOM.CUSTPO_27,
CPD.UDFREF_103, CPD.PRTNUM_103,SOD.CreationDate, CM.EMAIL1_23, SOD.ORDNUM_28
FROM SO_Detail AS SOD FULL OUTER JOIN
Customer_Part_Data AS CPD ON SOD.PRTNUM_28 = CPD.PRTNUM_103 FULL OUTER JOIN
SO_Master AS SOM ON SOD.ORDNUM_28 = SOM.ORDNUM_27 FULL OUTER JOIN
Part_Master AS PM ON SOD.PRTNUM_28 = PM.PRTNUM_01 FULL OUTER JOIN
Customer_Master AS CM ON SOD.CUSTID_28 = CUSTID_23
WHERE (STATUS_28 = '3') AND (SOD.CreationDate > '09 / 14 / 2017') AND
(CUSTPO_27 <> ' ') AND (SOM.STYPE_27 = 'CU')
AND (SOD.STK_28 NOT LIKE '%RMA%') AND (SOD.STYPE_28 = 'CU')) SOURCE
ON OrderNum = SOURCE.SalesOrd
WHEN MATCHED AND PartNum <> SOURCE.PRTNUM_28 OR Description <>
SOURCE.PMDES1_01 OR DueQty <> SOURCE.DUEQTY_28 OR CustPartNum <>
SOURCE.CUSTPRT_103 OR CustPo <> SOURCE.CUSTPO_27 OR CustRev <>
SOURCE.UDFREF_103 OR ShipDate <> SOURCE.CURDUE_28 OR email <>
SOURCE.EMAIL1_23 // This does not work
THEN
UPDATE
SET PartNum = SOURCE.PRTNUM_28, Description =
SOURCE.PMDES1_01, DueQty = SOURCE.DUEQTY_28, CustPartNum =
SOURCE.CUSTPRT_103, CustPo = SOURCE.CUSTPO_27, CustRev =
SOURCE.UDFREF_103, ShipDate = SOURCE.CURDUE_28, email = SOURCE.EMAIL1_23,
OrgDate = SOURCE.CreationDate, Ack = 'N'
I noted in the code what does not work (Everything after WHEN MATCHED) - no error just does not update when something is changed. If I remove the code after AND, then everything updates but every time the query is run - thus changing Ack to "N" when nothing really changed.
With some basic formatting the error became pretty obvious. You were missing a parenthesis in your predicates when matched. Notice how when this isn't a wall of sql you can actually see what is going on.
MERGE QA.dbo.SalesOrders AS TARGET USING
(
SELECT SOD.ORDNUM_28 + LINNUM_28 + DELNUM_28 AS [SalesOrd]
, SOD.PRTNUM_28
, PM.PMDES1_01
, SOD.CURDUE_28
, SOD.DUEQTY_28
, CPD.CUSTPRT_103
, SOM.CUSTPO_27
, CPD.UDFREF_103
, CPD.PRTNUM_103
, SOD.CreationDate
, CM.EMAIL1_23
, SOD.ORDNUM_28
FROM SO_Detail AS SOD
FULL OUTER JOIN Customer_Part_Data AS CPD ON SOD.PRTNUM_28 = CPD.PRTNUM_103
FULL OUTER JOIN SO_Master AS SOM ON SOD.ORDNUM_28 = SOM.ORDNUM_27
FULL OUTER JOIN Part_Master AS PM ON SOD.PRTNUM_28 = PM.PRTNUM_01
FULL OUTER JOIN Customer_Master AS CM ON SOD.CUSTID_28 = CUSTID_23
WHERE STATUS_28 = '3'
AND SOD.CreationDate > '09 / 14 / 2017' --Is this a date column? If so you need to use the ANSI standard YYYYmmdd
AND CUSTPO_27 <> ' '
AND SOM.STYPE_27 = 'CU'
AND SOD.STK_28 NOT LIKE '%RMA%'
AND SOD.STYPE_28 = 'CU'
) SOURCE
ON OrderNum = SOURCE.SalesOrd
WHEN MATCHED
AND
( --you need this here
PartNum <> SOURCE.PRTNUM_28
OR Description <> SOURCE.PMDES1_01
OR DueQty <> SOURCE.DUEQTY_28
OR CustPartNum <> SOURCE.CUSTPRT_103
OR CustPo <> SOURCE.CUSTPO_27
OR CustRev <> SOURCE.UDFREF_103
OR ShipDate <> SOURCE.CURDUE_28
OR email <> SOURCE.EMAIL1_23 --// This does not work
) --Without the parenthesis the update would fire when ANY of those conditions are met
THEN
UPDATE
SET PartNum = SOURCE.PRTNUM_28
, Description = SOURCE.PMDES1_01
, DueQty = SOURCE.DUEQTY_28
, CustPartNum = SOURCE.CUSTPRT_103
, CustPo = SOURCE.CUSTPO_27
, CustRev = SOURCE.UDFREF_103
, ShipDate = SOURCE.CURDUE_28
, email = SOURCE.EMAIL1_23
, OrgDate = SOURCE.CreationDate
, Ack = 'N'
Figured it out: My issue was, I am a victim of not understanding NULL. There is good discussion found here: Why does NULL = NULL evaluate to false in SQL server. Basically, the conditions could not fire the update because they simply did not know the answer (NULL). I needed to update my INSERT to include a CASE so if the value is NULL - insert ' ' (a Blank) so my conditions would have something to compare.
I need to merge a as seen below, but I only want to do this if the table dbo.Problem has the value 1 in the IsValid column. dbo.Problem is linked to dbo.delivery by dbo.Problem.ID = dbo.Delivery.IssueID.
merge [dbo].[Delivery] as D
using [ReportSvr].[Report].[dbo].[Delivery] as LinkD
on ( D.[ID] = LinkD.[ID])
WHEN NOT MATCHED BY TARGET
THEN INSERT(
[ID]
,[IssueID]
,[column2]
,[column3]
)
VALUES(
[ID]
,[IssueID]
,[column]
,[column]
);
GO
Is tried something like;
select d.IssueID, i.IsValid
From Delivery d
left join (Select* From Issue where IsValid = 1) i
on d.IssueID = i.ID
since first step must be to be able to the wanted rows.
Output:
IssueID IsValid
1 1
2 NULL
6 NULL
7 1
8 1
9 1
10 NULL
As you see I failed terribly....
I'm new (second day) to SQL, sorry if this is an easy task I should know about.
I hope you are able to help me.
Using an INNER JOIN will only show deliveries that have a matching record in the Issue table. An INNER JOIN will only show records where there's a match on the join field. In your attempt you're using a LEFT JOIN which will show everything in the left table, [Delivery] ("left" makes sense if you show the entire query on a single line) and matching records in the right table, [Issue].
SELECT d.IssueID, i.IsValid
FROM Delivery d
INNER JOIN Issue i
ON d.IssueID = i.ID
WHERE i.IsValid = 1
Thanks for helping me out, I managed to solved the problem with the help from Russell Fox. I just wanted to post the answer.
merge [dbo].[Delivery] as D
using(
select d.*, i.IsValid
From [ErrorReportSvr].[ErrorReport].[dbo].[Delivery] d
inner join (Select* From [ErrorReportSvr].[ErrorReport].[dbo].[Issue] where IsValid = 1) i
on d.IssueID = i.ID
) as LinkD
on ( D.[ID] = LinkD.[ID])
WHEN NOT MATCHED BY TARGET
THEN INSERT(
[ID]
,[IssueID]
,[column2]
,[column3]
)
VALUES(
[ID]
,[IssueID]
,[column2]
,[column3]
);
GO
I have three tables (say A,B and C) with same column names and datatype. And these tables can be joined using four unique columns, say name,company,Seq_Number and role. Now I want to select records of particular role from table A and cross check them with the records in Table B and C.If they do not exist in both the tables, then we need to deactivate/remove those records from Table A. The problem is, the records which does not exist in table B might exist in Table C. So, I need to remove the records of particular role only if they don't exist in both B & C tables. I tried with the below query. But it is not returning the expected result. Kindly help me on this
SELECT DISTINCT FAT_Cust.name
, FAT_Cust.Company
, FAT_Cust.role
, FAT_Cust.Seq_Number
, Cust.name
, Cus.Company
, Cust.role
, Cust.Seq_Numberfrom (
SELECT DISTINCT ALC.NAME, ALC.Company, ALC.ROLE, ALC.Seq_Number
FROM AL_Customer ALC
INNER JOIN BL_Customer LPC ON ALC.NAME = LPC.NAME
AND ALC.Company = LPC.Company
AND ALC.ROLE = LPC.ROLE
AND ALC.Seq_Number = LPC.Seq_Number
AND ALC.Record_Active = 1
UNION SELECT DISTINCT ALC.NAME, ALC.Company, ALC.ROLE, ALC.Seq_Number
FROM AL_Customer ALC
INNER JOIN CL_Customer CLC ON ALC.NAME = CLC.NAME
AND ALC.Company = CLC.Company
AND ALC.ROLE = CLC.ROLE AND ALC.Seq_Number = CLC.Seq_Number
AND ALC.Record_Active = 1
) Cust
RIGHT OUTER JOIN AL_Customer FAT_Cust ON FAT_Cust.NAME = Cust.NAME
AND FAT_Cust.Company = Cust.Company
AND FAT_Cust.ROLE = Cust.ROLE
AND FAT_Cust.Seq_Number = Cust.Seq_Number
AND FAT_Cust.Record_Active = 1
WHERE Cust.NAME IS NULL
AND Cust.Company IS NULL
AND Cust.ROLE IS NULL
AND Cust.Seq_Number IS NULL
AND Cust.ROLE < > 'OWN'
Please try the query given below
SELECT ALC.* FROM AL_Customer ALC
LEFT JOIN BL_Customer BPC ON ALC.NAME = BPC.NAME
AND ALC.Company = BPC.Company
AND ALC.ROLE = BPC.ROLE
AND ALC.Seq_Number = BPC.Seq_Number
AND ALC.Record_Active = 1
AND BLC.Record_Active = 1
LEFT JOIN CL_Customer CPC ON ALC.NAME = CPC.NAME
AND ALC.Company = CPC.Company
AND ALC.ROLE = CPC.ROLE
AND ALC.Seq_Number = CPC.Seq_Number
AND ALC.Record_Active = 1
AND CLC.Record_Active = 1
WHERE ALC.Record_Active = 1
AND (BPC.NAME IS NULL)
AND (CPC.NAME IS NULL)
you can add more condition is where class to narrow down the matching criteria. the above query is assuming that name is present for all the records in the table. I hope this will resolve your issue.
I have a feeling once i see the solution i'll slap my forehead, but right now I'm not seeing it.
I have a lookup table, say TableB, which looks like this. All fields are INT except the last two which are BOOL.
ID, TableA_ID, Value, Required, Disqualifies
I have a list of TableA_Id values (1, 2, 3 ) etc
For each record in this table, either Required can be true or disqualified can be true - they cant both be true at the same time. They can both be false or null though. There can be duplicate values of TableA_Id but there should never be duplicates of TableA_Id and Value
If required is true for any of those TableA_ID values, and none of those values are in my list, return no records. If none of the values are marked as required (required = 0 or null) then return records UNLESS any of the values are marked as Disqualifies and are in the list, in which case i want to return no records.
So - if a field is required and i dont have it, dont return any records. If a field is marked as disqualified and i have it, don't return any records. Only return a record if either i have a required value or don't have a disqualified value or there are no required values.
I hope I explained myself clearly.
Thanks in advance for pointing me in the right direction.
As an example of what my records might look like:
ID TableA_ID Value Required Disqualifies
-- --------- ----- -------- ------------
1 123 1 True False
2 123 2 True False
3 123 3 False False
4 123 4 False True
5 456 1 False True
6 456 2 False False
Given this set of sample data, if we're working with TableA_Id 123 and my list of values is lets say 1 and 3, i would get data returned because i have a required value and dont have any disqualified values. If my list of values were just 3, i'd get no records since i'm missing of the Required values. If my list of values were 1 and 4, i'd get no records because 4 is marked as disqualified.
Now if we're working with TableA_Id 456, the only list of values that would return any records is 2.
Maybe i should post the whole SQL query - i was trying to keep this short to make it easier for everyone, but it looks like maybe that's not working so well.
Here is the full dynamically generated query. The bit i am working on now is the 2nd line from the bottom. To equate this to my example, t.id would be TableA_ID, Value would be PDT_ID.
SELECT DISTINCT t.ID, t.BriefTitle, stat.Status, lstat.Status AS LocationStatus, st.SType, t.LAgency, l.City, state.StateCode
,( SELECT TOP 1 UserID
FROM TRecruiter
WHERE TrialID = t.ID AND Lead = 1 ), l.ID as LocationID
, l.WebBased
FROM Trial t
INNER JOIN Location l ON t.ID = l.TrialID
FULL JOIN pdt on t.ID = pdt.trialid
FULL JOIN pdm on t.ID = pdm.TrialID
FULL JOIN s on t.ID = s.TrialID
FULL JOIN hy on t.ID = hy.TrialID
FULL JOIN ta on t.ID = ta.TrialID
FULL JOIN stt on t.ID = stt.TrialID
FULL JOIN [Status] stat ON t.StatusID = stat.ID
FULL JOIN st ON t.StudyTypeID = st.ID
FULL JOIN State state ON l.StateID = state.ID
FULL JOIN [Status] lstat ON l.StatusID = lstat.ID
FULL JOIN ts ON t.ID = ts.TrialID
FULL JOIN tpdm ON t.ID = tpdm.TrialID
WHERE ((t.ID IS NOT NULL)
AND (EligibleHealthyVolunteers IS NULL OR EligibleHealthyVolunteers = 1 OR (0 = 0 AND EligibleHealthyVolunteers = 0))
AND (eligiblegenderid is null OR eligiblegenderid = 1 OR eligiblegenderid = 3)
AND ((EligibleMinAge <= 28 AND EligibleMaxAge >= 28) OR (EligibleMinAge <= 28 AND EligibleMaxAge is null) OR (EligibleMinAge IS NULL AND EligibleMaxAge >= 28))
AND (HYID = 6 AND (hy.Disqualify = 0 OR hy.Disqualify IS NULL AND NOT EXISTS (SELECT * FROM hy WHERE t.id = hy.TrialID AND hy.Req =1)) OR HYID = 6 AND hy.req = 1)
AND (PDT_ID IN (1) AND ( pdt.Disqualify = 0 OR pdt.Disqualify IS NULL AND NOT EXISTS (select * from pdt where t.id = pdt.TrialID AND pdt.Req = 1)) OR PDT_ID IN (1) AND (pdt.Req = 1 AND (pdt.Disqualify = 0 or pdt.Disqualify is null )))
) AND ((3959 * acos(cos(radians(34.18)) * cos(radians(l.Latitude)) * cos(radians(l.Longitude) - radians(-118.46)) + sin(radians(34.18)) * sin(radians(l.Latitude)))) <= 300 OR l.Latitude IS NULL) AND t.IsPublished = 1 AND (t.StatusID = 1 OR t.StatusID = 2)
I've changed/shortened some table names just for security/privacy reasons.
Edit:
I think i am close to getting this working, but I'm getting tripped up on the logic again.
I have the following bit of sql:
AND ( exists (SELECT * FROM pdt WHERE Req = 1 AND trialid = t.id AND pdT_ID IN (2) ) AND EXISTS (SELECT * FROM pdt WHERE Req = 1 AND trialid = t.id ) )
I'm not sure how to structure this. Those two exists statement should make the whole thing true in the following combination:
True & False
True & True
False & False
If it's False & True, then the whole thing is false. In other words if there is a Req =1 AND the PDT_ID that is marked as Req=1 is not in our list (in the example above the list just contains '2') then return false.
EDIT:
I think i finally got it.
AND NOT EXISTS (SELECT * FROM pdt WHERE Disqualify = 1 AND trialid = t.id AND PDT_ID IN (2) )
AND NOT ( NOT exists (SELECT * FROM pdt WHERE Req = 1 AND trialid = t.id AND PDT_ID IN (2) ) AND EXISTS (SELECT * FROM pdt WHERE Req = 1 AND trialid = t.id ) )
So far this seems to work in testing. Although I'm only working with two values of PDT_ID. If this does resolve my problem, i will come back and give someone the credit for helping me.
SELECT *
FROM TABLEB B
WHERE
(
B.REQUIRED = 1
AND EXISTS
(
SELECT 1
FROM TABLEA A
WHERE A.ID =B.TABLEA_ID
)
)
OR
(
B.REQUIRED != 1
AND B.DISQUALIFIES <> 1
)
OR
(
B.REQUIRED != 1
AND B.DISQUALIFIES = 1
AND EXISTS
(
SELECT 1
FROM TABLEA A
WHERE A.ID =B.TABLEA_ID
)
)
UPDATE - after the EDIT and explanation from OP:
Change the line
FULL JOIN pdt on t.ID = pdt.trialid
To
FULL JOIN (SELECT * FROM pdt BB WHERE
BB.TrialID IN (SELECT AA.ID FROM Trial AA WHERE AA.ID = BB.TrialID) AND
1 > (SELECT COUNT(*) FROM Trial A
LEFT OUTER JOIN pdt B ON B.Req != 1 AND B.Disqualify != 1 AND B.TrialID = A.ID
WHERE B.TrialID IS NULL)) pdt ON t.ID = pdt.TiralID
AND change the line before last from
AND (PDT_ID IN (1) AND ( pdt.Disqualify = 0 OR pdt.Disqualify IS NULL AND NOT EXISTS (select * from pdt where t.id = pdt.TrialID AND pdt.Req = 1)) OR PDT_ID IN (1) AND (pdt.Req = 1 AND (pdt.Disqualify = 0 or pdt.Disqualify is null )))
To
AND PDT_ID IN (1)
(You seem to have found a solution, yet I've decided to share my thoughts about this problem anyway.)
Given you've got a set of TableA IDs, each of which is accompanied by a set of some values, and you want to test the entire row set against this TableB thing using the rules you've set forth, I think the entire checking process might look like this:
Match every pair of TableA.ID and Value against TableB and get aggregate maximums of Required and Disqualifies for every TableA.ID along the way.
Derive a separate list of TableA_ID values with their corresponding maximum values of Required, from TableB. That will be for us to know whether a particular TableA_ID must have a required value at all.
Match the row set obtained at Stage 1 against the derived table (Stage 2) and check the aggregate values:
1) if the actual aggregate Disqualifies for a TableA_ID is 1, discard this TableA_ID set;
2) if a TableA_ID has a match in the Stage 2 derived table and the aggregate maximum of Required that we obtained at Stage 1 doesn't match the maximum Required in the derived table, discard the set as well.
Something tells me that it would be better at this point to move on to some sort of illustration. Here's a sample script, with comments explaining which part of the script implements which part of the description above:
;
WITH
/* this is the row set to be tested and which
is supposed to contain TableA.IDs and Values */
testedRowSet AS (
SELECT
TableA.ID AS TableA_ID,
SomethingElse.TestedValue AS Value,
...
FROM TableA
JOIN SomethingElse ON some_condition
...
),
/* at this point, we are getting the aggregate maximums
of TableB.Required and TableB.Disqualifies for every
TableA_ID in testedRowSet */
aggregated AS (
SELECT
testedRowSet.TableA_ID,
testedRowSet.Value,
...
DoesHaveRequiredValues = MAX(CASE TableB.Required WHEN 1 THEN 1 ELSE 0 END) OVER (PARTITION BY testedRowSet.TableA_ID),
HasDisqualifyingValues = MAX(CASE TableB.Disqualifies WHEN 1 THEN 1 ELSE 0 END) OVER (PARTITION BY testedRowSet.TableA_ID)
FROM testedRowSet
LEFT JOIN TableB ON testedRowSet.TableA_ID = TableB.TableA_ID
AND testedRowSet.Value = TableB.Value
),
/* this row set will let us see whether a particular
TableA_ID must have a required value */
properties AS (
SELECT
TableA_ID,
MustHaveRequiredValues = MAX(CASE Required WHEN 1 THEN 1 ELSE 0 END)
FROM TableB
GROUP BY TableA_ID
),
/* this is where we are actually checking the previously
obtained aggregate values of Required and Disqualifies */
tested AS (
SELECT
aggregated.TableA_ID,
aggregated.Value,
...
FROM aggregated
LEFT JOIN properties ON aggregated.TableA_ID = properties.TableA_ID
WHERE aggregated.HasDisqualifyingValues = 0
AND (properties.TableA_ID IS NULL
OR properties.MustHaveRequiredValues = aggregated.DoesHaveRequiredValues)
)
SELECT * FROM tested