Related
I know that there are quite many threads on "Windowed functions can only appear in the select or ORDER BY clases" topic, but I have read them through and just don't seem to get any of those trick to work for me. So here I go.
My Query you can see below the line. I have highlighted the part of the Query which is "causing" me issues or problems. The thing is that I would like to get
"LocalCurrentAmount not between -1.000000 and 1.000000"
in somehow. I have tried the CTE versioon, but somehow didn't work and my declare tables and then the details view started causing problems.
Would really appreciate your help!
Jaanis
declare #exceptions1 table (CustomerCode varchar(7), Exception_comment varchar(15));
insert into #exceptions1 (CustomerCode, Exception_comment)
VALUES
(3514437,'Exception'),(3500977,'Exception'),(3295142,'Exception'), ...
declare #exceptions2 table (CustomerCode2 varchar(7), Exception_comment2 varchar(15));
insert into #exceptions2 (CustomerCode2, Exception_comment2)
VALUES
(3390437,'VIP SE') ,(3390438,'VIP SE') ,(3390481,'VIP SE'), ...
declare #exceptions3 table (CustomerCode3 varchar(7), Exception_comment3 varchar(15));
insert into #exceptions1 (CustomerCode, Exception_comment)
VALUES
(1530350, 'DK Exception'), (1533834, 'DK Exception'), (1530002, 'DK Exception'), ...
with Details as
(
select ard.AccountsReceivableHeaderID, sum(ard.TransactionAmountOC) as DetailAmountOC , Max(DetailSequenceCode) as DetailSequenceCode, Max( ard.BatchDate) as BatchDate
from dmbase.fAccountsReceivableDetail ard
join dmbase.fAccountsReceivable arh on ard.AccountsReceivableHeaderID = arh.AccountsReceivableID
where ard.BatchDate <= getdate()
group by AccountsReceivableHeaderID
)
SELECT
comp.CompanyCode
,convert(varchar(10),ar.InvoiceDate, 103) as Invoice_date -- dd/MM/yyyy format
,case
when ar.IsCreditMemo = 'Y' then 'Memo (Credit/Debit)'
when ar.InvoiceCode = '0' then 'Payment'
else 'Invoice'
end as Description
,isnull(cm.SummaryInvoiceCode, ar.InvoiceSummaryCode) InvoiceSummaryCode
,case
when len(ar.InvoiceSequenceCode) = '1' then CONCAT(ar.InvoiceCode,'-000',ar.InvoiceSequenceCode)
when len(ar.InvoiceSequenceCode) = '2' then CONCAT(ar.InvoiceCode,'-00',ar.InvoiceSequenceCode)
when len(ar.InvoiceSequenceCode) = '3' then CONCAT(ar.InvoiceCode,'-0',ar.InvoiceSequenceCode)
else CONCAT(ar.InvoiceCode,'-',ar.InvoiceSequenceCode)
end as Invoice#
,**(ar.OriginalInvoiceAmountOC
+
case
when row_number() over (partition by AccountsReceivableID order by ar.InvoiceCode) = 1 then isnull(vat.vatAdjustment, 0)
else 0
end
+
coalesce(det.DetailAmountOC, 0)) * coalesce(cer.CurrencyExchangeRate, ar.CurrencyExchangeRate) AS LocalCurrentAmount**
,(ar.OriginalInvoiceAmountOC
+
case
when row_number() over (partition by AccountsReceivableID order by ar.InvoiceCode) = 1 then isnull(vat.vatAdjustment, 0)
else 0
end
+ coalesce(det.DetailAmountOC, 0)) AS CurrentAmount
,ar.OriginalInvoiceAmountOC
+
case
when row_number() over (partition by AccountsReceivableID order by ar.InvoiceCode) = 1 then isnull(vat.vatAdjustment, 0)
else 0
end as OriginalInvoiceAmountOC
,ar.InvoiceCurrencyCode
,cust.CustomerCode
,upper(cust.CustomerName) as CustomerName
from
dmbase.fAccountsReceivable ar
INNER JOIN dmbase.dlocation loc
ON loc.LocationID = ar.LocationID
INNER JOIN dmbase.dCustomer cust
ON cust.CustomerID = ar.CustomerID
LEFT JOIN dmbase.VatAdjustment vat
on ar.InvoiceCode = vat.contractNumber
and ar.InvoiceSequenceCode = vat.invoiceSequence
and cust.CustomerCode = vat.CustomerNumber
and loc.CompanyCode = vat.companyCode
inner join dmbase.dCompany comp
on (ar.CompanyID = comp.CompanyID)
left join dmbase.dAccountsReceivableInvoiceStatus aris
on (aris.ARInvoiceStatusAMID=ar.ARInvoiceStatusAMID)
left hash join Details det
on (ar.AccountsReceivableID = det.AccountsReceivableHeaderID)
left join dmbase.dCurrencyExchangeRate cer
on (comp.CompanyCode = cer.CompanyCode
and ar.InvoiceCurrencyCode = cer.CurrencyCode
and case ar.InvoiceDate when '1900-01-01' then getdate() else ar.InvoiceDate end between cer.ValidFrom and cer.ValidTo)
left join dmbase.fContractClosedHeader ccd
on ccd.ContractNumber = ar.InvoiceCode
and ccd.ContractSeqNumber = ar.InvoiceSequenceCode
and ccd.CompanyID = ar.CompanyID
and ccd.ContractNumber!='0'
and ccd.CreditMemoContractNumber != '0'
left join dmbase.fAccountsReceivableHeader cm
on ccd.CreditMemoContractNumber = cm.ContractNumber
and ccd.CreditMemoSequenceCode = cm.ContractSeqNumber
and cm.CompanyID = ccd.CompanyID
where
(aris.ARInvoiceStatusCode = 'OP' or (ar.LastPaymentDate >= getdate())
or (ar.TotalAdjustmentsAmountOC <> 0 and ar.CurrentAmountLC = 0
and (ar.OriginalInvoiceAmountOC + isnull(vat.vatAdjustment, 0) ) + coalesce(det.DetailAmountOC, 0) <> 0)
)
and ar.OriginalInvoiceAmountOC <= 0
and ar.IsCreditMemo = 'Y' -- ainult Memo (Credit/Debit)
and cust.InternalCustomerType = 'External'
and cust.CustomerName not in ('RR AJM', 'RAMIRENT', 'Ramirent')
and cust.CustomerName not like '%[7][0-9][0-9][0-9]%'
and ar.InvoiceDate <= EOMONTH(getdate(),-3
and cust.CustomerCode NOT IN
(select CustomerCode from #exceptions1
union
select CustomerCode2 from #exceptions2
union
select CustomerCode3 from #exceptions3)
order by Invoice_date
When using a Window function, like you said in your post, you can't use it in the WHERE clause. The common solution, therefore, is to use a CTE and reference it in the WHERE outside of it. I've used your CTE, however, without any kind of sample data this is a total guess. I've also left a lot of comments for you, and changed some other parts of the SQL, as some of the clauses you have will effect your query's performance.
WITH
Details AS
(SELECT ard.AccountsReceivableHeaderID,
SUM(ard.TransactionAmountOC) AS DetailAmountOC,
MAX(DetailSequenceCode) AS DetailSequenceCode,
MAX(ard.BatchDate) AS BatchDate
FROM dmbase.fAccountsReceivableDetail ard
JOIN dmbase.fAccountsReceivable arh ON ard.AccountsReceivableHeaderID = arh.AccountsReceivableID
WHERE ard.BatchDate <= GETDATE()
GROUP BY AccountsReceivableHeaderID),
Summary AS(
SELECT comp.CompanyCode,
CONVERT(varchar(10), ar.InvoiceDate, 103) AS Invoice_date, -- dd/MM/yyyy format
CASE
WHEN ar.IsCreditMemo = 'Y' THEN 'Memo (Credit/Debit)'
WHEN ar.InvoiceCode = '0' THEN 'Payment'
ELSE 'Invoice'
END AS Description,
ISNULL(cm.SummaryInvoiceCode, ar.InvoiceSummaryCode) AS InvoiceSummaryCode,
CASE
WHEN LEN(ar.InvoiceSequenceCode) = '1' THEN CONCAT(ar.InvoiceCode, '-000', ar.InvoiceSequenceCode)
WHEN LEN(ar.InvoiceSequenceCode) = '2' THEN CONCAT(ar.InvoiceCode, '-00', ar.InvoiceSequenceCode)
WHEN LEN(ar.InvoiceSequenceCode) = '3' THEN CONCAT(ar.InvoiceCode, '-0', ar.InvoiceSequenceCode)
ELSE CONCAT(ar.InvoiceCode, '-', ar.InvoiceSequenceCode)
END AS Invoice#,
(ar.OriginalInvoiceAmountOC + CASE
WHEN ROW_NUMBER() OVER (PARTITION BY AccountsReceivableID ORDER BY ar.InvoiceCode) = 1 THEN ISNULL(vat.vatAdjustment, 0)
ELSE 0
END + COALESCE(det.DetailAmountOC, 0)) * COALESCE(cer.CurrencyExchangeRate, ar.CurrencyExchangeRate) AS LocalCurrentAmount,
(ar.OriginalInvoiceAmountOC + CASE
WHEN ROW_NUMBER() OVER (PARTITION BY AccountsReceivableID ORDER BY ar.InvoiceCode) = 1 THEN ISNULL(vat.vatAdjustment, 0)
ELSE 0
END + COALESCE(det.DetailAmountOC, 0)) AS CurrentAmount,
ar.OriginalInvoiceAmountOC + CASE
WHEN ROW_NUMBER() OVER (PARTITION BY AccountsReceivableID ORDER BY ar.InvoiceCode) = 1 THEN ISNULL(vat.vatAdjustment, 0)
ELSE 0
END AS OriginalInvoiceAmountOC,
ar.InvoiceCurrencyCode,
cust.CustomerCode,
UPPER(cust.CustomerName) AS CustomerName
FROM dmbase.fAccountsReceivable ar
INNER JOIN dmbase.dlocation loc ON loc.LocationID = ar.LocationID
INNER JOIN dmbase.dCustomer cust ON cust.CustomerID = ar.CustomerID
LEFT JOIN dmbase.VatAdjustment vat ON ar.InvoiceCode = vat.contractNumber
AND ar.InvoiceSequenceCode = vat.invoiceSequence
AND cust.CustomerCode = vat.CustomerNumber
AND loc.CompanyCode = vat.companyCode
INNER JOIN dmbase.dCompany comp ON (ar.CompanyID = comp.CompanyID)
LEFT JOIN dmbase.dAccountsReceivableInvoiceStatus aris ON (aris.ARInvoiceStatusAMID = ar.ARInvoiceStatusAMID)
LEFT HASH JOIN Details det ON (ar.AccountsReceivableID = det.AccountsReceivableHeaderID)
LEFT JOIN dmbase.dCurrencyExchangeRate cer ON (comp.CompanyCode = cer.CompanyCode
AND ar.InvoiceCurrencyCode = cer.CurrencyCode
AND CASE ar.InvoiceDate WHEN '1900-01-01' THEN GETDATE()ELSE ar.InvoiceDate END BETWEEN cer.ValidFrom AND cer.ValidTo)
LEFT JOIN dmbase.fContractClosedHeader ccd ON ccd.ContractNumber = ar.InvoiceCode
AND ccd.ContractSeqNumber = ar.InvoiceSequenceCode
AND ccd.CompanyID = ar.CompanyID
AND ccd.ContractNumber != '0'
AND ccd.CreditMemoContractNumber != '0'
LEFT JOIN dmbase.fAccountsReceivableHeader cm ON ccd.CreditMemoContractNumber = cm.ContractNumber
AND ccd.CreditMemoSequenceCode = cm.ContractSeqNumber
AND cm.CompanyID = ccd.CompanyID
WHERE (aris.ARInvoiceStatusCode = 'OP'
OR (ar.LastPaymentDate >= GETDATE())
OR (ar.TotalAdjustmentsAmountOC <> 0
AND ar.CurrentAmountLC = 0
AND (ar.OriginalInvoiceAmountOC + ISNULL(vat.vatAdjustment, 0)) + COALESCE(det.DetailAmountOC, 0) <> 0)) --Why ISNULL for one, COALESCE for the other? Both will make the query non-SARGable
AND ar.OriginalInvoiceAmountOC <= 0
AND ar.IsCreditMemo = 'Y' -- ainult Memo (Credit/Debit)
AND cust.InternalCustomerType = 'External'
AND cust.CustomerName NOT IN ('RR AJM', 'RAMIRENT', 'Ramirent')
AND cust.CustomerName NOT LIKE '%[7][0-9][0-9][0-9]%' --A leading wildcard is going to perform slow
AND ar.InvoiceDate <= EOMONTH(DATEADD(DAY, -3, GETDATE())) --I have changed this from EOMONTH(GETDATE(), -3 (which made no sense)
AND NOT EXISTS (SELECT 1
FROM #exceptions1 E
WHERE E.CustomerCode = cust.CustomerCode) --Changed to EXISTS binned UNION you should use UNION ALL if you're doing something like this, it'll be quicker)
AND NOT EXISTS (SELECT 1
FROM #exceptions2 E
WHERE E.CustomerCode = cust.CustomerCode) --Changed to EXISTS binned UNION (you should use UNION ALL if you're doing something like this, it'll be quicker)
AND NOT EXISTS (SELECT 1
FROM #exceptions3 E
WHERE E.CustomerCode = cust.CustomerCode)) --Changed to EXISTS binned UNION you should use UNION ALL if you're doing something like this, it'll be quicker)
SELECT *
FROM Summary
WHERE LocalCurrentAmount NOT BETWEEN -1 AND 1
ORDER BY Invoice_date;
It's worth noting that the above sql is completely untested. I do not have access to your server, or data, so I'm purely relying on "my eye" to note any errors.
Question:
I get these results (column task_externalId) when I run my query:
TC229090-10000-3
TC229090-20000-3
TC229830-10000-3
TC229685-10000-3
A task (TC229090) can contain multiple rows (10000, 20000), but can also have just one (10000 only).
My full code:
SELECT
v.voorgemeld_handmatig, v.voorgemeld, v.mac, v.orn, v.cdm, v.blo,
v.bco, v.tarcode, (v.timeslotfrom + ' - ' + v.timeslottill),
cast (case
when
(Select ac2.actionSpecificationName
From [COMTECdefault].[dbo].[task] t2
Join [COMTECdefault].[dbo].[actionKind] ak2 On t2.id_actionKind = ak2.id_actionKind
Join [COMTECdefault].[dbo].[actionSpecification] ac2 On ak2.id_actionSpecification = ac2.id_actionSpecification
where t2.task_externalId = (left(t.task_externalId, 15) + '2')) = 'laden'
then 'Vol'
else 'Leeg'
end as text) as IMPORT,
t.task_externalId, a.addressName, a.cityName,
CONVERT(DATE, t.from_date) as [Date],
CONVERT(varchar(8), CONVERT(TIME, t.from_date)) as [Tijd],
res.resourceName, resk.resourceKindName, r.shipOwner,
t.reference,
(Select resourceName
From [COMTECdefault].[dbo].[resource] r5
Left Outer Join [COMTECdefault].[dbo].resourceKind rk5 On rk5.id_resourceKind = r5.id_resourceKind
Where (id_resource = SUBSTRING(pt.StartResources, 0, CHARINDEX(',', pt.StartResources))
or id_resource = PARSENAME(REPLACE(SUBSTRING(pt.StartResources, CHARINDEX(',', pt.StartResources) + 1, LEN(pt.StartResources)), ',', '.'), 4)
or id_resource = PARSENAME(REPLACE(SUBSTRING(pt.StartResources, CHARINDEX(',', pt.StartResources) + 1, LEN(pt.StartResources)), ',', '.'), 3)
or id_resource = PARSENAME(REPLACE(SUBSTRING(pt.StartResources, CHARINDEX(',', pt.StartResources) + 1, LEN(pt.StartResources)), ',', '.'), 2)
or id_resource = PARSENAME(REPLACE(SUBSTRING(pt.StartResources, CHARINDEX(',', pt.StartResources) + 1, LEN(pt.StartResources)), ',', '.'), 1))
and r5.id_resourceKind = 51),
udo.udf_Gewicht, udo.udf_Zegelnummer,
cast (case when a.addressName = 'APM Terminal 2'
then case when udo.udf_Zegelnummer IS NULL
then 0
else 1
end
when a.addressName IN ('ect delta', 'Euromax Terminam C.V.', 'Euromax Terminal')
then case when udo.udf_Gewicht = 0
then 0
else 1
end
else 1
end as bit) as [voormelden],
v.foutcode, v.foutcode_tekst, v.V_door
FROM
[COMTECdefault].[dbo].[task] t
JOIN
[COMTECdefault].[dbo].[actionKind] ak ON t.id_actionKind = ak.id_actionKind
JOIN
[COMTECdefault].[dbo].[actionSpecification] ac ON ak.id_actionSpecification = ac.id_actionSpecification
LEFT OUTER JOIN
[COMTECdefault].[dbo].[resourceOrder] ro ON t.id_order = ro.id_order
LEFT OUTER JOIN
[COMTECdefault].[dbo].[resource] res ON ro.id_resource = res.id_resource
LEFT OUTER JOIN
[COMTECdefault].[dbo].[resourceKind] resk ON res.id_resourceKind= resk.id_resourceKind
LEFT OUTER JOIN
[COMTECdefault].[dbo].[address] a ON t.id_address = a.id_address
LEFT OUTER JOIN
[COMTECdefault].[dbo].[resourceOrder] r ON t.id_order = r.id_order
LEFT OUTER JOIN
[COMTECdefault].[dbo].[order] o ON o.id_order = r.id_order
LEFT OUTER JOIN
[COMTECdefault].[dbo].[voormelden] v ON v.id_task_otd = t.task_externalId
LEFT OUTER JOIN
[COMTECdefault].[dbo].[ud_order] udo ON udo.id_order = t.id_order
LEFT OUTER JOIN
[COMTECdefault].[dbo].[plannedTask] pt on t.id_task = pt.id_task
WHERE
(pt.id_Task IS NULL OR pt.taskstate <> 'finished')
AND (v.canceled IS NULL OR v.canceled = 0)
AND left(res.resourceName, 4) <> 'XXXU'
AND t.canceled = 0
AND (t.from_date <= '2016-11-24 23:59:59')
AND (ac.actionSpecificationName = 'inleveren')
AND a.addressName IN ('APM Terminal 1','APM Terminal 2', 'ect delta', 'rwg', 'Euromax Terminam C.V.', 'Euromax Terminal')
ORDER BY
res.resourceName, a.addressName, t.task_externalId
I only want to show the 20000 task when its exists . Otherwise show the 10000.
Thanks in advance.
I am not going to touch all of that text there may well be was of narrowing down all of the joins etc, but here is the pattern for doing what you want with the task_exteralid column.
DECLARE #Table AS TABLE (task_externalid VARCHAR(100))
INSERT INTO #Table VALUES
('TC229090-10000-3')
,('TC229090-20000-3')
,('TC229830-10000-3')
,('TC229685-10000-3')
;WITH cte AS (
SELECT
*
,RowNumber = ROW_NUMBER() OVER (PARTITION BY
LEFT(task_externalid,CHARINDEX('-',task_externalid) - 1)
ORDER BY task_externalid DESC)
FROM
#Table
)
SELECT *
FROM
cte
WHERE
RowNumber = 1
Create a partitioned ROW_NUMBER on the LEFT most portion of the task_exteralid
Order that by task_exteralid DESC
When I try to run a huge query over Amazon Redshift I get the following error:
"m_num_steps < MAX_STEPS - exceeded maximum plan steps of 100"
There are 4 tables at Redshift. The fact table has about 400 million rows. The query is a "WITH" clause with multiple joins, agregations, analytic and window functions.
The question is: there is a way to avoid this "MAX_STEPS = 100" limitation?
The query (simplified):
WITH
-- Regr
a27 as (
select id_uc, case when count(*) = 0 or VAR_POP(numero_de_meses) = 0 then null else trunc(((SUM(consumo_normalizado * numero_de_meses) - SUM(numero_de_meses) * SUM(consumo_normalizado) / count(*)) / count(*)) / VAR_POP(numero_de_meses), 2) end N2542, case when count(*) = 0 or STDDEV_POP(consumo_normalizado) = 0 or STDDEV_POP(numero_de_meses) = 0 then null else trunc(POWER(((SUM(consumo_normalizado * numero_de_meses) - SUM(numero_de_meses) * SUM(consumo_normalizado) / count(*)) / count(*)) / (STDDEV_POP(consumo_normalizado) * STDDEV_POP(numero_de_meses)), 2), 2) end N2554 from (
select id_uc,
rank() over (partition by canal.id_uc order by canal.data_referencia) numero_de_meses,
case
when stddev(consumo_faturado) over (partition by canal.id_uc) <> 0
then (consumo_faturado - avg(consumo_faturado) over (partition by canal.id_uc)) / stddev(consumo_faturado) over (partition by canal.id_uc)
else null end as consumo_normalizado
from fato_uc canal
where exists (select 1 from eventos_resultado_atual evt where canal.id_uc = evt.id_uc and canal.data_referencia between E_Fat_12_meses_atras and ultimo_faturamento)
) group by id_uc)
-- Regr
,a28 as (
select id_uc, case when count(*) = 0 or VAR_POP(numero_de_meses) = 0 then null else case when count(*) = 0 or VAR_POP(numero_de_meses) = 0 then null else trunc(((SUM(consumo_normalizado * numero_de_meses) - SUM(numero_de_meses) * SUM(consumo_normalizado) / count(*)) / count(*)) / VAR_POP(numero_de_meses), 2) end end N2545, case when count(*) = 0 or STDDEV_POP(consumo_normalizado) = 0 or STDDEV_POP(numero_de_meses) = 0 then null else trunc(POWER(((SUM(consumo_normalizado * numero_de_meses) - SUM(numero_de_meses) * SUM(consumo_normalizado) / count(*)) / count(*)) / (STDDEV_POP(consumo_normalizado) * STDDEV_POP(numero_de_meses)), 2), 2) end N2557 from (
select id_uc,
rank() over (partition by canal.id_uc order by canal.data_referencia) numero_de_meses,
case
when stddev(consumo_faturado) over (partition by canal.id_uc) <> 0
then (consumo_faturado - avg(consumo_faturado) over (partition by canal.id_uc)) / stddev(consumo_faturado) over (partition by canal.id_uc)
else null end as consumo_normalizado
from fato_uc canal
where exists (select 1 from eventos_resultado_atual evt where canal.id_uc = evt.id_uc and canal.data_referencia between E_Fat_12_meses_atras and E_2457)
) group by id_uc)
-- Corr
,a65 as (
select evt.id_uc, case when count(*) = 0 or STDDEV_POP(c1.avg_cons) = 0 or STDDEV_POP(c2.consumo_faturado) = 0 then null else trunc(((SUM(c1.avg_cons * c2.consumo_faturado) - SUM(c2.consumo_faturado) * SUM(c1.avg_cons) / count(*)) / count(*)) / (STDDEV_POP(c1.avg_cons) * STDDEV_POP(c2.consumo_faturado)), 2) end N2720
from eventos_resultado_atual evt inner join VIZ_FAT c1 on evt.id_uc = c1.id_uc
inner join fato_uc c2 on evt.id_uc = c2.id_uc and date_trunc('month', c1.mes_ref) = date_trunc('month', c2.data_referencia)
where c1.mes_ref between E_Fat_12_meses_atras and ultimo_faturamento
and c2.data_referencia between E_Fat_12_meses_atras and ultimo_faturamento
group by evt.id_uc)
-- Corr
,a66 as (
select evt.id_uc, case when count(*) = 0 or STDDEV_POP(c1.avg_cons) = 0 or STDDEV_POP(c2.consumo_faturado) = 0 then null else trunc(((SUM(c1.avg_cons * c2.consumo_faturado) - SUM(c2.consumo_faturado) * SUM(c1.avg_cons) / count(*)) / count(*)) / (STDDEV_POP(c1.avg_cons) * STDDEV_POP(c2.consumo_faturado)), 2) end N2723
from eventos_resultado_atual evt inner join VIZ_FAT c1 on evt.id_uc = c1.id_uc
inner join fato_uc c2 on evt.id_uc = c2.id_uc and date_trunc('month', c1.mes_ref) = date_trunc('month', c2.data_referencia)
where c1.mes_ref between E_Fat_12_meses_atras and E_2457
and c2.data_referencia between E_Fat_12_meses_atras and E_2457
group by evt.id_uc)
SELECT *
FROM eventos_resultado_atual
left join a27 on eventos_resultado_atual.id_uc = a27.id_uc
left join a28 on eventos_resultado_atual.id_uc = a28.id_uc
left join a65 on eventos_resultado_atual.id_uc = a65.id_uc
left join a66 on eventos_resultado_atual.id_uc = a66.id_uc
Your problem is with redshift and as always amazon have no solution,try breaking the query in little tasks, be careful with working with lot of data in redshift have serious issues with big data
My recommendation is to break your query into a few sub queries and insert the results into tables along the process. Please share query and I will help you break it up into a few queries that will run more efficient. If you can also share the redshift provided explain plain I I can go over how to use that to do what I would do...
Here's my Count_query:
Declare #yes_count decimal;
Declare #no_count decimal;
set #yes_count=(Select count(*) from Master_Data where Received_Data='Yes');
set #no_count=(Select count(*) from Master_Data where Received_Data='No');
select #yes_count As Yes_Count,#no_count as No_Count,(#yes_count/(#yes_count+#no_count)) As Submission_Count
I am having trouble making joins on these two queries
This is the rest of the query:
Select Distinct D.Member_Id,d.Name,d.Region_Name, D.Domain,e.Goal_Abbreviation,
e.Received_Data, case when Received_Data = 'Service Not Provided' then null
when Received_Data = 'No' then null else e.Improvement end as
Percent_Improvement , case when Received_Data = 'Service Not Provided' then null
when Received_Data = 'No' then null else e.Met_40_20 end as Met_40_20
FROM (
select distinct member_Domains.*,
(case when NoData.Member_Id is null then 'Participating' else ' ' end) as Participating
from
(
select distinct members.Member_Id, members.Name, Members.Region_Name,
case when Domains.Goal_Abbreviation = 'EED Reduction' then 'EED'
When Domains.Goal_Abbreviation = 'Pressure Ulcers' then 'PRU'
when Domains.Goal_Abbreviation = 'Readmissions' then 'READ' else Domains.Goal_Abbreviation end as Domain from
(select g.* from Program_Structure as ps inner join Goal as g on ps.Goal_Id = g.Goal_Id
and ps.Parent_Goal_ID = 0) as Domains
cross join
(select distinct hc.Member_ID, hc.Name,hc.Region_Name from zsheet as z
inner join Hospital_Customers$ as hc on z.CCN = hc.Mcare_Id) as Members
) as member_Domains
left outer join Z_Values_Hospitals as NoData on member_Domains.member_ID = NoData.Member_Id
and Member_Domains.Domain = noData.ReportName) D
Left Outer JOIN
(SELECT B.Member_ID, B.Goal_Abbreviation, B.minRate, C.maxRate, B.BLine, C.Curr_Quarter, B.S_Domain,
(CASE WHEN B.Member_ID IN
(SELECT member_id
FROM Null_Report
WHERE ReportName = B.S_Domain) THEN 'Service Not Provided' WHEN Curr_Quarter = 240 THEN 'Yes' ELSE 'No' END) AS Received_Data,
ROUND((CASE WHEN minRate = 0 AND maxRate = 0 THEN 0 WHEN minRate = 0 AND maxRate > 0 THEN 1 ELSE (((maxRate - minRate) / minRate) * 100) END), .2) AS Improvement,
(CASE WHEN ((CASE WHEN minRate = 0 AND maxRate = 0 THEN 0 WHEN minRate = 0 AND maxRate > 0 THEN 1 ELSE (maxRate - minRate) / minRate END)) <= - 0.4 OR
maxRate = 0 THEN 'Yes' WHEN ((CASE WHEN minRate = 0 AND maxRate = 0 THEN 0 WHEN minRate = 0 AND maxRate > 0 THEN 1 ELSE (maxRate - minRate) / minRate END))
<= - 0.2 OR maxRate = 0 THEN 'Yes' ELSE 'No' END) AS Met_40_20
FROM (SELECT tab.Member_ID, tab.Measure_Value AS minRate, tab.Goal_Abbreviation, A.BLine, tab.S_Domain
FROM Measure_Table_Description AS tab INNER JOIN
(SELECT DISTINCT
Member_ID AS new_memid, Goal_Abbreviation AS new_measure, MIN(Reporting_Period_ID) AS BLine, MAX(Reporting_Period_ID)
AS Curr_Quarter
FROM Measure_Table_Description
WHERE (Member_ID > 1) AND (Measure_Value IS NOT NULL) AND (Measure_ID LIKE '%O%')
GROUP BY Goal_Abbreviation, Member_ID) AS A ON tab.Member_ID = A.new_memid AND tab.Reporting_Period_ID = A.BLine AND
tab.Goal_Abbreviation = A.new_measure) AS B FULL OUTER JOIN
(SELECT tab.Member_ID, tab.Measure_Value AS maxRate, tab.Goal_Abbreviation, A_1.Curr_Quarter
FROM Measure_Table_Description AS tab INNER JOIN
(SELECT DISTINCT
Member_ID AS new_memid, Goal_Abbreviation AS new_measure,
MIN(Reporting_Period_ID) AS BLine, MAX(Reporting_Period_ID)
AS Curr_Quarter
FROM Measure_Table_Description AS Measure_Table_Description_1
WHERE (Member_ID >1) AND (Measure_Value IS NOT NULL) AND (Measure_ID LIKE '%O%')
GROUP BY Goal_Abbreviation, Member_ID) AS A_1 ON tab.Member_ID = A_1.new_memid
AND tab.Reporting_Period_ID = A_1.Curr_Quarter AND
tab.Goal_Abbreviation = A_1.new_measure) AS C ON B.Member_ID = C.Member_ID
WHERE (B.Goal_Abbreviation = C.Goal_Abbreviation) ) E ON D.Member_Id = E.Member_ID AND d.Domain = E.S_Domain
ORDER BY D.Domain,D.Member_ID
How do I get a count of the 'yes'/ (count(yes)+count(no)) for each member_ID as column1 and also display the rank of each member_ID against all the member_IDs in the result as column2. I have come up with a query that generates the count for the entire table, but how do I restrict it each Member_ID.
Thanks for your help.
I haven't taken the time to digest your provided query, but if abstracted to the concept of having an aggregate over a range of data repeated on each row, you should look at using windowing functions. There are other methods, such as using a CTE to do your aggregation and then JOINing back to your detailed data. That might work better for more complex calculations, but the window functions are arguably the more elegant option.
DECLARE #MasterData AS TABLE
(
MemberID varchar(50),
MemberAnswer int
);
INSERT INTO #MasterData (MemberID, MemberAnswer) VALUES ('Jim', 1);
INSERT INTO #MasterData (MemberID, MemberAnswer) VALUES ('Jim', 0);
INSERT INTO #MasterData (MemberID, MemberAnswer) VALUES ('Jim', 1);
INSERT INTO #MasterData (MemberID, MemberAnswer) VALUES ('Jim', 1);
INSERT INTO #MasterData (MemberID, MemberAnswer) VALUES ('Jane', 1);
INSERT INTO #MasterData (MemberID, MemberAnswer) VALUES ('Jane', 0);
INSERT INTO #MasterData (MemberID, MemberAnswer) VALUES ('Jane', 1);
-- Method 1, using windowing functions (preferred for performance and syntactical compactness)
SELECT
MemberID,
MemberAnswer,
CONVERT(numeric(19,4),SUM(MemberAnswer) OVER (PARTITION BY MemberID)) / CONVERT(numeric(19,4),COUNT(MemberAnswer) OVER (PARTITION BY MemberID)) AS PercentYes
FROM #MasterData;
-- Method 2, using a CTE
WITH MemberSummary AS
(
SELECT
MemberID,
SUM(MemberAnswer) AS MemberYes,
COUNT(MemberAnswer) AS MemberTotal
FROM #MasterData
GROUP BY MemberID
)
SELECT
md.MemberID,
md.MemberAnswer,
CONVERT(numeric(19,4),MemberYes) / CONVERT(numeric(19,4),MemberTotal) AS PercentYes
FROM #MasterData md
JOIN MemberSummary ms
ON md.MemberID = ms.MemberID;
First thought is: your query is much, much too complicated. I have spent about 10 minutes now trying to make sense of it and haven't gotten anywhere, so it's obviously going to pose a long-term maintenance challenge to those within your organization going forward as well. I would really recommend you try to find some way of simplifying it.
That said, here is a simplified, general example of how to query on a calculated value and rank the results:
CREATE TABLE member (member_id INT PRIMARY KEY);
CREATE TABLE master_data (
transaction_id INT PRIMARY KEY,
member_id INT FOREIGN KEY REFERENCES member(member_id),
received_data BIT
);
-- INSERT data here
; WITH member_data_counts AS (
SELECT
m.member_id,
(SELECT COUNT(*) FROM master_data d WHERE d.member_id = m.member_id AND d.received_data = 1) num_yes,
(SELECT COUNT(*) FROM master_data d WHERE d.member_id = m.member_id AND d.received_data = 0) num_no
FROM member m
), member_data_calc AS (
SELECT
*,
CASE
WHEN (num_yes + num_no) = 0 THEN NULL -- avoid division-by-zero error
ELSE num_yes / (num_yes + num_no)
END pct_yes
FROM member_data_counts
), member_data_rank AS (
SELECT *, RANK() OVER (ORDER BY pct_yes DESC) AS RankValue
FROM member_data_calc
)
SELECT *
FROM member_data_rank
ORDER BY RankValue ASC;
I would like to use the code below to group results by OrgName but rollup the count using TotalCount
Results would look similar to the screenshot below:(sorry I had to redact information due to confidentiality agreement).
The Orgname is grouped correctly. However, I am having a little troubling rolling up totalCount.
Thanks
WITH
ctePreAgg AS
(
SELECT (ElectionName + ' - ' + CAST(ClosingDate AS VARCHAR(12))) electionName, CASE WHEN Position='Member' THEN '' ELSE Position END As Position, CASE WHEN c.CurrentOfficeHolder='Incumbent' THEN CandidateName + '('+ c.CurrentOfficeHolder + ')' ELSE CandidateName END As CandidateName , c.PositionId,COUNT(*) TotalVotes
FROM Candidates c
JOIN Positions p ON c.PositionId = p.PositionId
JOIN Elections e on c.ElectionId = e.ElectionId
WHERE c.ElectionId IN (1,2,3)
GROUP BY Position, CandidateId, CandidateName,c.PositionId,CurrentOfficeHolder,AnswerType,ElectionName, ClosingDate
)
SELECT [OrgName] = CASE WHEN GROUPING(mh.PositionId) = 0 THEN MAX(mh.Position) ELSE mh.ElectionName END
,Names = CASE WHEN GROUPING(mh.PositionId) = 0 THEN MAX(mh.CandidateName) ELSE '' END
,PositionId = CASE WHEN GROUPING(mh.PositionId) = 0 THEN mh.PositionId ELSE '' END
,TotalCount = CASE WHEN GROUPING(mh.PositionId) = 0 THEN COUNT(*) ELSE '' END
FROM ctePreAgg mh
GROUP BY ElectionName,PositionId WITH ROLLUP
HAVING GROUPING(mh.ElectionName) = 0
ORDER BY mh.ElectionName, GROUPING(mh.PositionId) DESC, mh.PositionID;