I'm trying to replace NULL values with 0 in the following query, however without luck. I know I need to create another column list for the SELECT statement, but I've been unsuccessful so far.
How do I do this properly?
My query:
DECLARE #SQL AS NVARCHAR(MAX);
DECLARE #COLS AS NVARCHAR(MAX);
SELECT #COLS = ISNULL(#COLS + ',', '') + QUOTENAME(ItemCategoryName)
FROM
(
SELECT DISTINCT
ItemCategoryName
FROM sgdw.fact.RetailSales RS
JOIN SGDW.dim.Item I ON RS.ItemID = I.ItemID
WHERE RS.CalendarID >= '20190101'
AND storeid = '92'
AND DATALENGTH(ItemCategoryName) != 0
) AS Properties;
SET #SQL = N'SELECT *
FROM
(
SELECT RetailSalesAmountIncludingVATDKK,
ItemCategoryName,
ReceiptCode
FROM
(
SELECT *
FROM sgdw.fact.RetailSales
WHERE storeid = ''92''
UNION ALL
SELECT *
FROM sgdw.fact.RetailSales_hist
WHERE CalendarID >= ''20190101''
AND storeid = ''92''
) RS
JOIN SGDW.dim.Item I ON RS.ItemID = I.ItemID
JOIN SGDW.dim.Store S ON S.StoreID = RS.StoreID
) SourceTable
PIVOT (Sum(RetailSalesAmountIncludingVATDKK) FOR [ItemCategoryName] IN (' + #COLS + ')) as PivotTable';
EXEC sp_executesql
#SQL;
If you use SQL Server (starting with 2008), Azure SQL Database, Azure SQL Data Warehouse, Parallel Data Warehousen you can bye COALESCE method
like
SELECT #COLS_SUM = #COLS_SUM + 'COALESCE(' + QUOTENAME(finmonth) + ',0)+'
FROM (SELECT DISTINCT finmonth FROM YOUR_TABLE ) AS tmp
SELECT #COLS_SUM = ','+ SUBSTRING(#COLS_SUM, 0, LEN(#COLS_SUM)) +' AS TOTAL'
for more info check https://www.codeproject.com/Questions/1391827/Dealing-with-nulls-dynamic-columns-in-pivot
Append This
SELECT #COLS = 'ISNULL('+REPLACE(#COLS,',',',0 ) , ISNULL(') + ',0 )'
After this block
SELECT #COLS = ISNULL(#COLS + ',', '') + QUOTENAME(ItemCategoryName)
FROM
(
SELECT DISTINCT
ItemCategoryName
FROM sgdw.fact.RetailSales RS
JOIN SGDW.dim.Item I ON RS.ItemID = I.ItemID
WHERE RS.CalendarID >= '20190101'
AND storeid = '92'
AND DATALENGTH(ItemCategoryName) != 0
) AS Properties;
Related
I was tasked to validate the decimal and integer values of the columns from a list of tables. I have around 10-12 tables having different column names.
I created a lookup table which has the table name and the column names of decimal and integer as shown below. for example 'Pricedetails' and 'Itemdetails' tables have many columns of which only the ones mentioned in the Lookup table are required.
lkpTable
TableName
requiredcolumns
Pricedetails
sellingPrice,RetailPrice,Wholesaleprice
Itemdetails
ItemID,Itemprice
Pricedetails
Priceid
Mafdate
MafName
sellingPrice
RetailPrice
Wholesaleprice
01
2020-01-01
Americas
25.00
43.33
33.66
02
2020-01-01
Americas
43.45
22.55
11.11
03
2021-01-01
Asia
-23.00
-34.00
23.00
Itemdetails
ItemID
ItemPrice
Itemlocation
ItemManuf
01
45.11
Americas
SA
02
25.00
Americas
SA
03
35.67
Americas
SA
I have created a stored procedure with table name as input parameter, and able to pull the required column names of the tables (input parameter) from the lookup table and store that resultset into a table variable, below is the code.
declare #resultset Table
(
id INT identity(1,1),
tablename varchar(200) ,
ColumnNames varchar(max)
)
declare #tblname varchar(200),#sql varchar(max),#cols varchar(max),
INSERT INTO #resultset
select tablename,ColumnNames
from lkptable where tablename ='itemdetails'
select #cols = ColumnNames from #resultset;
select #tblname = TableName from #resultset;
----- Split the comma separated columnnames
Create table ##splitcols
(
ID int identity(1,1),
Name varchar(50)
)
Insert into ##splitcols
select value from string_split(#cols,',')
set #sql = 'select ' +#cols + ' from ' +#tblname
--print (#cols)
exec (#sql)
select * from ##splitcols
On executing the above code i get the below result sets, similarly what ever table name i provide i can get the required columns and its relevant data, now i am stuck at this point on how to validate whether the columns are decimal or int. I tried using while loop and cursor to pass the Name value from Resultset2, to the new dynamic query, somehow i don't find any way on how to validate.
ItemID
ItemPrice
01
45.11
02
25.00
03
35.67
Resultset2
ID
Name
01
ItemID
02
ItemPrice
you can validate in this way
insert into #splitcols values (1.1),(1.11)
select case when (t * 100)%10 = 0 then 1 else 0 end as valid from #splitcols
Similarly for number/integer
You can generate one giant UNION ALL query using dynamic SQL, then run it.
Each query would be of the form:
SELECT
TableName = 'SomeTable',
ColumnName,
IsInt
FROM (
SELECT
[Column1] = CASE WHEN COUNT(CASE WHEN ROUND([Column1], 0) <> [Column1] THEN 1 END) = 0 THEN 'All ints' ELSE 'Not All ints' END,
[Column2] = CASE WHEN COUNT(CASE WHEN ROUND([Column2], 0) <> [Column2] THEN 1 END) = 0 THEN 'All ints' ELSE 'Not All ints' END
FROM SomeTable
) t
UNPIVOT (
ColumnName FOR IsInt IN (
[Column1], [Column2]
)
) u
The script is as follows
DECLARE #sql nvarchar(max);
SELECT #sql = STRING_AGG('
SELECT
TableName = ' + QUOTENAME(t.name, '''') + ',
ColumnName,
IsInt
FROM (
SELECT ' + c.BeforePivotCoumns + '
FROM ' + QUOTENAME(t.name) + '
) t
UNPIVOT (
ColumnName FOR IsInt IN (
' + c.UnpivotColumns + '
)
) u
', '
UNION ALL '
)
FROM sys.tables t
JOIN lkpTable lkp ON lkp.TableName = t.name
CROSS APPLY (
SELECT
BeforePivotCoumns = STRING_AGG(CAST('
' + QUOTENAME(c.name) + ' = CASE WHEN COUNT(CASE WHEN ROUND(' + QUOTENAME(c.name) + ', 0) <> ' + QUOTENAME(c.name) + ' THEN 1 END) = 0 THEN ''All ints'' ELSE ''Not All ints'' END'
AS nvarchar(max)), ','),
UnpivotColumns = STRING_AGG(QUOTENAME(c.name), ', ')
FROM sys.columns c
JOIN STRING_SPLIT(lkp.requiredcolumns, ',') req ON req.value = c.name
WHERE c.object_id = t.object_id
) c;
PRINT #sql;
EXEC sp_executesql #sql;
db<>fiddle
If you are on an older version of SQL Server then you can't use STRING_AGG and instead you need to hack it with FOR XML and STUFF.
DECLARE #unionall nvarchar(100) = '
UNION ALL ';
DECLARE #sql nvarchar(max);
SET #sql = STUFF(
(SELECT #unionall + '
SELECT
TableName = ' + QUOTENAME(t.name, '''') + ',
ColumnName,
IsInt
FROM (
SELECT ' + STUFF(c1.BeforePivotCoumns.value('text()[1]','nvarchar(max)'), 1, 1, '') + '
FROM ' + QUOTENAME(t.name) + '
) t
UNPIVOT (
ColumnName FOR IsInt IN (
' + STUFF(c2.UnpivotColumns.value('text()[1]','nvarchar(max)'), 1, 1, '') + '
)
) u'
FROM sys.tables t
JOIN (
SELECT DISTINCT
lkp.TableName
FROM lkpTable lkp
) lkp ON lkp.TableName = t.name
CROSS APPLY (
SELECT
',
' + QUOTENAME(c.name) + ' = CASE WHEN COUNT(CASE WHEN ROUND(' + QUOTENAME(c.name) + ', 0) <> ' + QUOTENAME(c.name) + ' THEN 1 END) = 0 THEN ''All ints'' ELSE ''Not All ints'' END'
FROM lkpTable lkp2
CROSS APPLY STRING_SPLIT(lkp2.requiredcolumns, ',') req
JOIN sys.columns c ON req.value = c.name
WHERE c.object_id = t.object_id
AND lkp2.TableName = lkp.TableName
FOR XML PATH(''), TYPE
) c1(BeforePivotCoumns)
CROSS APPLY (
SELECT
', ' + QUOTENAME(c.name)
FROM lkpTable lkp2
CROSS APPLY STRING_SPLIT(lkp2.requiredcolumns, ',') req
JOIN sys.columns c ON req.value = c.name
WHERE c.object_id = t.object_id
AND lkp2.TableName = lkp.TableName
FOR XML PATH(''), TYPE
) c2(UnpivotColumns)
FOR XML PATH(''), TYPE
).value('text()[1]','nvarchar(max)'), 1, LEN(#unionall), '');
PRINT #sql;
EXEC sp_executesql #sql;
db<>fiddle
I have two tables whose structures as follows:
table_A
CREATE TABLE table_A
(
col_a varchar(100),
col_b bigint,
col_c datetime
)
table_b
--Note that columns are same--
CREATE TABLE table_B
(
col_a varchar(10),
col_b varchar(10),
col_c varchar(20)
)
Now I want to INSERT data into table_A from table_B with proper data type conversion.
Below is the SQL string:
INSERT INTO table_A(col_a,col_b,col_c)
SELECT CONVERT(varchar,col_a),CONVERT(INT,col_b),CONVERT(datetime,col_c) FROM table_B
So far so good.
Now I want generate the SQL dynamically with the help of INFORMATION_SCHEMA.COLUMNS.
For this I have followed the below steps:
Step 1:
Join the Information Schema for the above two tables viz table_A and table_B and store them in a #TempTable. Lets assume that #TempTable has an ID column that is IDENTITY(1,1) but that doesn't follow any sequence like 1,2,3...(Typically this happens in Synapse SQL)
INSERT INTO #TempTable
SELECT S.COLUMN_NAME AS Src_Col,
S.DATA_TYPE AS Src_dtype,
D.COLUMN_NAME AS Dest_Col,
D.DATA_TYPE AS Dest_dtype,
CASE WHEN S.DATA_TYPE NOT LIKE D.DATA_TYPE THEN
'CONVERT('+ '''' + D.DATA_TYPE + '''' + ',' + '''' + S.DATA_TYPE + '''' + ')'
ELSE S.DATA_TYPE AS Modified_Col
FROM INFORMATION_SCHEMA S
JOIN INFORMATION_SCHEMA.COLUMNS D
ON S.COLUMN_NAME = D.COLUMN_NAME AND S.TABLE_NAME = REPLACE(D.TABLE_NAME,'_B','_A')
Step 2:
Iterate over #TempTable to fetch the Modified_Col values
SET #Max_ID = (SELECT MAX(ID) FROM #TempTable);
SET #Min_ID = (SELECT MIN(ID) FROM #TempTable);
SET #ColToInsert = '';
SET #Dest_Col = '';
WHILE #Min_ID <= #Max_ID
BEGIN
SET #ColToInsert = (SELECT #ColToInsert + Modified_Col FROM #TempTable T WHERE T.ID = #Min_ID);
SET #Dest_Col = (SELECT #Dest_Col + Dest_Col FROM #TempTable T WHERE T.ID = #Min_ID);
SET #Min_ID = #Min_ID + 1;
END
Step 3:
Use that #ColToInsert in the below Dynamic SQL
SET #DySQL = 'INSERT INTO Table_A(' + #Dest_Col + ') SELECT ' + #ColToInsert + ' FROM table_B';
exec (#DySQL);
Now at this step 3 I am not getting the expected result. No data is getting inserted into table_A. I can understand that in the CASE statement I have to make some fixes so that convert... portion becomes a string. And I am not able to do so.
Any clue would be appreciated.
I don't understand why you need the temp table at all. You just need to aggregate using STRING_AGG.
You also need to quote the objects and columns using QUOTENAME, and you should use sys.columns etc rather than INFORMATION_SCHEMA, which is for compatibility only.
DECLARE #tableA sysname = 't';
DECLARE #tableB sysname = 's';
DECLARE #sql nvarchar(max) = (
SELECT CONCAT(
'INSERT INTO ',
QUOTENAME(#tableA),
'(',
STRING_AGG(CAST(QUOTENAME(cA.name) AS nvarchar(max)), ', '),
')
SELECT ',
STRING_AGG(
CASE WHEN cA.user_type_id <> cB.user_type_id THEN
CONCAT(
'CONVERT(',
typ.name,
CASE
WHEN typ.name IN ('varchar','nvarchar','char','nchar','varbinary','binary')
THEN CONCAT('(', CASE WHEN cA.max_length = -1 THEN 'max' END, NULLIF(cA.max_length, -1), ')')
WHEN typ.name IN ('datetime2','datetimeoffset','time','float','real')
THEN CONCAT('(', cA.scale, ')')
WHEN typ.name IN ('float','real')
THEN CONCAT('(', cA.precision, ')')
WHEN typ.name IN ('decimal','numeric')
THEN CONCAT('(', cA.precision, ',', cA.scale, ')')
END,
', ',
CAST(QUOTENAME(cB.name) AS nvarchar(max)),
')'
)
ELSE
CAST(QUOTENAME(cB.name) AS nvarchar(max))
END
, ', '),
'
FROM ',
QUOTENAME(#tableB)
)
FROM sys.columns cA
JOIN sys.tables tA ON ta.object_id = cA.object_id AND tA.name = #tableA
JOIN sys.types typ ON typ.user_type_id = cA.user_type_id
JOIN sys.columns cB ON cB.name = cA.name
JOIN sys.tables tB ON tB.object_id = cB.object_id AND tB.name = #tableB
);
PRINT #sql; -- your friend
EXEC sp_executesql #sql;
db<>fiddle
I would like to convert row data into columns, where the column names are not from the data. I think that using Pivot will not give me the correct solution. Please see Image of what my data looks like and how I want it to look.
The number of rows returned in the example will continue to grow over time.
My solution :
Based on #Triv answered, I have managed to solve the problem using the rank function to create a new column and then using dynamic pivot SQL to transform the data.
You could use a dynamic sql + PIVOT
CREATE TABLE #SampleData
(
AccountNumber int,
Product varchar(20),
ProductEndDate datetime
)
INSERT INTO #SampleData VALUES (1,'Fixed 10','2016-01-01'),(1,'Fixed 11','2016-02-01'),(1,'Fixed 13','2016-03-01'),(1,'Fixed 12','2016-04-01'),
(2,'Fixed 10','2016-01-01'),(2,'Fixed 11','2016-02-01'),(2,'Fixed 13','2016-03-01')
DECLARE #HeaderAll nvarchar(max)
DECLARE #ColumnPivotProduct nvarchar(max)
DECLARE #ColumnPivotProductEndDate nvarchar(max)
;WITH temp AS
(
SELECT DISTINCT
CONCAT('Product' ,row_number() OVER(PARTITION BY sd.AccountNumber ORDER BY sd.Product)) AS ProductGroup,
CONCAT('ProductEndDate' ,row_number() OVER(PARTITION BY sd.AccountNumber ORDER BY sd.Product)) AS ProductEndDateGroup
FROM #SampleData sd
)
SELECT #HeaderAll = STUFF((SELECT CONCAT(',',t.ProductGroup,'= MAX(',t.ProductGroup, '),', t.ProductEndDateGroup ,'= MAX(', t.ProductEndDateGroup,')') FROM temp t FOR XML PATH('')), 1,1,''),
#ColumnPivotProduct = STUFF((SELECT CONCAT(',',t.ProductGroup) FROM temp t FOR XML PATH('')), 1,1,''),
#ColumnPivotProductEndDate = STUFF((SELECT CONCAT(',', t.ProductEndDateGroup) FROM temp t FOR XML PATH('')), 1,1,'')
--SELECT #HeaderAll, #ColumnPivotProduct, #ColumnPivotProductEndDate
DECLARE #query nvarchar(max) = CONCAT(
';WITH temp AS
(
SELECT *,
CONCAT(''Product'' ,row_number() OVER(PARTITION BY sd.AccountNumber ORDER BY sd.Product)) AS ProductGroup,
CONCAT(''ProductEndDate'' ,row_number() OVER(PARTITION BY sd.AccountNumber ORDER BY sd.Product)) AS ProductEndDateGroup
FROM #SampleData sd
)
SELECT AccountNumber, ',#HeaderAll,' FROM
(
SELECT t.AccountNumber, t.Product, t.ProductEndDate, t.ProductGroup,t.ProductEndDateGroup FROM temp t
) src
PIVOT
(
MIN(Product) FOR ProductGroup IN (',#ColumnPivotProduct,')
) pvt
PIVOT
(
MIN(ProductEndDate) FOR ProductEndDateGroup IN (',#ColumnPivotProductEndDate,')
) pvt1
GROUP BY AccountNumber
')
PRINT #query
exec(#query)
DROP TABLE #SampleData
Demo link: http://rextester.com/AEQBZ56634
Note: CONCAT is avaiable in sql-server 2012+. IF you are using an older version, then use + for concatenating string
It works, check this out, Instead of #testTable use your table name
declare #temptable varchar(1000) = 'declare #tempTable1 table (',
#inserStatement varchar(1000) = 'insert into #tempTable1 (',
#insertValues varchar(1000) = ''
DECLARE #AcountNumber VARCHAR(50),#Product varchar(40),#ProductEndData varchar(50), #increment int = 0;
DECLARE db_cursor CURSOR FOR
select
Product,
ProductEndData from #testTable
OPEN db_cursor
FETCH NEXT FROM db_cursor INTO #Product, #ProductEndData
WHILE ##FETCH_STATUS = 0
BEGIN
set #increment = #increment+1;
SET #temptable += 'Product'+ cast(#increment as varchar) +' varchar(100),' + 'Product' + cast(#increment as varchar) + 'EndDate varchar(100),' ;
set #inserStatement += 'Product'+ cast(#increment as varchar) +',' + 'Product' + cast(#increment as varchar) + 'EndDate,';
set #insertValues += '(''' + #Product +''''+ ',' + ''''+ #ProductEndData + '''' + ')';
FETCH NEXT FROM db_cursor INTO #Product, #ProductEndData
END
CLOSE db_cursor
DEALLOCATE db_cursor
set #temptable = STUFF(#temptable, LEN(#temptable), 1, ')')
set #inserStatement = STUFF(#inserStatement, LEN(#inserStatement), 1, ')')
set #insertValues = replace(#insertValues, ')(', ',')
exec (#temptable + #inserStatement + ' values ' + #insertValues + 'select * from #tempTable1')
DECLARE #FirstDayOfWeek DATETIME
SELECT #FirstDayOfWeek =
COALESCE(
cast(#FirstDayOfWeek AS VARCHAR(MAX)) + ',[' + cast(#FirstDayOfWeek AS VARCHAR(MAX)) + ']',
'[' + cast(#FirstDayOfWeek AS VARCHAR(MAX))+ ']'
)
FROM [Calendar] c
LEFT JOIN Stores s ON c.FirstDayOfMonth = s.Validfrom
left join POSProductDivPeriod pos ON c.Period = pos.Period
DECLARE #PivotTableSQL NVARCHAR(MAX)
SET #PivotTableSQL = N'
SELECT *
FROM (
SELECT
YEAR(FirstDayOfWeek) [Year],
#FirstDayOfWeek,
pos.quantity
FROM [Calendar] c
LEFT JOIN Stores s ON c.FirstDayOfMonth = s.Validfrom
left join POSProductDivPeriod pos ON c.Period = pos.Period
) AS PivotData
PIVOT (
SUM(pos.quantity)
FOR #FirstDayOfWeek IN (
' +CAST(#FirstDayOfWeek AS VARCHAR(MAX)) + '
)
) AS PivotTable
'
EXECUTE(#PivotTableSQL)
when i run all of this i get command complete
when i run execute i get must declare scarlar
im trying to run my code for my pivot table and for it to execute right after
any ideas ?
May this article will help you,
http://www.codeproject.com/Articles/201320/Dynamic-Pivoting-with-Cubes-and-eventhandlers-in-S
I have these 2 example of CTE statement, one is hardcoded and the other is dynamic. The hardcoded works but not the dynamic. Can you check what's wrong with my dynamic statement? Thanks
-- THIS WORKS
WITH CTE AS
(
SELECT TOP 1 *
FROM Citi_v823_21Nov2013.dbo.GroupRelation_Audit
WHERE Citi_v823_21Nov2013.dbo.GroupRelation_Audit.ParentEntityIDCounter = #ParentEntityIDCounter
AND Citi_v823_21Nov2013.dbo.GroupRelation_Audit.ChildEntityIDCounter = #ChildEntityIDCounter
AND IsNull(Citi_v823_21Nov2013.dbo.GroupRelation_Audit.AuditDateModified, '1900-01-01') < GETDATE()
ORDER BY GroupRelationCounter DESC
)
UPDATE CTE SET DateEffectiveTo = #DateEffectiveFrom_GroupRelation
--THIS DOESN'T WORK
DECLARE #TargetDB NVARCHAR(Max)
DECLARE #SourceDB NVARCHAR(Max)
DECLARE #DateEffectiveFrom_GroupRelation DATETIME
DECLARE #UpdateRecords_GroupRelation NVARCHAR(Max)
SET #TargetDB = 'Citi_v823_21Nov2013'
SET #SourceDB = 'UATCitiv82320131018'
SET #DateEffectiveFrom_GroupRelation = '2013-09-29'
SET #UpdateRecords_GroupRelation = '
;WITH CTE AS
(
SELECT TOP 1 *
FROM ' + #TargetDB + '.dbo.GroupRelation_Audit
WHERE ' + #TargetDB + '.dbo.GroupRelation_Audit.ParentEntityIDCounter = ' + CONVERT(NVARCHAR(Max), #ParentEntityIDCounter) +'
AND ' + #TargetDB + '.dbo.GroupRelation_Audit.ChildEntityIDCounter = ' + CONVERT(NVARCHAR(Max), #ChildEntityIDCounter) +'
AND IsNull(' + #TargetDB + '.dbo.GroupRelation_Audit.AuditDateModified, ''1900-01-01'') < GETDATE() ''
ORDER BY ' + #TargetDB + '.dbo.GroupRelation_Audit.GroupRelationCounter DESC
)'
UPDATE CTE SET DateEffectiveTo = #DateEffectiveFrom_GroupRelation
EXEC sp_executesql #UpdateRecords_GroupRelation
Move your UPDATE statement inside the SQL statement.
The CTE is locally-scoped, so that statement won't known what CTE is and will simply throw an invalid object error.
Sample code:
declare #sql nvarchar(max)
select *
into ##t
from
(select 3 as b) tmp
select * from ##t;
set #sql = ';WITH a as (select 1 as b) update ##t set b = (select top 1 * from a) '
EXEC sp_executesql #sql
select * from ##t
drop table ##t