I am writing a query which unpivots the table. The issue is that based on the mapping file I would like to either input values as a constant or variable.
For example - if in mapping file the ExtDate is constant, e.g. ExtDate = '2017-12-31' I want to use this value (2017-12-31). However if ExtDate starts with 'Var' I would like to use variable values - e.g. when ExtDate = VarOpenDate, then I want to fulfill the column with values from column OpenDate. Below exemplary rows from MappingFile:
CREATE TABLE MappingFile (ColNum INT, Variable CHAR(50), IsUsed CHAR(50), ID CHAR(50), ExtDate CHAR(50), DataDate CHAR(50), ValueDate CHAR(50), Flag CHAR(50), Unit CHAR(50))
INSERT INTO MappingFile VALUES (1, 'ClientId', 'YES', 'VarAcctID', '2017-12-31', 'VarSigningDate', 'VarSigningDate', 'X', '')
INSERT INTO MappingFile VALUES (2, 'ProductGroup', 'YES', 'VarAcctID', 'VarOpenDate', 'VarSigningDate', 'VarSigningDate', 'X', '')
INSERT INTO MappingFile VALUES (3, 'ProductType', 'YES', 'VarAcctID', 'VarOpenDate', 'VarSigningDate', 'VarSigningDate', 'X', '')
In order to do this I wrote a code below (this is a simplification, as there are more columns and whole query is in the inserting while loop).
DECLARE #I INT = 2
DECLARE #COL CHAR(50)
DECLARE #ID CHAR(50)
DECLARE #EDT CHAR(50)
DECLARE #DDT CHAR(50)
DECLARE #DTS CHAR(50) = 'dataset_name'
SET #ID = (SELECT ID FROM MappingFile WHERE ColNum = #I)
SET #EDT = (SELECT ExtDate FROM MappingFile WHERE ColNum = #I)
SET #DDT = (SELECT DataDate FROM MappingFile WHERE ColNum = #I)
SET #COL = (SELECT Variable FROM MappingFile WHERE ColNum = #I)
EXEC('
SELECT ''' + #DTS + ''',
CASE WHEN SUBSTRING(''' + #ID + ''', 1, 3) = ''Var'' THEN ' + #ID + ' ELSE ''' + #ID + ''' END,
CASE WHEN SUBSTRING(''' + #EDT + ''', 1, 3) = ''Var'' THEN ' + #EDT + ' ELSE ''' + #EDT + ''' END,
CASE WHEN SUBSTRING(''' + #DDT + ''', 1, 3) = ''Var'' THEN ' + #DDT + ' ELSE ''' + #DDT + ''' END,
''' + #COL + ''',
' + #COL + '
FROM ' + #DTS + '
WHERE ' + #COL + ' IS NOT NULL
')
Unfortunately in case when ExtDate is constant string the query gives me an error. It is caused by the fact that the result expression:
THEN ' + #EDT + '
returns string which is not a name of column. This gives me an error, altough it shouldn't because if
SUBSTRING(''' + #ID + ''', 1, 3) <> ''Var''
then the result of the case is
' ELSE ''' + #DDT + '''
which is not a column name, but a constant string.
You have dynamically built a query that cannot parse.
Consider the test query below:
DECLARE #T TABLE (
Col int
);
INSERT INTO #T VALUES (1);
SELECT CASE
WHEN 1=0 THEN NoSuch ELSE 'Hi' END
FROM #T;
Run this, and it results in
Msg 207, Level 16, State 1, Line 8 Invalid column name 'NoSuch'.
Because even though the CASE doesn't execute the THEN and skips to the ELSE, the parser doesn't know that in advance and checks to make sure NoSuch is a column. If it isn't, it rejects the query before it even tries to run it.
One solution is to check your variables OUTSIDE of the dynamic string.
Thanks to #Tab Alleman I managed to find the solution. The phenomenon regarding parser is exactly true, therefore I had to check the variables outside the EXEC command.
Below find the code which works:
DECLARE #I INT = 2
DECLARE #COL CHAR(50)
DECLARE #ID CHAR(50)
DECLARE #EDT CHAR(50)
DECLARE #DDT CHAR(50)
DECLARE #DTS CHAR(50) = 'dataset_name'
SET #ID = (SELECT CASE WHEN SUBSTRING(ID , 1, 3) = 'Var' THEN LTRIM(RTRIM(RIGHT(ID , LEN(ID ) - 3))) WHEN ID = '' THEN 'NULL' ELSE CONCAT('''', LTRIM(RTRIM(ID )), '''') END FROM MappingFile WHERE ColNum = #I)
SET #EDT = (SELECT CASE WHEN SUBSTRING(ExtDate , 1, 3) = 'Var' THEN LTRIM(RTRIM(RIGHT(ExtDate , LEN(ExtDate ) - 3))) WHEN ExtDate = '' THEN 'NULL' ELSE CONCAT('''', LTRIM(RTRIM(ExtDate )), '''') END FROM MappingFile WHERE ColNum = #I)
SET #DDT = (SELECT CASE WHEN SUBSTRING(DataDate , 1, 3) = 'Var' THEN LTRIM(RTRIM(RIGHT(DataDate , LEN(DataDate ) - 3))) WHEN DataDate = '' THEN 'NULL' ELSE CONCAT('''', LTRIM(RTRIM(DataDate )), '''') END FROM MappingFile WHERE ColNum = #I)
SET #COL = (SELECT Variable FROM MappingFile WHERE ColNum = #I)
EXEC('
SELECT ''' + #DTS + ''',
' + #ID + ',
' + #EDT + ',
' + #DDT + ',
''' + #COL + ''',
' + #COL + '
FROM ' + #DTS + '
WHERE ' + #COL + ' IS NOT NULL
')
Related
This is the code I used to run, but while running I get the following error:
Msg 156, Level 15, State 1, Line 24
Incorrect syntax near the keyword 'AS'.
Msg 153, Level 15, State 2, Line 34
Invalid usage of the option NEXT in the FETCH statement
Code:
ALTER PROCEDURE [dbo].[GetCustomers_Pager] --'RowNumber','asc',0,10,''
(#sortColumn VARCHAR(50),
#sortOrder VARCHAR(50),
#OffsetValue INT,
#PagingSize INT,
#SearchText VARCHAR(50) )
AS
BEGIN
DECLARE #sqlQuery NVARCHAR(MAX) =
'SELECT
ROW_NUMBER() OVER (ORDER BY document_id ASC) AS RowNumber,
document_id AS ID,
document_title AS Title,
document_url AS Download,
online_filing_link AS FileOnline,
CASE
WHEN LTRIM(document_fee) IS NULL THEN ''
ELSE CASE
WHEN LEN(document_fee) = 0 THEN ''
ELSE ''$'' + CONVERT(VARCHAR, document_fee)
END
END AS FilingFee ,
CASE
WHEN LTRIM(document_fee_enhanced) IS NULL THEN ''
ELSE CASE
WHEN LEN(document_fee_enhanced) = 0 THEN ''
ELSE ''$'' + CONVERT(VARCHAR, document_fee_enhanced)
END
END AS EnhancedFee,
COUNT(document_id) OVER() AS FilterTotalCount
FROM
documents';
SET #sqlQuery = #sqlQuery +
' WHERE ((''' + #SearchText + ''' <> '''' AND (document_title LIKE ''%' + #SearchText + '%'' OR FilingFee LIKE ''%' + #SearchText + '%'')) OR ('''+#SearchText+''' = ''''))';
SET #sqlQuery = #sqlQuery + ' ORDER BY ' + #sortColumn + ' ' + #sortOrder;
SET #sqlQuery = #sqlQuery + ' OFFSET ' + CAST(#OffsetValue AS varchar(100)) + ' ROWS FETCH NEXT ' + CAST(#PagingSize AS varchar(100)) + ' ROWS ONLY';
EXEC (#sqlQuery);
END
The error because the generated SQL statement syntax is invalid. The literals THEN '' should be THEN '''' so the resultant statement has matching quotes. I suggest you add a PRINT statement for the generated SQL during debugging like the below example.
More importantly, use parameters for values that vary by execution instead of building the statement with literals for those values. Make sure the values for #SortColumn and #sortOrder are provided from a trusted source or validate the values against the catalog views in the proc code.
CREATE OR ALTER PROCEDURE dbo.GetCustomers_Pager
#sortColumn VARCHAR(50)
, #sortOrder VARCHAR(50)
, #OffsetValue INT
, #PagingSize INT
, #SearchText VARCHAR(50)
AS
DECLARE #sqlQuery NVARCHAR(MAX) =
N'SELECT
ROW_NUMBER() OVER (ORDER BY document_id ASC) AS RowNumber,
document_id AS ID,
document_title AS Title,
document_url AS Download,
online_filing_link AS FileOnline,
CASE
WHEN LTRIM(document_fee) IS NULL THEN ''''
ELSE CASE
WHEN LEN(document_fee) = 0 THEN ''''
ELSE ''$'' + CONVERT(VARCHAR, document_fee)
END
END AS FilingFee ,
CASE
WHEN LTRIM(document_fee_enhanced) IS NULL THEN ''''
ELSE CASE
WHEN LEN(document_fee_enhanced) = 0 THEN ''''
ELSE ''$'' + CONVERT(VARCHAR, document_fee_enhanced)
END
END AS EnhancedFee,
COUNT(document_id) OVER() AS FilterTotalCount
FROM
documents';
SET #sqlQuery += N' WHERE ((#SearchText <> '''' AND (document_title LIKE ''%'' + #SearchText + ''%'' OR FilingFee LIKE ''%'' + #SearchText + ''%'')) OR (#SearchText = ''''))';
SET #sqlQuery += N' ORDER BY ' + #sortColumn + ' ' + #sortOrder;
SET #sqlQuery += N' OFFSET #OffsetValue ROWS FETCH NEXT #PagingSize ROWS ONLY;';
--use PRINT to debug SQL query text
--PRINT #sqlQuery;
EXEC sp_executesql
#sqlQuery
, N'#OffsetValue INT, #PagingSize INT, #SearchText VARCHAR(50)'
, #OffsetValue = 0
, #PagingSize = 10
, #SearchText = '';
GO
EXECUTE [dbo].[GetCustomers_Pager]
#sortColumn ='RowNumber'
,#sortOrder = 'asc'
,#OffsetValue = 0
,#PagingSize = 10
,#SearchText = '';
GO
I'm looking for an efficient way to convert rows to columns in SQL server, I heard that PIVOT is not very fast, and I need to deal with lot of records.
This is my example:
-------------------------------
| Id | Value | ColumnName |
-------------------------------
| 1 | John | FirstName |
| 2 | 2.4 | Amount |
| 3 | ZH1E4A | PostalCode |
| 4 | Fork | LastName |
| 5 | 857685 | AccountNumber |
-------------------------------
This is my result:
---------------------------------------------------------------------
| FirstName |Amount| PostalCode | LastName | AccountNumber |
---------------------------------------------------------------------
| John | 2.4 | ZH1E4A | Fork | 857685 |
---------------------------------------------------------------------
How can I build the result?
There are several ways that you can transform data from multiple rows into columns.
Using PIVOT
In SQL Server you can use the PIVOT function to transform the data from rows to columns:
select Firstname, Amount, PostalCode, LastName, AccountNumber
from
(
select value, columnname
from yourtable
) d
pivot
(
max(value)
for columnname in (Firstname, Amount, PostalCode, LastName, AccountNumber)
) piv;
See Demo.
Pivot with unknown number of columnnames
If you have an unknown number of columnnames that you want to transpose, then you can use dynamic SQL:
DECLARE #cols AS NVARCHAR(MAX),
#query AS NVARCHAR(MAX)
select #cols = STUFF((SELECT ',' + QUOTENAME(ColumnName)
from yourtable
group by ColumnName, id
order by id
FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
set #query = N'SELECT ' + #cols + N' from
(
select value, ColumnName
from yourtable
) x
pivot
(
max(value)
for ColumnName in (' + #cols + N')
) p '
exec sp_executesql #query;
See Demo.
Using an aggregate function
If you do not want to use the PIVOT function, then you can use an aggregate function with a CASE expression:
select
max(case when columnname = 'FirstName' then value end) Firstname,
max(case when columnname = 'Amount' then value end) Amount,
max(case when columnname = 'PostalCode' then value end) PostalCode,
max(case when columnname = 'LastName' then value end) LastName,
max(case when columnname = 'AccountNumber' then value end) AccountNumber
from yourtable
See Demo.
Using multiple joins
This could also be completed using multiple joins, but you will need some column to associate each of the rows which you do not have in your sample data. But the basic syntax would be:
select fn.value as FirstName,
a.value as Amount,
pc.value as PostalCode,
ln.value as LastName,
an.value as AccountNumber
from yourtable fn
left join yourtable a
on fn.somecol = a.somecol
and a.columnname = 'Amount'
left join yourtable pc
on fn.somecol = pc.somecol
and pc.columnname = 'PostalCode'
left join yourtable ln
on fn.somecol = ln.somecol
and ln.columnname = 'LastName'
left join yourtable an
on fn.somecol = an.somecol
and an.columnname = 'AccountNumber'
where fn.columnname = 'Firstname'
This is rather a method than just a single script but gives you much more flexibility.
First of all There are 3 objects:
User defined TABLE type [ColumnActionList] -> holds data as
parameter
SP [proc_PivotPrepare] -> prepares our data
SP [proc_PivotExecute] -> execute the script
CREATE TYPE [dbo].[ColumnActionList] AS TABLE
(
[ID] [smallint] NOT NULL,
[ColumnName] nvarchar NOT NULL,
[Action] nchar NOT NULL
);
GO
CREATE PROCEDURE [dbo].[proc_PivotPrepare]
(
#DB_Name nvarchar(128),
#TableName nvarchar(128)
)
AS
SELECT #DB_Name = ISNULL(#DB_Name,db_name())
DECLARE #SQL_Code nvarchar(max)
DECLARE #MyTab TABLE (ID smallint identity(1,1), [Column_Name] nvarchar(128), [Type] nchar(1), [Set Action SQL] nvarchar(max));
SELECT #SQL_Code = 'SELECT [<| SQL_Code |>] = '' '' '
+ 'UNION ALL '
+ 'SELECT ''----------------------------------------------------------------------------------------------------'' '
+ 'UNION ALL '
+ 'SELECT ''-----| Declare user defined type [ID] / [ColumnName] / [PivotAction] '' '
+ 'UNION ALL '
+ 'SELECT ''----------------------------------------------------------------------------------------------------'' '
+ 'UNION ALL '
+ 'SELECT ''DECLARE #ColumnListWithActions ColumnActionList;'''
+ 'UNION ALL '
+ 'SELECT ''----------------------------------------------------------------------------------------------------'' '
+ 'UNION ALL '
+ 'SELECT ''-----| Set [PivotAction] (''''S'''' as default) to select dimentions and values '' '
+ 'UNION ALL '
+ 'SELECT ''-----|'''
+ 'UNION ALL '
+ 'SELECT ''-----| ''''S'''' = Stable column || ''''D'''' = Dimention column || ''''V'''' = Value column '' '
+ 'UNION ALL '
+ 'SELECT ''----------------------------------------------------------------------------------------------------'' '
+ 'UNION ALL '
+ 'SELECT ''INSERT INTO #ColumnListWithActions VALUES ('' + CAST( ROW_NUMBER() OVER (ORDER BY [NAME]) as nvarchar(10)) + '', '' + '''''''' + [NAME] + ''''''''+ '', ''''S'''');'''
+ 'FROM [' + #DB_Name + '].sys.columns '
+ 'WHERE object_id = object_id(''[' + #DB_Name + ']..[' + #TableName + ']'') '
+ 'UNION ALL '
+ 'SELECT ''----------------------------------------------------------------------------------------------------'' '
+ 'UNION ALL '
+ 'SELECT ''-----| Execute sp_PivotExecute with parameters: columns and dimentions and main table name'' '
+ 'UNION ALL '
+ 'SELECT ''----------------------------------------------------------------------------------------------------'' '
+ 'UNION ALL '
+ 'SELECT ''EXEC [dbo].[sp_PivotExecute] #ColumnListWithActions, ' + '''''' + #TableName + '''''' + ';'''
+ 'UNION ALL '
+ 'SELECT ''----------------------------------------------------------------------------------------------------'' '
EXECUTE SP_EXECUTESQL #SQL_Code;
GO
CREATE PROCEDURE [dbo].[sp_PivotExecute]
(
#ColumnListWithActions ColumnActionList ReadOnly
,#TableName nvarchar(128)
)
AS
--#######################################################################################################################
--###| Step 1 - Select our user-defined-table-variable into temp table
--#######################################################################################################################
IF OBJECT_ID('tempdb.dbo.#ColumnListWithActions', 'U') IS NOT NULL DROP TABLE #ColumnListWithActions;
SELECT * INTO #ColumnListWithActions FROM #ColumnListWithActions;
--#######################################################################################################################
--###| Step 2 - Preparing lists of column groups as strings:
--#######################################################################################################################
DECLARE #ColumnName nvarchar(128)
DECLARE #Destiny nchar(1)
DECLARE #ListOfColumns_Stable nvarchar(max)
DECLARE #ListOfColumns_Dimension nvarchar(max)
DECLARE #ListOfColumns_Variable nvarchar(max)
--############################
--###| Cursor for List of Stable Columns
--############################
DECLARE ColumnListStringCreator_S CURSOR FOR
SELECT [ColumnName]
FROM #ColumnListWithActions
WHERE [Action] = 'S'
OPEN ColumnListStringCreator_S;
FETCH NEXT FROM ColumnListStringCreator_S
INTO #ColumnName
WHILE ##FETCH_STATUS = 0
BEGIN
SELECT #ListOfColumns_Stable = ISNULL(#ListOfColumns_Stable, '') + ' [' + #ColumnName + '] ,';
FETCH NEXT FROM ColumnListStringCreator_S INTO #ColumnName
END
CLOSE ColumnListStringCreator_S;
DEALLOCATE ColumnListStringCreator_S;
--############################
--###| Cursor for List of Dimension Columns
--############################
DECLARE ColumnListStringCreator_D CURSOR FOR
SELECT [ColumnName]
FROM #ColumnListWithActions
WHERE [Action] = 'D'
OPEN ColumnListStringCreator_D;
FETCH NEXT FROM ColumnListStringCreator_D
INTO #ColumnName
WHILE ##FETCH_STATUS = 0
BEGIN
SELECT #ListOfColumns_Dimension = ISNULL(#ListOfColumns_Dimension, '') + ' [' + #ColumnName + '] ,';
FETCH NEXT FROM ColumnListStringCreator_D INTO #ColumnName
END
CLOSE ColumnListStringCreator_D;
DEALLOCATE ColumnListStringCreator_D;
--############################
--###| Cursor for List of Variable Columns
--############################
DECLARE ColumnListStringCreator_V CURSOR FOR
SELECT [ColumnName]
FROM #ColumnListWithActions
WHERE [Action] = 'V'
OPEN ColumnListStringCreator_V;
FETCH NEXT FROM ColumnListStringCreator_V
INTO #ColumnName
WHILE ##FETCH_STATUS = 0
BEGIN
SELECT #ListOfColumns_Variable = ISNULL(#ListOfColumns_Variable, '') + ' [' + #ColumnName + '] ,';
FETCH NEXT FROM ColumnListStringCreator_V INTO #ColumnName
END
CLOSE ColumnListStringCreator_V;
DEALLOCATE ColumnListStringCreator_V;
SELECT #ListOfColumns_Variable = LEFT(#ListOfColumns_Variable, LEN(#ListOfColumns_Variable) - 1);
SELECT #ListOfColumns_Dimension = LEFT(#ListOfColumns_Dimension, LEN(#ListOfColumns_Dimension) - 1);
SELECT #ListOfColumns_Stable = LEFT(#ListOfColumns_Stable, LEN(#ListOfColumns_Stable) - 1);
--#######################################################################################################################
--###| Step 3 - Preparing table with all possible connections between Dimension columns excluding NULLs
--#######################################################################################################################
DECLARE #DIM_TAB TABLE ([DIM_ID] smallint, [ColumnName] nvarchar(128))
INSERT INTO #DIM_TAB
SELECT [DIM_ID] = ROW_NUMBER() OVER(ORDER BY [ColumnName]), [ColumnName] FROM #ColumnListWithActions WHERE [Action] = 'D';
DECLARE #DIM_ID smallint;
SELECT #DIM_ID = 1;
DECLARE #SQL_Dimentions nvarchar(max);
IF OBJECT_ID('tempdb.dbo.##ALL_Dimentions', 'U') IS NOT NULL DROP TABLE ##ALL_Dimentions;
SELECT #SQL_Dimentions = 'SELECT [xxx_ID_xxx] = ROW_NUMBER() OVER (ORDER BY ' + #ListOfColumns_Dimension + '), ' + #ListOfColumns_Dimension
+ ' INTO ##ALL_Dimentions '
+ ' FROM (SELECT DISTINCT' + #ListOfColumns_Dimension + ' FROM ' + #TableName
+ ' WHERE ' + (SELECT [ColumnName] FROM #DIM_TAB WHERE [DIM_ID] = #DIM_ID) + ' IS NOT NULL ';
SELECT #DIM_ID = #DIM_ID + 1;
WHILE #DIM_ID <= (SELECT MAX([DIM_ID]) FROM #DIM_TAB)
BEGIN
SELECT #SQL_Dimentions = #SQL_Dimentions + 'AND ' + (SELECT [ColumnName] FROM #DIM_TAB WHERE [DIM_ID] = #DIM_ID) + ' IS NOT NULL ';
SELECT #DIM_ID = #DIM_ID + 1;
END
SELECT #SQL_Dimentions = #SQL_Dimentions + ' )x';
EXECUTE SP_EXECUTESQL #SQL_Dimentions;
--#######################################################################################################################
--###| Step 4 - Preparing table with all possible connections between Stable columns excluding NULLs
--#######################################################################################################################
DECLARE #StabPos_TAB TABLE ([StabPos_ID] smallint, [ColumnName] nvarchar(128))
INSERT INTO #StabPos_TAB
SELECT [StabPos_ID] = ROW_NUMBER() OVER(ORDER BY [ColumnName]), [ColumnName] FROM #ColumnListWithActions WHERE [Action] = 'S';
DECLARE #StabPos_ID smallint;
SELECT #StabPos_ID = 1;
DECLARE #SQL_MainStableColumnTable nvarchar(max);
IF OBJECT_ID('tempdb.dbo.##ALL_StableColumns', 'U') IS NOT NULL DROP TABLE ##ALL_StableColumns;
SELECT #SQL_MainStableColumnTable = 'SELECT xxx_ID_xxx = ROW_NUMBER() OVER (ORDER BY ' + #ListOfColumns_Stable + '), ' + #ListOfColumns_Stable
+ ' INTO ##ALL_StableColumns '
+ ' FROM (SELECT DISTINCT' + #ListOfColumns_Stable + ' FROM ' + #TableName
+ ' WHERE ' + (SELECT [ColumnName] FROM #StabPos_TAB WHERE [StabPos_ID] = #StabPos_ID) + ' IS NOT NULL ';
SELECT #StabPos_ID = #StabPos_ID + 1;
WHILE #StabPos_ID <= (SELECT MAX([StabPos_ID]) FROM #StabPos_TAB)
BEGIN
SELECT #SQL_MainStableColumnTable = #SQL_MainStableColumnTable + 'AND ' + (SELECT [ColumnName] FROM #StabPos_TAB WHERE [StabPos_ID] = #StabPos_ID) + ' IS NOT NULL ';
SELECT #StabPos_ID = #StabPos_ID + 1;
END
SELECT #SQL_MainStableColumnTable = #SQL_MainStableColumnTable + ' )x';
EXECUTE SP_EXECUTESQL #SQL_MainStableColumnTable;
--#######################################################################################################################
--###| Step 5 - Preparing table with all options ID
--#######################################################################################################################
DECLARE #FULL_SQL_1 NVARCHAR(MAX)
SELECT #FULL_SQL_1 = ''
DECLARE #i smallint
IF OBJECT_ID('tempdb.dbo.##FinalTab', 'U') IS NOT NULL DROP TABLE ##FinalTab;
SELECT #FULL_SQL_1 = 'SELECT t.*, dim.[xxx_ID_xxx] '
+ ' INTO ##FinalTab '
+ 'FROM ' + #TableName + ' t '
+ 'JOIN ##ALL_Dimentions dim '
+ 'ON t.' + (SELECT [ColumnName] FROM #DIM_TAB WHERE [DIM_ID] = 1) + ' = dim.' + (SELECT [ColumnName] FROM #DIM_TAB WHERE [DIM_ID] = 1);
SELECT #i = 2
WHILE #i <= (SELECT MAX([DIM_ID]) FROM #DIM_TAB)
BEGIN
SELECT #FULL_SQL_1 = #FULL_SQL_1 + ' AND t.' + (SELECT [ColumnName] FROM #DIM_TAB WHERE [DIM_ID] = #i) + ' = dim.' + (SELECT [ColumnName] FROM #DIM_TAB WHERE [DIM_ID] = #i)
SELECT #i = #i +1
END
EXECUTE SP_EXECUTESQL #FULL_SQL_1
--#######################################################################################################################
--###| Step 6 - Selecting final data
--#######################################################################################################################
DECLARE #STAB_TAB TABLE ([STAB_ID] smallint, [ColumnName] nvarchar(128))
INSERT INTO #STAB_TAB
SELECT [STAB_ID] = ROW_NUMBER() OVER(ORDER BY [ColumnName]), [ColumnName]
FROM #ColumnListWithActions WHERE [Action] = 'S';
DECLARE #VAR_TAB TABLE ([VAR_ID] smallint, [ColumnName] nvarchar(128))
INSERT INTO #VAR_TAB
SELECT [VAR_ID] = ROW_NUMBER() OVER(ORDER BY [ColumnName]), [ColumnName]
FROM #ColumnListWithActions WHERE [Action] = 'V';
DECLARE #y smallint;
DECLARE #x smallint;
DECLARE #z smallint;
DECLARE #FinalCode nvarchar(max)
SELECT #FinalCode = ' SELECT ID1.*'
SELECT #y = 1
WHILE #y <= (SELECT MAX([xxx_ID_xxx]) FROM ##FinalTab)
BEGIN
SELECT #z = 1
WHILE #z <= (SELECT MAX([VAR_ID]) FROM #VAR_TAB)
BEGIN
SELECT #FinalCode = #FinalCode + ', [ID' + CAST((#y) as varchar(10)) + '.' + (SELECT [ColumnName] FROM #VAR_TAB WHERE [VAR_ID] = #z) + '] = ID' + CAST((#y + 1) as varchar(10)) + '.' + (SELECT [ColumnName] FROM #VAR_TAB WHERE [VAR_ID] = #z)
SELECT #z = #z + 1
END
SELECT #y = #y + 1
END
SELECT #FinalCode = #FinalCode +
' FROM ( SELECT * FROM ##ALL_StableColumns)ID1';
SELECT #y = 1
WHILE #y <= (SELECT MAX([xxx_ID_xxx]) FROM ##FinalTab)
BEGIN
SELECT #x = 1
SELECT #FinalCode = #FinalCode
+ ' LEFT JOIN (SELECT ' + #ListOfColumns_Stable + ' , ' + #ListOfColumns_Variable
+ ' FROM ##FinalTab WHERE [xxx_ID_xxx] = '
+ CAST(#y as varchar(10)) + ' )ID' + CAST((#y + 1) as varchar(10))
+ ' ON 1 = 1'
WHILE #x <= (SELECT MAX([STAB_ID]) FROM #STAB_TAB)
BEGIN
SELECT #FinalCode = #FinalCode + ' AND ID1.' + (SELECT [ColumnName] FROM #STAB_TAB WHERE [STAB_ID] = #x) + ' = ID' + CAST((#y+1) as varchar(10)) + '.' + (SELECT [ColumnName] FROM #STAB_TAB WHERE [STAB_ID] = #x)
SELECT #x = #x +1
END
SELECT #y = #y + 1
END
SELECT * FROM ##ALL_Dimentions;
EXECUTE SP_EXECUTESQL #FinalCode;
From executing the first query (by passing source DB and table name) you will get a pre-created execution query for the second SP, all you have to do is define is the column from your source:
+ Stable
+ Value (will be used to concentrate values based on that)
+ Dim (column you want to use to pivot by)
Names and datatypes will be defined automatically!
I cant recommend it for any production environments but does the job for adhoc BI requests.
I modified Taryn's answer ("Pivot with unknown number of columnnames" version) to show more than 1 row in the result. This requires to have an additional "Group" column
DROP TABLE #yourtable
CREATE table #yourtable
([Id] int,[Group] int, [Value] varchar(6), [ColumnName] varchar(13))
;
INSERT INTO #yourtable
([Id],[Group], [Value], [ColumnName])
VALUES
(1,1, 'John', 'FirstName'),
(2,1, '2.4', 'Amount'),
(3,1, 'ZH1E4A', 'PostalCode'),
(4,1, 'Fork', 'LastName'),
(5,1, '857685', 'AccountNumber'),
(6,2, 'Pedro', 'FirstName'),
(7,2, '5.1', 'Amount'),
(8,2, '123456', 'PostalCode'),
(9,2, 'Torres', 'LastName'),
(10,2, '857686', 'AccountNumber')
;
;
DECLARE #cols AS NVARCHAR(MAX),
#query AS NVARCHAR(MAX)
select #cols = STUFF((SELECT ',' + QUOTENAME(ColumnName)
from #yourtable
group by [Group], ColumnName, id
having [group] = (SELECT TOP 1 MIN([Group])FROM #yourtable)
order by id
FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
set #query = N'SELECT ' + #cols + N' from
(
select value, ColumnName,[Group]
from #yourtable
GROUP BY [Group],ColumnName,Value
) x
pivot
(
max(value)
for ColumnName in (' + #cols + N')
) p '
exec sp_executesql #query;
Please try
CREATE TABLE pvt (Present int, [Absent] int);
GO
INSERT INTO pvt VALUES (10,40);
GO
--Unpivot the table.
SELECT Code, Value
FROM
(SELECT Present, Absent
FROM pvt) p
UNPIVOT
(Value FOR Code IN
(Present, [Absent])
)AS unpvt;
GO
DROP TABLE pvt
One more option which could be very useful is using CROSS APPLY
-- Original data
SELECT * FROM (VALUES ('1', 1, 2, 3),('2', 11, 22, 33)) AS Stage(id,col1,col2,col3)
-- row to columns using CROSS APPLY
SELECT Stage.id,v.idd, v.colc
FROM (VALUES ('1', 1, 2, 3),('2', 11, 22, 33)) AS Stage(id,col1,col2,col3)
CROSS APPLY (VALUES ('col1', col1),('col2', col2),('col3', col3)) AS v(idd,colc)
GO
I have a procedure that I working on and I don't know what's going wrong. I have reviewed all other sites and could not find the issue that I'm having
I want to create procedure that has a dynamic where clause base on a combination of bits being sent to the procedure. I don't want to have to create a bunch of similar procedures because they have slightly different conditions.
I'm placing the below query into a cursor then looping through the cursor. Please help.
CREATE PROCEDURE [dbo].[procContainTest] (
#USE_A BIT,
#USE_B BIT,
#ValueA VARCHAR(50),
#ValueB VARCHAR(50),
#USERID VARCHAR(50)
)
AS
DECLARE #TEMP_Col1 INT,
#TEMP_Col2 INT,
#TEMP_Col3 VARCHAR(50),
#TEMP_Col4 VARCHAR(50),
#TEMP_Col5 VARCHAR(50),
#POINT_ONE NVARCHAR(50),
#POINT_TWO NVARCHAR(50)
SET #TRIGGER = 0
WHILE #TRIGGER = 0
BEGIN
-- F2 Booking Term
IF #USE_A = 1
AND #USE_B = 1
BEGIN
SET #POINT_ONE = 'ColName2'
SET #POINT_TWO = 'ColName3'
END
-- F6 Booking Term
IF #USE_A = 0
AND #USE_B = 1
BEGIN
SET #POINT_ONE = 'ColName1'
SET #POINT_TWO = 'ColName2'
END
DECLARE INNER_CURSOR CURSOR
FOR
SELECT TOP 1 TEMP_Col1 INT,
TEMP_Col2,
TEMP_Col3,
#TEMP_Col4,
#TEMP_Col5
FROM TEMP_Table
WHERE #POINT_ONE = + '''' + #ValueA + ''''
AND #POINT_TWO = + '''' + #ValueB + ''''
AND USERID = #USERID
ORDER BY LENGTH
l
You can put your Select Statmement in a variable like:
declare #YourSelectStatement nvarchar(max)
set #YourSelectStatement = ' SELECT TOP 1 TEMP_Col1 INT,
TEMP_Col2,
TEMP_Col3,
FROM TEMP_Table
WHERE ' + #POINT_ONE + '=' + #ValueA + '
AND ' + #POINT_TWO + '=' + #ValueB + '
AND USERID = ' + #USERID + '
ORDER BY LENGTH'
sp_executesql(#YourSelectStatement)
this is probably help you:
SELECT
id, first, last, email, notes
FROM
My_Table
WHERE
CASE ''''+#column_name_variable+''''
WHEN ''''+column_1+''''=1 THEN column_1
WHEN ''''+column_2+''''=2 THEN column_2
...
ELSE 'not null'
END IS NOT NULL
I would avoid the use of dynamic SQL altogether. You can eliminate your IF statements and embed the same logid in your SELECT statement like this:
SELECT TOP 1
TEMP_Col1,
TEMP_Col2,
TEMP_Col3,
TEMP_Col4,
TEMP_Col5
FROM TEMP_Table
WHERE (#USE_A = 1 AND #USE_B = 1 AND ColName2 = '''' + #ValueA + '''' AND ColName3 = '''' + #ValueB + '''')
OR (#USE_A = 0 AND #USE_B = 1 AND ColName1 = '''' + #ValueA + '''' AND ColName2 = '''' + #ValueB + '''')
I keep getting the following error:
Msg 241, Level 16, State 1, Line 9 Conversion failed when converting
datetime from character string.
Here is the code I am trying to execute:
DECLARE #v_sql varchar(max),
#v_database varchar(25),
#vStartTime DATETIME,
#vEndTime DATETIME
SELECT #v_database = N'[DATABASE_NAME]', #vStartTime = '2012-09-27', #vEndTime = '2012-11-27'
SELECT #v_sql = N'SELECT
(SELECT ID FROM DATASTORE.DBO.PLANT WHERE DESCRIPTION = ''Henderson''),
SEQ,
AID,
NAME,
GRP,
AREA,
PRIO,
CASE ITIME WHEN ''-'' THEN NULL ELSE CAST(SUBSTRING(ITIME,1,8) + '' '' + SUBSTRING(ITIME,9,2) + '':'' + SUBSTRING(ITIME,11,2) + '':'' + SUBSTRING(ITIME,13,2) AS DATETIME) END ITIME,
CASE ATIME WHEN ''-'' THEN NULL ELSE CAST(SUBSTRING(ATIME,1,8) + '' '' + SUBSTRING(ATIME,9,2) + '':'' + SUBSTRING(ATIME,11,2) + '':'' + SUBSTRING(ATIME,13,2) AS DATETIME) END ATIME,
CASE NTIME WHEN ''-'' THEN NULL ELSE CAST(SUBSTRING(NTIME,1,8) + '' '' + SUBSTRING(NTIME,9,2) + '':'' + SUBSTRING(NTIME,11,2) + '':'' + SUBSTRING(NTIME,13,2) AS DATETIME) END NTIME,
DUR,
MSG,
VAR1,
VAR2,
VAR3,
VAR4,
OPR,
USER_COMMENT
FROM ' + #v_database + '.PROD.ALARM
WHERE CAST(substring(ITIME, 1, 4) + ''-'' + substring(ITIME, 5, 2) + ''-'' + substring(ITIME, 7, 2) + '' '' + substring(ITIME,9,2) + '':'' + substring(ITIME,11,2) + '':'' + substring(ITIME,13,2) + substring(ITIME,15,3) AS DATETIME) BETWEEN ' + #vStartTime +' AND ' + #vEndTime + ' ORDER BY ITIME'
EXEC(#v_sql)
Any help would be much appreciated, I am looking into this for a co-worker and it's got us both stumped.
Edit with a bit more digging, we were able to resolve it ourselves, passing parameters to sp_executesql:
declare
#vSql NVARCHAR(MAX),
#vParam NVARCHAR(MAX),
#vDatabase VARCHAR(15)
SET #vParam = '#vStartTime DATETIME, #vEndTime DATETIME'
SELECT #vSql = '
SELECT ''
'+ #vDatabase + ''',
ITEM_CODE,
SOURCE,
DEST,
TRAN_DT,
MILL_NAME,
NULL
FROM ' + #vDatabase + '.PROD.GRD_LOG
WHERE TRAN_DT BETWEEN #vStartTime AND #vEndTime'
EXEC sp_executesql #vSql, #vParam, #vStartTime, #vEndTime
By making the variables NVARCHAR(MAX), and then using sp_executesql instead of just executing the #vSql variable, we were able to resolve our issue.
Thanks to anyone who might have been looking into this.
In your original dynamic SQL, you were trying to add datetime variable to a text string which results in SQL attempting to convert the text string to a datetime value (by the order of precedence of conversion). You need to set the variables to nvarchar as well to avoid conversion in the original dynamic SQL.
We've just been given the following code as a solution for a complicated search query in a new application provided by offshore developers. I'm skeptical of the use of dynamic SQL because I could close the SQL statement using '; and then excute a nasty that will be performed on the database!
Any ideas on how to fix the injection attack?
ALTER procedure [dbo].[SearchVenues] --'','',10,1,1,''
#selectedFeature as varchar(MAX),
#searchStr as varchar(100),
#pageCount as int,
#startIndex as int,
#searchId as int,
#venueName as varchar(100),
#range int,
#latitude varchar(100),
#longitude varchar(100),
#showAll int,
#OrderBy varchar(50),
#SearchOrder varchar(10)
AS
DECLARE #sqlRowNum as varchar(max)
DECLARE #sqlRowNumWhere as varchar(max)
DECLARE #withFunction as varchar(max)
DECLARE #withFunction1 as varchar(max)
DECLARE #endIndex as int
SET #endIndex = #startIndex + #pageCount -1
SET #sqlRowNum = ' SELECT Row_Number() OVER (ORDER BY '
IF #OrderBy = 'Distance'
SET #sqlRowNum = #sqlRowNum + 'dbo.GeocodeDistanceMiles(Latitude,Longitude,' + #latitude + ',' + #longitude + ') ' +#SearchOrder
ELSE
SET #sqlRowNum = #sqlRowNum + #OrderBy + ' '+ #SearchOrder
SET #sqlRowNum = #sqlRowNum + ' ) AS RowNumber,ID,RecordId,EliteStatus,Name,Description,
Address,TotalReviews,AverageFacilityRating,AverageServiceRating,Address1,Address2,Address3,Address4,Address5,Address6,PhoneNumber,
visitCount,referalCount,requestCount,imgUrl,Latitude,Longitude,
Convert(decimal(10,2),dbo.GeocodeDistanceMiles(Latitude,Longitude,' + #latitude + ',' + #longitude + ')) as distance
FROM VenueAllData '
SET #sqlRowNumWhere = 'where Enabled=1 and EliteStatus <> 3 '
--PRINT('#sqlRowNum ='+#sqlRowNum)
IF #searchStr <> ''
BEGIN
IF (#searchId = 1) -- county search
BEGIN
SET #sqlRowNumWhere = #sqlRowNumWhere + ' and Address5 like ''' + #searchStr + '%'''
END
ELSE IF(#searchId = 2 ) -- Town search
BEGIN
SET #sqlRowNumWhere = #sqlRowNumWhere + ' and Address4 like ''' + #searchStr + '%'''
END
ELSE IF(#searchId = 3 ) -- postcode search
BEGIN
SET #sqlRowNumWhere = #sqlRowNumWhere + ' and Address6 like ''' + #searchStr + '%'''
END
IF (#searchId = 4) -- Search By Name
BEGIN
IF #venueName <> ''
SET #sqlRowNumWhere = #sqlRowNumWhere + ' and ( Name like ''%' + #venueName + '%'' OR Address like ''%'+ #venueName+'%'' ) '
ELSE
SET #sqlRowNumWhere = #sqlRowNumWhere + ' and ( Name like ''%' + #searchStr + '%'' OR Address like ''%'+ #searchStr+'%'' ) '
END
END
IF #venueName <> '' AND #searchId <> 4
SET #sqlRowNumWhere = #sqlRowNumWhere + ' and ( Name like ''%' + #venueName + '%'' OR Address like ''%'+ #venueName+'%'' ) '
set #sqlRowNum = #sqlRowNum + ' ' + #sqlRowNumWhere
--PRINT(#sqlRowNum)
IF #selectedFeature <> ''
BEGIN
DECLARE #val1 varchar (255)
Declare #SQLAttributes varchar(max)
Set #SQLAttributes = ''
Declare #tempAttribute varchar(max)
Declare #AttrId int
while (#selectedFeature <> '')
BEGIN
SET #AttrId = CAST(SUBSTRING(#selectedFeature,1,CHARINDEX(',',#selectedFeature)-1) AS Int)
Select #tempAttribute = ColumnName from Attribute where id = #AttrId
SET #selectedFeature = SUBSTRING(#selectedFeature,len(#AttrId)+2,len(#selectedFeature))
SET #SQLAttributes = #SQLAttributes + ' ' + #tempAttribute + ' = 1 And '
END
Set #SQLAttributes = SUBSTRING(#SQLAttributes,0,LEN(#SQLAttributes)-3)
set #sqlRowNum = #sqlRowNum + ' and ID in (Select VenueId from '
set #sqlRowNum = #sqlRowNum + ' CachedVenueAttributes WHERE ' + #SQLAttributes + ') '
END
IF #showAll <> 1
set #sqlRowNum = #sqlRowNum + ' and dbo.GeocodeDistanceMiles(Latitude,Longitude,' + #latitude + ',' + #longitude + ') <= ' + convert(varchar,#range )
set #withFunction = 'WITH LogEntries AS (' + #sqlRowNum + ')
SELECT * FROM LogEntries WHERE RowNumber between '+ Convert(varchar,#startIndex) +
' and ' + Convert(varchar,#endIndex) + ' ORDER BY ' + #OrderBy + ' ' + #SearchOrder
print(#withFunction)
exec(#withFunction)
As an aside, I would not use EXEC; rather I would use sp_executesql. See this superb article, The Curse and Blessings of Dynamic SQL, for the reason and other info on using dynamic sql.
See this answer.
Also, these:
Am I immune to SQL injections if I use stored procedures?
Avoiding SQL Injection in SQL query with Like Operator using parameters?
Can I protect against SQL Injection by escaping single-quote and surrounding user input with single-quotes?
Here's an optimized version of the query above that doesn't use dynamic SQL...
Declare #selectedFeature as varchar(MAX),
#searchStr as varchar(100),
#pageCount as int,
#startIndex as int,
#searchId as int,
#venueName as varchar(100),
#range int,
#latitude varchar(100),
#longitude varchar(100),
#showAll int,
#OrderBy varchar(50),
#SearchOrder varchar(10)
Set #startIndex = 1
Set #pageCount = 50
Set #searchStr = 'e'
Set #searchId = 4
Set #OrderBy = 'Address1'
Set #showAll = 1
--Select dbo.GeocodeDistanceMiles(Latitude,Longitude,#latitude,#longitude)
DECLARE #endIndex int
SET #endIndex = #startIndex + #pageCount -1
;
WITH LogEntries as (
SELECT
Row_Number()
OVER (ORDER BY
CASE #OrderBy
WHEN 'Distance' THEN Cast(dbo.GeocodeDistanceMiles(Latitude,Longitude,#latitude,#longitude) as varchar(10))
WHEN 'Name' THEN Name
WHEN 'Address1' THEN Address1
WHEN 'RecordId' THEN Cast(RecordId as varchar(10))
WHEN 'EliteStatus' THEN Cast(EliteStatus as varchar(10))
END) AS RowNumber,
RecordId,EliteStatus,Name,Description,
Address,TotalReviews,AverageFacilityRating,AverageServiceRating,Address1,Address2,Address3,Address4,Address5,Address6,PhoneNumber,
visitCount,referalCount,requestCount,imgUrl,Latitude,Longitude,
Convert(decimal(10,2),dbo.GeocodeDistanceMiles(Latitude,Longitude,#latitude,#longitude)) as distance
FROM VenueAllData
where Enabled=1 and EliteStatus <> 3
And
(
(Address5 like #searchStr + '%' And #searchId = 1) OR
(Address4 like #searchStr + '%' And #searchId = 2) OR
(Address6 like #searchStr + '%' And #searchId = 3) OR
(
(
#searchId = 4 And
(Name like '%' + #venueName + '%' OR Address like '%'+ #searchStr+'%')
)
)
)
And
ID in (
Select VenueID
From CachedVenueAttributes
--Extra Where Clause for the processing of VenueAttributes using #selectedFeature
)
And
(
(#showAll = 1) Or
(#showAll <> 1 and dbo.GeocodeDistanceMiles(Latitude,Longitude,#latitude,#longitude) <= convert(varchar,#range ))
)
)
SELECT * FROM LogEntries
WHERE RowNumber between #startIndex and #endIndex
ORDER BY CASE #OrderBy
WHEN 'Distance' THEN Cast(Distance as varchar(10))
WHEN 'Name' THEN Name
WHEN 'Address1' THEN Address1
WHEN 'RecordId' THEN Cast(RecordId as varchar(10))
WHEN 'EliteStatus' THEN Cast(EliteStatus as varchar(10))
END
The only thing I haven't fixed is the selection from CachedVenueAttributes that seems to build up a where statement in a loop. I think I might put this in a table valued function, and refactor it in isolation to the rest of the procedure.
I like dynamic SQL for search.
Where I have used it in the past I have used .Net prepared statements with any user generated string being passed in as a parameter NOT included as text in the SQL.
To run with the existing solution you can do a number of thing to mitigate risk.
White list input, validate input so that it can only contain a-zA-Z0-9\w (alpha numerics and white space) (bad if you need to support unicode chars)
Execute any dynamic sql as a restricted user. Set owner of stored proc to a user which has only read access to the tables concerned. deny write to all tables ect. Also when calling this stored proc you may need to do it with a user with similar restrictions on what they can do, as it appares MS-SQL executes dynamic sql within a storedproc as the calling user not the owner of the storedproc.
I've realized that this is a really old post, but when doing things like:
AND
(
(#showAll = 1)
OR (#showAll <> 1
AND dbo.GeocodeDistanceMiles(Latitude,Longitude,#latitude,#longitude) <= convert(varchar,#range))
)
... an OPTION(RECOMPILE) will usually help pick a more concise plan, as long as it's not going to be executed a thousand times per second or anything.