Related
Context:
I have a view in SQL Server that tracks parameters a user inputs when they run an SSRS report (ReportServer.dbo.ExecutionLog). About 50 report parameters are saved as a string in a single column with ntext datatype. I would like to break this single column up into multiple columns for each parameter.
Details:
I query the report parameters like this:
SELECT ReportID, [Parameters]
FROM ReportServer.dbo.ExecutionLog
WHERE ReportID in (N'redacted')
and [Status] in (N'rsSuccess')
ORDER BY TimeEnd DESC
And here's a small subset of what the results look like:
alpha=123&bravo=9%2C33%2C76%2C23&charlie=91&delta=29&echo=11%2F2%2F2018%2012%3A00%3A00%20AM&foxtrot=11%2F1%2F2030%2012%3A00%3A00%20AM
Quesitons:
How can I get the results to look like this:
SQL Server 2017 is Python friendly. Is Python a better language to use in this scenario just for parsing purposes?
I've seen similar topics posted here, here & here. The parameters are dynamic so parsing via SQL string functions that involve counting characters doesn't apply. This question is relevant to more people than just me because there's a large population of people using SSRS. Tracking & formatting parameters in a more digestible way is valuable for all users of SSRS.
Here is a way using the built in STRING_SPLIT. I'm just not sure what the logic is for the stuff AFTER the date, so I would discarded it but I left it for you to decide.
DEMO
declare #table table (ReportID int identity(1,1), [Parameters] varchar(8000))
insert into #table
values
('alpha=123&bravo=9%2C33%2C76%2C23&charlie=91&delta=29&echo=11%2F2%2F2018%2012%3A00%3A00%20AM&foxtrot=11%2F1%2F2030%2012%3A00%3A00%20AM')
,('alpha=457893&bravo=9%2C33%2C76%2C23&charlie=91&delta=29&echo=11%2F2%2F2018%2012%3A00%3A00%20AM&foxtrot=11%2F1%2F2030%2012%3A00%3A00%20AM')
select
ReportID
,[Parameters]
,alpha = max(iif(value like 'alpha%',substring(value,charindex('=',value) + 1,99),null))
,bravo = max(iif(value like 'bravo%',substring(value,charindex('=',value) + 1,99),null))
,charlie = max(iif(value like 'charlie%',substring(value,charindex('=',value) + 1,99),null))
,delta = max(iif(value like 'delta%',substring(value,charindex('=',value) + 1,99),null))
,echo = max(iif(value like 'echo%',substring(value,charindex('=',value) + 1,99),null))
,foxtrot = max(iif(value like 'foxtrot%',substring(value,charindex('=',value) + 1,99),null))
from #table
cross apply string_split(replace(replace([Parameters],'%2C',','),'%2F','/'),'&')
group by ReportID, [Parameters]
Or, if they aren't static you can use a dynamic pivot. It'll take some massaging to get your columns in the correct order.
DEMO
DECLARE #cols AS NVARCHAR(MAX),
#query AS NVARCHAR(MAX);
SET #cols = STUFF((SELECT distinct ',' + QUOTENAME(substring([value],0,charindex('=',[value])))
from myTable
cross apply string_split(replace(replace([Parameters],'%2C',','),'%2F','/'),'&')
FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
select #cols
set #query = 'SELECT ReportID, ' + #cols + ' from
(
select ReportID
, ColName = substring([value],0,charindex(''='',[value]))
, ColVal = substring([value],charindex(''='',[value]) + 1,99)
from myTable
cross apply string_split(replace(replace([Parameters],''%2C'','',''),''%2F'',''/''),''&'')
) x
pivot
(
max(ColVal)
for ColName in (' + #cols + ')
) p '
execute(#query)
Split the string on the ampersand character.
Further split each row into two columns on the equals character.
In the second column, replace %2C with the comma character, and %2F with the forward-slash character, and so on with any other replacements as needed.
Use a dynamic-pivot to query the above in the format that you want.
Here's a method that starts with a lot of replaces.
To url-decode the string and transform it into an XML type.
Then it uses the XML functions to get the values for the columns.
Example snippet:
declare #Table table ([Parameters] varchar(200));
insert into #Table ([Parameters]) values
('alpha=123&bravo=9%2C33%2C76%2C23&charlie=91&delta=29&echo=11%2F2%2F2018%2012%3A00%3A00%20AM&foxtrot=11%2F1%2F2030%2012%3A00%3A00%20AM');
select
x.query('/x[key="alpha"]/val').value('.', 'int') as alpha,
x.query('/x[key="bravo"]/val').value('.', 'varchar(30)') as bravo,
x.query('/x[key="charlie"]/val').value('.', 'varchar(30)') as charlie,
x.query('/x[key="delta"]/val').value('.', 'varchar(30)') as delta,
convert(date, x.query('/x[key="echo"]/val').value('.', 'varchar(30)'), 103)as echo,
convert(date, x.query('/x[key="foxtrot"]/val').value('.', 'varchar(30)'), 103) as foxtrot
from #Table
cross apply (select cast('<x><key>'+
replace(replace(replace(replace(replace(
replace([Parameters],
'%2C',','),
'%2F','/'),
'%20',' '),
'%3A',':'),
'=','</key><val>'),
'&','</val></x><x><key>')
+'</val></x>' as XML) as x) ca
Test on db<>fiddle here
I have a question about SQL Server: I have a database column with a pattern which is like this:
up to 10 digits
then a comma
up to 10 digits
then a semicolon
e.g.
100000161, 100000031; 100000243, 100000021;
100000161, 100000031; 100000243, 100000021;
and I want to extract within the pattern the first digits (up to 10) (1.) and then a semicolon (4.)
(or, in other words, remove everything from the semicolon to the next semicolon)
100000161; 100000243; 100000161; 100000243;
Can you please advice me how to establish this in SQL Server? Im not very familiar with regex and therefore have no clue how to fix this.
Thanks,
Alex
Try this
Declare #Sql Table (SqlCol nvarchar(max))
INSERT INTO #Sql
SELECT'100000161,100000031;100000243,100000021;100000161,100000031;100000243,100000021;'
;WITH cte
AS (SELECT Row_number()
OVER(
ORDER BY (SELECT NULL)) AS Rno,
split.a.value('.', 'VARCHAR(1000)') AS Data
FROM (SELECT Cast('<S>'
+ Replace( Replace(sqlcol, ';', ','), ',',
'</S><S>')
+ '</S>'AS XML) AS Data
FROM #Sql)AS A
CROSS apply data.nodes('/S') AS Split(a))
SELECT Stuff((SELECT '; ' + data
FROM cte
WHERE rno%2 <> 0
AND data <> ''
FOR xml path ('')), 1, 2, '') AS ExpectedData
ExpectedData
-------------
100000161; 100000243; 100000161; 100000243
I believe this will get you what you are after as long as that pattern truly holds. If not it's fairly easy to ensure it does conform to that pattern and then apply this
Select Substring(TargetCol, 1, 10) + ';' From TargetTable
You can take advantage of SQL Server's XML support to convert the input string into an XML value and query it with XQuery and XPath expressions.
For example, the following query will replace each ; with </b><a> and each , to </a><b> to turn each string into <a>100000161</a><a>100000243</a><a />. After that, you can select individual <a> nodes with /a[1], /a[2] :
declare #table table (it nvarchar(200))
insert into #table values
('100000161, 100000031; 100000243, 100000021;'),
('100000161, 100000031; 100000243, 100000021;')
select
xCol.value('/a[1]','nvarchar(200)'),
xCol.value('/a[2]','nvarchar(200)')
from (
select convert(xml, '<a>'
+ replace(replace(replace(it,';','</b><a>'),',','</a><b>'),' ','')
+ '</a>')
.query('a') as xCol
from #table) as tmp
-------------------------
A1 A2
100000161 100000243
100000161 100000243
value extracts a single value from an XML field. nodes returns a table of nodes that match the XPath expression. The following query will return all "keys" :
select
a.value('.','nvarchar(200)')
from (
select convert(xml, '<a>'
+ replace(replace(replace(it,';','</b><a>'),',','</a><b>'),' ','')
+ '</a>')
.query('a') as xCol
from #table) as tmp
cross apply xCol.nodes('a') as y(a)
where a.value('.','nvarchar(200)')<>''
------------
100000161
100000243
100000161
100000243
With 200K rows of data though, I'd seriously consider transforming the data when loading it and storing it in indivisual, indexable columns, or add a separate, related table. Applying string manipulation functions on a column means that the server can't use any covering indexes to speed up queries.
If that's not possible (why?) I'd consider at least adding a separate XML-typed column that would contain the same data in XML form, to allow the creation of an XML index.
So basically to explain my situation I have a program where a user can select code numbers, that are alpha numeric. These codes are stored in my SQL database as datatype char.
When they select all the codes they want, the program then sends a few parameters(the codes being one of them). The codes are strung together and look something like this:
',01,1,A3' etc. etc. with commas separating the codes. I have the comma in front, but changing the comma to the back does not change anything.
the #reasonCode variable is the reason codes strung together.
In my where clause I have a statement that is this:
(#reasonCode = 'ALL') OR
((#reasonCode <> 'ALL' AND (charindex(',' + ro_reason_code, #reasonCode) > 0)))
Basically I want to restrict my results to just those that have those specific reason codes the user selected(among other parameters). I am trying to achieve that by stringing together the codes, and then searching through them using charindex, seperated by commas.
However I am running into an issue. Here are the results using a few different variations of reason codes:
',1' = 625 records (correct number)
',01' = 1015(correct number)
',01,1 = 1640(correct number)
',1,01' = 1015(for whatever reason it isn't picking up the 1 reason codes)
That is my issue right there.
When I put the 1 in front of the 01, it doesn't pick up the 1 reason codes. But if I do it flip-flopped it works fine...
Any ideas as to why this happens?
(I have tried also using the concat function and get the same results, and also tried forcing everything to be char datatype.)
In the end I would like the same result, regardless if it is ,01,1 or ,1,01.
I'm pretty sure this is because you said you're using the char type instead of varchar. Try replacing your charindex expression with this:
charindex(',' + rtrim(ro_reason_code), #reasonCode)
When I used a type of char(2) in the table and char(16) for the #reasonCode, I could reproduce your result, and I found that adding the rtrim fixed the problem. But unfortunately I can't explain exactly what's going here, why having ',1' at the end of the string should work without the trim whereas having it at the beginning does not. Hopefully someone can provide a more in-depth answer that gets into the "why," but I thought I'd still post this for the time being to get you running.
Reproduction:
-- Forgive the "hackish" way of populating this table. I'm assuming sysobjects has >=1015 records.
declare #Code table (ro_reason_code char(2));
insert #Code select top 625 '1' from sysobjects;
insert #Code select top 1015 '01' from sysobjects;
declare #reasonCode char(16);
set #reasonCode = ',1,01';
select count(1) from #Code where #reasonCode = 'ALL' or charindex(',' + ro_reason_code, #reasonCode) > 0; -- Result: 1015
select count(1) from #Code where #reasonCode = 'ALL' or charindex(',' + rtrim(ro_reason_code), #reasonCode) > 0; -- Result: 1640
set #reasonCode = ',01,1';
select count(1) from #Code where #reasonCode = 'ALL' or charindex(',' + ro_reason_code, #reasonCode) > 0; -- Result: 1640
select count(1) from #Code where #reasonCode = 'ALL' or charindex(',' + rtrim(ro_reason_code), #reasonCode) > 0; -- Result: 1640
Because you are using char, which is a fixed length field, your data is stored padded out to the length of the field. So '1' is stored as '1 '
DECLARE #Code CHAR(2)
SET #Code = '1'
SELECT '''' + #Code + ''''
-- Printes '1 '
For that reason, when you add ',' to the value, you now have ',1 ' (notice the trailing whitespace)
DECLARE #Code CHAR(2)
SET #Code = '1'
SELECT '''' + ',' + #Code + ''''
-- prints ',1 '
Now if you're comparing off another char field, there will also be padded whitespace if the character data is less than the length of the field. So what appers to be ',11,1' is actually something like ',11,1 ' which does match the pattern of ',1 '
BUT, when you reverse the order, ',1,11' becomes ',1,11 ' which does not match the pattern of ',1 '
Unrelated
I just want to point out there is a subtle issue with the implementation. By only appending the leading comma, you may get false positives depending on your data. For example, ,2 will match the pattern ,25.
,2 does match 1,11,25,A01
You've gotta append the comma on both sides of each side of the evaluation.
CHARINDEX( ',' + RTRIM(ro_reason_code) + ',',
',' + RTRIM(#reasonCode) + ',') > 0
So to illustrate the difference it becomes
,2, does not match ,1,11,25,A01,
In a SqlServer database I use, the database name is something like StackExchange.Audio.Meta, or StackExchange.Audio or StackOverflow . By sheer luck this is also the url for a website. I only need split it on the dots and reverse it: meta.audio.stackexchange. Adding http:// and .com and I'm done. Obviously Stackoverflow doesn't need any reversing.
Using the SqlServer 2016 string_split function I can easy split and reorder its result:
select value
from string_split(db_name(),'.')
order by row_number() over( order by (select 1)) desc
This gives me
| Value |
-----------------
| Meta |
| Audio |
| StackExchange |
As I need to have the url in a variable I hoped to concatenate it using this answer so my attempt looks like this:
declare #revname nvarchar(150)
select #revname = coalesce(#revname +'.','') + value
from string_split(db_name(),'.')
order by row_number() over( order by (select 1)) desc
However this only returns me the last value, StackExchange. I already noticed the warnings on that answer that this trick only works for certain execution plans as explained here.
The problem seems to be caused by the order by clause. Without that I get all values, but then in the wrong order. I tried to a add ltrimand rtrim function as suggested in the Microsoft article as well as a subquery but so far without luck.
Is there a way I can nudge the Sql Server 2016 Query Engine to concatenate the ordered result from that string_split in a variable?
I do know I can use for XML or even a plain cursor to get the result I need but I don't want to give up this elegant solution yet.
As I'm running this on the Stack Exchange Data Explorer I can't use functions, as we lack the permission to create those. I can do Stored procedures but I hoped I could evade those.
I prepared a SEDE Query to experiment with. The database names to expect are either without dots, aka StackOverflow, with 1 dot: StackOverflow.Meta or 2 dots, `StackExchange.Audio.Meta, the full list of databases is here
I think you are over-complicating things. You could use PARSENAME:
SELECT 'http://' + PARSENAME(db_name(),1) +
ISNULL('.' + PARSENAME(db_name(),2),'') + ISNULL('.'+PARSENAME(db_name(),3),'')
+ '.com'
This is exactly why I have the Presentation Sequence (PS) in my split function. People often scoff at using a UDF for such items, but it is generally a one-time hit to parse something for later consumption.
Select * from [dbo].[udf-Str-Parse]('meta.audio.stackexchange','.')
Returns
Key_PS Key_Value
1 meta
2 audio
3 stackexchange
The UDF
CREATE FUNCTION [dbo].[udf-Str-Parse] (#String varchar(max),#delimeter varchar(10))
--Usage: Select * from [dbo].[udf-Str-Parse]('meta.audio.stackexchange','.')
-- Select * from [dbo].[udf-Str-Parse]('John Cappelletti was here',' ')
-- Select * from [dbo].[udf-Str-Parse]('id26,id46|id658,id967','|')
Returns #ReturnTable Table (Key_PS int IDENTITY(1,1) NOT NULL , Key_Value varchar(max))
As
Begin
Declare #intPos int,#SubStr varchar(max)
Set #IntPos = CharIndex(#delimeter, #String)
Set #String = Replace(#String,#delimeter+#delimeter,#delimeter)
While #IntPos > 0
Begin
Set #SubStr = Substring(#String, 0, #IntPos)
Insert into #ReturnTable (Key_Value) values (#SubStr)
Set #String = Replace(#String, #SubStr + #delimeter, '')
Set #IntPos = CharIndex(#delimeter, #String)
End
Insert into #ReturnTable (Key_Value) values (#String)
Return
End
Probably less elegant solution but it takes only a few lines and works with any number of dots.
;with cte as (--build xml
select 1 num, cast('<str><s>'+replace(db_name(),'.','</s><s>')+'</s></str>' as xml) str
)
,x as (--make table from xml
select row_number() over(order by num) rn, --add numbers to sort later
t.v.value('.[1]','varchar(50)') s
from cte cross apply cte.str.nodes('str/s') t(v)
)
--combine into string
select STUFF((SELECT '.' + s AS [text()]
FROM x
order by rn desc --in reverse order
FOR XML PATH('')
), 1, 1, '' ) name
Is there a way I can nudge the Sql Server 2016 Query Engine to concatenate the ordered result from that string_split in a variable?
You can just use CONCAT:
DECLARE #URL NVARCHAR(MAX)
SELECT #URL = CONCAT(value, '.', #URL) FROM STRING_SPLIT(DB_NAME(), '.')
SET #URL = CONCAT('http://', LOWER(#URL), 'com');
The reversal is accomplished by the order of parameters to CONCAT. Here's an example.
It changes StackExchange.Garage.Meta to http://meta.garage.stackexchange.com.
This can be used to split and reverse strings in general, but note that it does leave a trailing delimiter. I'm sure you could add some logic or a COALESCE in there to make that not happen.
Also note that vNext will be adding STRING_AGG.
To answer the 'X' of this XY problem, and to address the HTTPS switch (especially for Meta sites) and some other site name changes, I've written the following SEDE query which outputs all site names in the format used on the network site list.
SELECT name,
LOWER('https://' +
IIF(PATINDEX('%.Mathoverflow%', name) > 0,
IIF(PATINDEX('%.Meta', name) > 0, 'meta.mathoverflow.net', 'mathoverflow.net'),
IIF(PATINDEX('%.Ubuntu%', name) > 0,
IIF(PATINDEX('%.Meta', name) > 0, 'meta.askubuntu.com', 'askubuntu.com'),
IIF(PATINDEX('StackExchange.%', name) > 0,
CASE SUBSTRING(name, 15, 200)
WHEN 'Audio' THEN 'video'
WHEN 'Audio.Meta' THEN 'video.meta'
WHEN 'Beer' THEN 'alcohol'
WHEN 'Beer.Meta' THEN 'alcohol.meta'
WHEN 'CogSci' THEN 'psychology'
WHEN 'CogSci.Meta' THEN 'psychology.meta'
WHEN 'Garage' THEN 'mechanics'
WHEN 'Garage.Meta' THEN 'mechanics.meta'
WHEN 'Health' THEN 'medicalsciences'
WHEN 'Health.Meta' THEN 'medicalsciences.meta'
WHEN 'Moderators' THEN 'communitybuilding'
WHEN 'Moderators.Meta' THEN 'communitybuilding.meta'
WHEN 'Photography' THEN 'photo'
WHEN 'Photography.Meta' THEN 'photo.meta'
WHEN 'Programmers' THEN 'softwareengineering'
WHEN 'Programmers.Meta' THEN 'softwareengineering.meta'
WHEN 'Vegetarian' THEN 'vegetarianism'
WHEN 'Vegetarian.Meta' THEN 'vegetarianism.meta'
WHEN 'Writers' THEN 'writing'
WHEN 'Writers.Meta' THEN 'writing.meta'
ELSE SUBSTRING(name, 15, 200)
END + '.stackexchange.com',
IIF(PATINDEX('StackOverflow.%', name) > 0,
CASE SUBSTRING(name, 15, 200)
WHEN 'Br' THEN 'pt'
WHEN 'Br.Meta' THEN 'pt.meta'
ELSE SUBSTRING(name, 15, 200)
END + '.stackoverflow.com',
IIF(PATINDEX('%.Meta', name) > 0,
'meta.' + SUBSTRING(name, 0, PATINDEX('%.Meta', name)) + '.com',
name + '.com'
)
)
)
)
) + '/'
)
FROM sys.databases WHERE database_id > 5
I have a field in my table which has multiple reason codes concatenated in 1 column.
e.g. 2 records
Reason_Codes
Record1: 001,002,004,009,010
Record2: 001,003,005,006
In my SSRS report the user will be searching for data using one of the above reason codes. e.g.
001 will retrieve both records.
005 will retrieve the second record
and so on.
Kindly advise how this can be achieved using SQL or Stored Procedure.
Many thanks.
If you are just passing in a single Reason Code to search on, you don't even need to bother with splitting the comma-separated list: you can just use a LIKE clause as follows:
SELECT tb.field1, tb.field2
FROM SchemaName.TableName tb
WHERE ',' + tb.Reason_Codes + ',' LIKE '%,' + #ReasonCode + ',%';
Try the following to see:
DECLARE #Bob TABLE (ID INT IDENTITY(1, 1) NOT NULL, ReasonCodes VARCHAR(50));
INSERT INTO #Bob (ReasonCodes) VALUES ('001,002,004,009,010');
INSERT INTO #Bob (ReasonCodes) VALUES ('001,003,005,006');
DECLARE #ReasonCode VARCHAR(5);
SET #ReasonCode = '001';
SELECT tb.ID, tb.ReasonCodes
FROM #Bob tb
WHERE ',' + tb.ReasonCodes + ',' LIKE '%,' + #ReasonCode + ',%';
-- returns both rows
SET #ReasonCode = '005';
SELECT tb.ID, tb.ReasonCodes
FROM #Bob tb
WHERE ',' + tb.ReasonCodes + ',' LIKE '%,' + #ReasonCode + ',%';
-- returns only row #2
I have blogged about something like this a long time ago. May be this will help: http://dotnetinternal.blogspot.com/2013/10/comma-separated-to-temp-table.html
The core solution would be to convert the comma separated values into a temporary table and then do a simple query on the temporary table to get your desired result.