Related
I hope you are all well.
I would like your help on a data transformation task that I have.
I would like to convert the first row of a table to a column name
I am working on SQL Server Azure and I get daily data from another service.
This service loads a table that is of the same form.
and I would like to transform the data in the same manner
Do You have any idea how to do it ?
The way to solve this is by using a little dynamic SQL magic:
First, create and populate sample table (Please save us thus step in your future questions):
DECLARE #T As Table
(
Row_num int,
Line nvarchar(4000)
);
INSERT INTO #T (Row_Num, Line) VALUES
(1, 'Col1;Col2;Col3'),
(2, 'Val1;Val2;Val3'),
(3, 'Value1;Value2;Value1'),
(4, 'Val A; val B;Val A'),
(5, 'Value A; Value B;Value C');
Then, build a union all query that selects the values from every row but the first, replacing the semicolon (;) separator with a comma (,) surrounded by apostrophes ('). Add an apostrophe before and after the string (which means we are treating all the data as strings):
DECLARE #Sql nvarchar(max) = '';
SELECT #Sql += 'UNION ALL SELECT '''+ REPLACE(Line, ';', ''',''') + ''' '
FROM #T
WHERE Row_Num > 1;
Next, use stuff to replace the first UNION ALL with a common table expression declaration, specifying the column names in the declaration itself. Note that here we don't need the apostrophes anymore, just to replace the semicolon with a comma:
SELECT #Sql = STUFF(#Sql, 1, 10, 'WITH CTE('+ REPLACE(Line, ';', ',') +') AS (') + ') SELECT * FROM CTE'
FROM #T
WHERE Row_Num = 1;
Finally, execute the sql:
EXEC(#Sql)
Results:
Col1 Col2 Col3
Val1 Val2 Val3
Value1 Value2 Value1
Val A val B Val A
Value A Value B Value C
You can see a live demo on rextester.
Another possible approach is to transform your text data into valid JSON arrays and then use OPENJSON() with an explicit schema and dynamic statement.
Working example:
Input:
CREATE TABLE #Data (
RowNum int,
Line nvarchar(max)
)
INSERT INTO #Data
(RowNum, Line)
VALUES
(1, 'ColumnA;ColumnB;ColumnC'),
(2, 'ValueA1;ValueB1;ValueC1'),
(3, 'ValueA2;ValueB2;ValueC2'),
(4, 'ValueA3;ValueB3;ValueC3'),
(5, 'ValueA4;ValueB4;ValueC4'),
(6, 'ValueA5;ValueB5;ValueC5')
T-SQL:
-- Explicit schema generation
DECLARE #schema nvarchar(max)
SELECT #schema = STUFF((
SELECT CONCAT(N',', j.[value], N' nvarchar(max) ''$[', j.[key], N']''')
FROM #Data d
CROSS APPLY OPENJSON(CONCAT(N'["', REPLACE(d.Line, ';', '","'), N'"]')) j
WHERE d.RowNum = 1
FOR XML PATH('')
), 1, 1, N'')
-- Dymanic statement
DECLARE #stm nvarchar(max)
SET #stm = CONCAT(
N'SELECT j.* FROM #Data d ',
N'CROSS APPLY OPENJSON(CONCAT(N''[["'', REPLACE(d.Line, '';'', ''","''), N''"]]'')) ',
N'WITH (',
#schema,
N') j WHERE d.RowNum > 1'
)
-- Execution
EXEC sp_executesql #stm
Output:
-----------------------
ColumnA ColumnB ColumnC
-----------------------
ValueA1 ValueB1 ValueC1
ValueA2 ValueB2 ValueC2
ValueA3 ValueB3 ValueC3
ValueA4 ValueB4 ValueC4
ValueA5 ValueB5 ValueC5
Explanations:
The main part is to transform each row's data into valid JSON arrays. The count of the columns can be different.
Data from the first row will be used for explicit schema generation and values ColumnA;ColumnB;ColumnC are transformed into ["ColumnA","ColumnB","ColumnC"]. Values from subsequent rows ValueA1;ValueB1;ValueC1 are transformed into [["ValueA1","ValueB1","ValueC1"]].
Next simple examples demonstrate how OPENJSON() returns data with default and explicit schema:
With default schema:
DECLARE #json nvarchar(max)
SET #json = '["ValueA1", "ValueB1", "ValueC1"]'
SELECT *
FROM OPENJSON(#json)
Output for default schema:
----------------
key value type
----------------
0 ValueA1 1
1 ValueB1 1
2 ValueC1 1
With explicit schema:
SET #json = '[["ValueA1", "ValueB1", "ValueC1"]]'
SELECT *
FROM OPENJSON(#json)
WITH (
ColumnA nvarchar(max) '$[0]',
ColumnB nvarchar(max) '$[1]',
ColumnC nvarchar(max) '$[2]'
)
Output for explicit schema:
-----------------------
ColumnA ColumnB ColumnC
-----------------------
ValueA1 ValueB1 ValueC1
Following is the stored procedure
ALTER PROCEDURE [dbo].[get_data_Dyna]
{
#param1 varchar(max) = null,
#param2 varchar(max) = null,
#start varchar(max) = null,
#end varchar(max) = null
}
AS
SELECT * from table where
(#param1 IS NULL OR column1 IN (SELECT data FROM dbo.delimited_list_to_table(#param1,',')))
AND (#param2 IS NULL OR column2 IN (SELECT data FROM dbo.delimited_list_to_table(#param2,',')))
AND ....?????
How this is working :
All the parameters can be comma seperated
#param1 value can be 'Germany' or 'Germany,USA' or null. This is working as expected.
Same goes for #param2
I'm trying to include rest of the parameters which is expected to work as follows :
#start='0' and #end='100' : In this case, where clause will look like this
...AND val BETWEEN #start AND #end
#start='48,60' and #end='51,99' : In this case, where clause will look like this
...AND ((val Between 48 and 51) or (val Between 60 and 99))
#start='48,60,75' and #end='51,99,203' : In this case, where clause will look like this
...AND ((val Between 48 and 51) or (val Between 60 and 99) or (val Between 75 and 203))
I'm unable to include above 2nd/3rd point correctly. I tried to write it dynamically which is working for single values [Point 1], but how to write point 2/3 ?
Any help is greatly appreciated.
Ok, i think the best approach here would be to use temp tables or table variable.
Lets go with temp tables.
create table #StartEnd (start int not null, end int not null, primary key (start,end))
then we insert from #start and #end into it using dbo.delimited_list_to_table. Now i am not sure about your implementation of it, so i will assume the values are numbered
insert into #StartEnd
select starts.data, ends.data
from dbo.delimited_list_to_table(#start,',') as starts
join dbo.delimited_list_to_table(#end,',') as ends
on starts.index = ends.index
Now we have to filter the values. Two approaches. Join or Exists condition
...
join #StartEnd on val between start and end
...
and exists (select 1 from #StartEnd where val between start and end)
Hope this helps
there you go. The comments / explainations are within the query
-- create a sample table
declare #tbl table
(
val int
)
-- put in some sample data
insert into #tbl
values (48), (60), (51), (99), (75), (203)
-- these are the input parameter
declare #start varchar(100),
#end varchar(100)
-- and these are the input value
select #start = '48,60,75',
#end = '51,99,203'
-- the actual query
; with
start_end as
(
-- here i am using [DelimitedSplit8K][1]
select s = s.Item, e = e.Item
from dbo.[DelimitedSplit8K](#start, ',') s
inner join dbo.[DelimitedSplit8K](#end, ',') e
on s.ItemNumber = e.ItemNumber
)
select t.val
from #tbl t
where exists
(
select *
from start_end x
where t.val between x.s and x.e
)
you can get it here DelimitedSplit8K
Sample input (from our understanding, we guess the your data):
select
* into ##delimit
from (
values
(1 ,'Ger','Ind', 100 )
,(2 ,'Ind',Null, 10 )
,(3 ,'Ger',Null, 24 )
,(4 ,'Ind','Ger', 54 )
,(5 ,'USA','Ind', 56 )
,(6 ,Null,'USA', 75 )-- NULL. But USA is three time came.
,(7 ,'USA','USA', 60 )-- same country with diff val.
,(8 ,'USA','USA', 80 )-- same country with diff val.
) demilit(Id,FromPr,ToPr,Val)
select * from ##delimit
Procedure (you just use this instead of your procedure):
CREATE PROCEDURE [dbo].[get_data_Dyna]
(#param1 varchar(max) = NULL,
#param2 varchar(max) = NULL,
#start varchar(max) = NULL,
#end varchar(max) = NULL)
AS
BEGIN
SELECT *
FROM ##delimit d
JOIN
( --| Here we check the val btw #start and #end
SELECT DISTINCT
s.FinalColumn StartVal, --|
e.FinalColumn EndVal --|
FROM
dbo.WithoutDelimit (#start, ',') s --| S means 'Start'
JOIN
(SELECT *
FROM dbo.WithoutDelimit (#end, ',')) e ON s.id = e.id --| E means 'End'
) se --| se mean StartEnd
ON d.val BETWEEN se.StartVal AND se.EndVal --| Here YOUR CONDITION is accomplished
WHERE
( -- | checks whether
frompr IN -- | column1 in #param1 or not
(SELECT FinalColumn FROM dbo.WithoutDelimit (#param1,',') -- | frompr means, 'column1'
) OR #param1 is NULL -- |
)
and ( -- | checks whether
ToPr in ( -- | column2 in #param2 or not
select FinalColumn from dbo.WithoutDelimit (#param2,',') -- | frompr means, 'column2'
) or #param2 is null -- |
)
end
Call stored procedure:
[get_data_Dyna] null,'usa','75','100,' -- 6 rows
[get_data_Dyna] 'Ind,Ger',null,'1,15','20,30' --2 and 3 rows are selected.
[get_data_Dyna] 'usa','usa','50,60','55,79'
-- 7 and 8 has same country. But due to Val, 8 has been rejected.
[get_data_Dyna] NULL,'usa','70,60','80,79'
-- 6 and 7 and 8 has been selected. Due to val condition.
Function (called from the stored procedure):
alter function WithoutDelimit -- We use one function for all conditions.
(#Parameter varchar (max),
#delimit varchar (1))
returns #FinalTable table (
Id int identity (1,1) -- Auto increment
, FinalColumn varchar (max) -- It returns the values as a column.
) as
begin
;with cte as -- recursive cte.
(
select convert (varchar (255), #Parameter + #delimit) con
, convert (varchar (255), #Parameter + #delimit) want
union all
select convert (varchar (255), stuff (con, 1, CHARINDEX (#demilit,con),'') )
, substring (con, 1, CHARINDEX (#delimit, con) - 1)
from cte
where con <> ''
) insert into #FinalTable (FinalColumn)
select want from cte
where con <> want
return
end
Revert us, if query need update.
How do I get:
id Name Value
1 A 4
1 B 8
2 C 9
to
id Column
1 A:4, B:8
2 C:9
No CURSOR, WHILE loop, or User-Defined Function needed.
Just need to be creative with FOR XML and PATH.
[Note: This solution only works on SQL 2005 and later. Original question didn't specify the version in use.]
CREATE TABLE #YourTable ([ID] INT, [Name] CHAR(1), [Value] INT)
INSERT INTO #YourTable ([ID],[Name],[Value]) VALUES (1,'A',4)
INSERT INTO #YourTable ([ID],[Name],[Value]) VALUES (1,'B',8)
INSERT INTO #YourTable ([ID],[Name],[Value]) VALUES (2,'C',9)
SELECT
[ID],
STUFF((
SELECT ', ' + [Name] + ':' + CAST([Value] AS VARCHAR(MAX))
FROM #YourTable
WHERE (ID = Results.ID)
FOR XML PATH(''),TYPE).value('(./text())[1]','VARCHAR(MAX)')
,1,2,'') AS NameValues
FROM #YourTable Results
GROUP BY ID
DROP TABLE #YourTable
If it is SQL Server 2017 or SQL Server Vnext, SQL Azure you can use STRING_AGG as below:
SELECT id, STRING_AGG(CONCAT(name, ':', [value]), ', ')
FROM #YourTable
GROUP BY id
using XML path will not perfectly concatenate as you might expect... it will replace "&" with "&" and will also mess with <" and ">
...maybe a few other things, not sure...but you can try this
I came across a workaround for this... you need to replace:
FOR XML PATH('')
)
with:
FOR XML PATH(''),TYPE
).value('(./text())[1]','VARCHAR(MAX)')
...or NVARCHAR(MAX) if thats what youre using.
why the hell doesn't SQL have a concatenate aggregate function? this is a PITA.
I ran into a couple of problems when I tried converting Kevin Fairchild's suggestion to work with strings containing spaces and special XML characters (&, <, >) which were encoded.
The final version of my code (which doesn't answer the original question but may be useful to someone) looks like this:
CREATE TABLE #YourTable ([ID] INT, [Name] VARCHAR(MAX), [Value] INT)
INSERT INTO #YourTable ([ID],[Name],[Value]) VALUES (1,'Oranges & Lemons',4)
INSERT INTO #YourTable ([ID],[Name],[Value]) VALUES (1,'1 < 2',8)
INSERT INTO #YourTable ([ID],[Name],[Value]) VALUES (2,'C',9)
SELECT [ID],
STUFF((
SELECT ', ' + CAST([Name] AS VARCHAR(MAX))
FROM #YourTable WHERE (ID = Results.ID)
FOR XML PATH(''),TYPE
/* Use .value to uncomment XML entities e.g. > < etc*/
).value('.','VARCHAR(MAX)')
,1,2,'') as NameValues
FROM #YourTable Results
GROUP BY ID
DROP TABLE #YourTable
Rather than using a space as a delimiter and replacing all the spaces with commas, it just pre-pends a comma and space to each value then uses STUFF to remove the first two characters.
The XML encoding is taken care of automatically by using the TYPE directive.
Another option using Sql Server 2005 and above
---- test data
declare #t table (OUTPUTID int, SCHME varchar(10), DESCR varchar(10))
insert #t select 1125439 ,'CKT','Approved'
insert #t select 1125439 ,'RENO','Approved'
insert #t select 1134691 ,'CKT','Approved'
insert #t select 1134691 ,'RENO','Approved'
insert #t select 1134691 ,'pn','Approved'
---- actual query
;with cte(outputid,combined,rn)
as
(
select outputid, SCHME + ' ('+DESCR+')', rn=ROW_NUMBER() over (PARTITION by outputid order by schme, descr)
from #t
)
,cte2(outputid,finalstatus,rn)
as
(
select OUTPUTID, convert(varchar(max),combined), 1 from cte where rn=1
union all
select cte2.outputid, convert(varchar(max),cte2.finalstatus+', '+cte.combined), cte2.rn+1
from cte2
inner join cte on cte.OUTPUTID = cte2.outputid and cte.rn=cte2.rn+1
)
select outputid, MAX(finalstatus) from cte2 group by outputid
Install the SQLCLR Aggregates from http://groupconcat.codeplex.com
Then you can write code like this to get the result you asked for:
CREATE TABLE foo
(
id INT,
name CHAR(1),
Value CHAR(1)
);
INSERT INTO dbo.foo
(id, name, Value)
VALUES (1, 'A', '4'),
(1, 'B', '8'),
(2, 'C', '9');
SELECT id,
dbo.GROUP_CONCAT(name + ':' + Value) AS [Column]
FROM dbo.foo
GROUP BY id;
Eight years later... Microsoft SQL Server vNext Database Engine has finally enhanced Transact-SQL to directly support grouped string concatenation. The Community Technical Preview version 1.0 added the STRING_AGG function and CTP 1.1 added the WITHIN GROUP clause for the STRING_AGG function.
Reference: https://msdn.microsoft.com/en-us/library/mt775028.aspx
SQL Server 2005 and later allow you to create your own custom aggregate functions, including for things like concatenation- see the sample at the bottom of the linked article.
This is just an addition to Kevin Fairchild's post (very clever by the way). I would have added it as a comment, but I don't have enough points yet :)
I was using this idea for a view I was working on, however the items I was concatinating contained spaces. So I modified the code slightly to not use spaces as delimiters.
Again thanks for the cool workaround Kevin!
CREATE TABLE #YourTable ( [ID] INT, [Name] CHAR(1), [Value] INT )
INSERT INTO #YourTable ([ID], [Name], [Value]) VALUES (1, 'A', 4)
INSERT INTO #YourTable ([ID], [Name], [Value]) VALUES (1, 'B', 8)
INSERT INTO #YourTable ([ID], [Name], [Value]) VALUES (2, 'C', 9)
SELECT [ID],
REPLACE(REPLACE(REPLACE(
(SELECT [Name] + ':' + CAST([Value] AS VARCHAR(MAX)) as A
FROM #YourTable
WHERE ( ID = Results.ID )
FOR XML PATH (''))
, '</A><A>', ', ')
,'<A>','')
,'</A>','') AS NameValues
FROM #YourTable Results
GROUP BY ID
DROP TABLE #YourTable
An example would be
In Oracle you can use LISTAGG aggregate function.
Original records
name type
------------
name1 type1
name2 type2
name2 type3
Sql
SELECT name, LISTAGG(type, '; ') WITHIN GROUP(ORDER BY name)
FROM table
GROUP BY name
Result in
name type
------------
name1 type1
name2 type2; type3
This kind of question is asked here very often, and the solution is going to depend a lot on the underlying requirements:
https://stackoverflow.com/search?q=sql+pivot
and
https://stackoverflow.com/search?q=sql+concatenate
Typically, there is no SQL-only way to do this without either dynamic sql, a user-defined function, or a cursor.
Just to add to what Cade said, this is usually a front-end display thing and should therefore be handled there. I know that sometimes it's easier to write something 100% in SQL for things like file export or other "SQL only" solutions, but most of the times this concatenation should be handled in your display layer.
Don't need a cursor... a while loop is sufficient.
------------------------------
-- Setup
------------------------------
DECLARE #Source TABLE
(
id int,
Name varchar(30),
Value int
)
DECLARE #Target TABLE
(
id int,
Result varchar(max)
)
INSERT INTO #Source(id, Name, Value) SELECT 1, 'A', 4
INSERT INTO #Source(id, Name, Value) SELECT 1, 'B', 8
INSERT INTO #Source(id, Name, Value) SELECT 2, 'C', 9
------------------------------
-- Technique
------------------------------
INSERT INTO #Target (id)
SELECT id
FROM #Source
GROUP BY id
DECLARE #id int, #Result varchar(max)
SET #id = (SELECT MIN(id) FROM #Target)
WHILE #id is not null
BEGIN
SET #Result = null
SELECT #Result =
CASE
WHEN #Result is null
THEN ''
ELSE #Result + ', '
END + s.Name + ':' + convert(varchar(30),s.Value)
FROM #Source s
WHERE id = #id
UPDATE #Target
SET Result = #Result
WHERE id = #id
SET #id = (SELECT MIN(id) FROM #Target WHERE #id < id)
END
SELECT *
FROM #Target
Let's get very simple:
SELECT stuff(
(
select ', ' + x from (SELECT 'xxx' x union select 'yyyy') tb
FOR XML PATH('')
)
, 1, 2, '')
Replace this line:
select ', ' + x from (SELECT 'xxx' x union select 'yyyy') tb
With your query.
You can improve performance significant the following way if group by contains mostly one item:
SELECT
[ID],
CASE WHEN MAX( [Name]) = MIN( [Name]) THEN
MAX( [Name]) NameValues
ELSE
STUFF((
SELECT ', ' + [Name] + ':' + CAST([Value] AS VARCHAR(MAX))
FROM #YourTable
WHERE (ID = Results.ID)
FOR XML PATH(''),TYPE).value('(./text())[1]','VARCHAR(MAX)')
,1,2,'') AS NameValues
END
FROM #YourTable Results
GROUP BY ID
didn't see any cross apply answers, also no need for xml extraction. Here is a slightly different version of what Kevin Fairchild wrote. It's faster and easier to use in more complex queries:
select T.ID
,MAX(X.cl) NameValues
from #YourTable T
CROSS APPLY
(select STUFF((
SELECT ', ' + [Name] + ':' + CAST([Value] AS VARCHAR(MAX))
FROM #YourTable
WHERE (ID = T.ID)
FOR XML PATH(''))
,1,2,'') [cl]) X
GROUP BY T.ID
Using the Stuff and for xml path operator to concatenate rows to string :Group By two columns -->
CREATE TABLE #YourTable ([ID] INT, [Name] CHAR(1), [Value] INT)
INSERT INTO #YourTable ([ID],[Name],[Value]) VALUES (1,'A',4)
INSERT INTO #YourTable ([ID],[Name],[Value]) VALUES (1,'B',8)
INSERT INTO #YourTable ([ID],[Name],[Value]) VALUES (1,'B',5)
INSERT INTO #YourTable ([ID],[Name],[Value]) VALUES (2,'C',9)
-- retrieve each unique id and name columns and concatonate the values into one column
SELECT
[ID],
STUFF((
SELECT ', ' + [Name] + ':' + CAST([Value] AS VARCHAR(MAX)) -- CONCATONATES EACH APPLICATION : VALUE SET
FROM #YourTable
WHERE (ID = Results.ID and Name = results.[name] )
FOR XML PATH(''),TYPE).value('(./text())[1]','VARCHAR(MAX)')
,1,2,'') AS NameValues
FROM #YourTable Results
GROUP BY ID
SELECT
[ID],[Name] , --these are acting as the group by clause
STUFF((
SELECT ', '+ CAST([Value] AS VARCHAR(MAX)) -- CONCATONATES THE VALUES FOR EACH ID NAME COMBINATION
FROM #YourTable
WHERE (ID = Results.ID and Name = results.[name] )
FOR XML PATH(''),TYPE).value('(./text())[1]','VARCHAR(MAX)')
,1,2,'') AS NameValues
FROM #YourTable Results
GROUP BY ID, name
DROP TABLE #YourTable
Using Replace Function and FOR JSON PATH
SELECT T3.DEPT, REPLACE(REPLACE(T3.ENAME,'{"ENAME":"',''),'"}','') AS ENAME_LIST
FROM (
SELECT DEPT, (SELECT ENAME AS [ENAME]
FROM EMPLOYEE T2
WHERE T2.DEPT=T1.DEPT
FOR JSON PATH,WITHOUT_ARRAY_WRAPPER) ENAME
FROM EMPLOYEE T1
GROUP BY DEPT) T3
For sample data and more ways click here
If you have clr enabled you could use the Group_Concat library from GitHub
Another example without the garbage: ",TYPE).value('(./text())[1]','VARCHAR(MAX)')"
WITH t AS (
SELECT 1 n, 1 g, 1 v
UNION ALL
SELECT 2 n, 1 g, 2 v
UNION ALL
SELECT 3 n, 2 g, 3 v
)
SELECT g
, STUFF (
(
SELECT ', ' + CAST(v AS VARCHAR(MAX))
FROM t sub_t
WHERE sub_t.g = main_t.g
FOR XML PATH('')
)
, 1, 2, ''
) cg
FROM t main_t
GROUP BY g
Input-output is
************************* -> *********************
* n * g * v * * g * cg *
* - * - * - * * - * - *
* 1 * 1 * 1 * * 1 * 1, 2 *
* 2 * 1 * 2 * * 2 * 3 *
* 3 * 2 * 3 * *********************
*************************
I used this approach which may be easier to grasp. Get a root element, then concat to choices any item with the same ID but not the 'official' name
Declare #IdxList as Table(id int, choices varchar(max),AisName varchar(255))
Insert into #IdxLIst(id,choices,AisName)
Select IdxId,''''+Max(Title)+'''',Max(Title) From [dbo].[dta_Alias]
where IdxId is not null group by IdxId
Update #IdxLIst
set choices=choices +','''+Title+''''
From #IdxLIst JOIN [dta_Alias] ON id=IdxId And Title <> AisName
where IdxId is not null
Select * from #IdxList where choices like '%,%'
For all my healthcare folks out there:
SELECT
s.NOTE_ID
,STUFF ((
SELECT
[note_text] + ' '
FROM
HNO_NOTE_TEXT s1
WHERE
(s1.NOTE_ID = s.NOTE_ID)
ORDER BY [line] ASC
FOR XML PATH(''),TYPE).value('(./text())[1]','VARCHAR(MAX)')
,
1,
2,
'') AS NOTE_TEXT_CONCATINATED
FROM
HNO_NOTE_TEXT s
GROUP BY NOTE_ID
I am trying to get information for products that have an ID that is contained in a list. The problem is that the list contains some single values and some range values:
PX03 - PX069, PX20, PX202, PX25 - PX270, PX250 - PX2509, PX251, PX2511 -
PX2513
Basically what I am looking for is some way to take a list or string containing both values and ranges and the end output is a table or list that has all of the values within the ranges individually so that I can loop through them.
I have a stored procedure that loops through all the ID's in the main products table that use the 'PX' prefix, but the table has all ids (i.e. PX 1 - 9999, LX 00001 - 99999) and I only want to search through those contained in the above list. I could write it out all the id's individually but some of the ranges contain many values, which would be time consuming to go through.
My idea was to create a separate table containing this list, in which there would be three columns: an identity column, and then one column each for the beginning and end of the range. Any items that do not have a range would just have the same value for beginning and end range, i.e.:
----------------------------------
rownum | range_start | range_end|
----------------------------------
1 PX03 PX069
2 PX20 PX20
3 PX202 PX202
4 PX25 PX25
5 PX250 PX2509
and then populating a table using something like:
SELECT id from product_table
WHERE id BETWEEN listtable.range_start AND listtable.range_end
where product_table is my original table with the product id's and their information and listtable is the new table I just created. This would give me:
id|
---
PX03
PX030
PX031
PX032
PX033
.
.
.
PX067
PX068
PX069
PX20
PX202
PX25
PX250
PX251
etc.
but I am thinking I would need to iterate through the list and I am not sure how to do that. Any ideas, hints or suggestions?
UPDATE
After creating the table using the solution given by #asantaballa, it was as simple as using an inner join:
SELECT d.id
FROM product_table d
INNER JOIN #RangeTable r
ON d.id BETWEEN r.RangeFrom AND r.RangeTo
See if this works for you for the part about converting the string to a table.
Declare #StrList Varchar(1000) = 'PX03 - PX069, PX20, PX202, PX25 - PX270, PX250 - PX2509, PX251, PX2511 - PX2513'
Declare #RangeTable Table (RangeFrom VarChar(32), RangeTo VarChar(32))
Select #StrList = Replace(#StrList,' ', '') + ','
Declare #StrListItem Varchar(32)
While CHARINDEX(',', #StrList) > 0
Begin
Select #StrListItem = SUBSTRING(#StrList,1,CHARINDEX(',', #StrList) - 1)
Declare
#RangeFrom VarChar(32)
, #RangeTo VarChar(32)
If CHARINDEX('-', #StrListItem) = 0
Begin
Select
#RangeFrom = #StrListItem
, #RangeTo = #StrListItem
End
Else
Begin
Select
#RangeFrom = SUBSTRING(#StrListItem, 1, CHARINDEX('-', #StrListItem) - 1)
, #RangeTo = SUBSTRING(#StrListItem, CHARINDEX('-', #StrListItem) + 1, LEN(#StrListItem) - CHARINDEX('-', #StrListItem))
End
Insert Into #RangeTable (RangeFrom, RangeTo) Values (#RangeFrom, #RangeTo)
Select #StrList = SUBSTRING(#StrList, CHARINDEX(',', #StrList) + 1, LEN(#StrList) - CHARINDEX(',', #StrList))
End
Select * From #RangeTable
Here is your string and product_table
DECLARE #STR VARCHAR(100) = 'PX03 - PX069, PX20, PX202, PX25 - PX270, PX250 - PX2509, PX251, PX2511 - PX2513'
SELECT * INTO #product_table
FROM
(
SELECT 'PX4' PRODID
UNION ALL
SELECT 'PX26'
UNION ALL
SELECT 'PX75'
UNION ALL
SELECT 'PX77'
)TAB
Now create a table to hold the value
CREATE TABLE #listtable(ROWNUM int IDENTITY(1,1),range_start VARCHAR(100),range_end VARCHAR(100))
Now insert the splitted value to the table.
INSERT INTO #listtable
SELECT
ISNULL(PARSENAME(REPLACE(Split.a.value('.', 'VARCHAR(100)'),'-','.'),2),Split.a.value('.', 'VARCHAR(100)')) 'range_start' ,
PARSENAME(REPLACE(Split.a.value('.', 'VARCHAR(100)'),'-','.'),1) 'range_end'
FROM
(
SELECT CAST ('<M>' + REPLACE(#STR, ',', '</M><M>') + '</M>' AS XML) AS Data
) AS A
CROSS APPLY Data.nodes ('/M') AS Split(a)
Since Id is string, you need a function to extract numbers from Id(function created by God of SQL Server - Pinal Dave)
CREATE FUNCTION dbo.udf_GetNumeric
(#strAlphaNumeric VARCHAR(256))
RETURNS VARCHAR(256)
AS
BEGIN
DECLARE #intAlpha INT
SET #intAlpha = PATINDEX('%[^0-9]%', #strAlphaNumeric)
BEGIN
WHILE #intAlpha > 0
BEGIN
SET #strAlphaNumeric = STUFF(#strAlphaNumeric, #intAlpha, 1, '' )
SET #intAlpha = PATINDEX('%[^0-9]%', #strAlphaNumeric )
END
END
RETURN ISNULL(#strAlphaNumeric,0)
END
First of all keep in mind that we will not get PX1,PX2,PX3,PX4 if you give id BETWEEN listtable.range_start AND listtable.range_end because those are of varchar type and not numbers. So we need to extract numbers from each PX and get the values between them and append PX.
Here is the query which filters the IDs in product_table which are in the range between listtable
;WITH CTE AS
(
SELECT ROWNUM,CAST(dbo.udf_GetNumeric(range_start)AS INT) NUMBERS,
CAST(dbo.udf_GetNumeric(range_end)AS INT) RTO1
FROM #listtable
UNION ALL
SELECT T.ROWNUM,NUMBERS+1,RTO1
FROM #listtable T
JOIN CTE ON CTE.ROWNUM = T.ROWNUM
WHERE NUMBERS < RTO1
)
SELECT PRODID IDS--,ROWNUM,NUMBERS NUMS,'PX'+CAST(NUMBERS AS VARCHAR(10)) IDS2
FROM CTE
JOIN #product_table ON PRODID='PX'+CAST(NUMBERS AS VARCHAR(10))
ORDER BY NUMBERS
option (MaxRecursion 0)
SQL FIDDLE
How can rows with non-ASCII characters be returned using SQL Server?
If you can show how to do it for one column would be great.
I am doing something like this now, but it is not working
select *
from Staging.APARMRE1 as ar
where ar.Line like '%[^!-~ ]%'
For extra credit, if it can span all varchar columns in a table, that would be outstanding! In this solution, it would be nice to return three columns:
The identity field for that record. (This will allow the whole record to be reviewed with another query.)
The column name
The text with the invalid character
Id | FieldName | InvalidText |
----+-----------+-------------------+
25 | LastName | Solís |
56 | FirstName | François |
100 | Address1 | 123 Ümlaut street |
Invalid characters would be any outside the range of SPACE (3210) through ~ (12710)
Here is a solution for the single column search using PATINDEX.
It also displays the StartPosition, InvalidCharacter and ASCII code.
select line,
patindex('%[^ !-~]%' COLLATE Latin1_General_BIN,Line) as [Position],
substring(line,patindex('%[^ !-~]%' COLLATE Latin1_General_BIN,Line),1) as [InvalidCharacter],
ascii(substring(line,patindex('%[^ !-~]%' COLLATE Latin1_General_BIN,Line),1)) as [ASCIICode]
from staging.APARMRE1
where patindex('%[^ !-~]%' COLLATE Latin1_General_BIN,Line) >0
I've been running this bit of code with success
declare #UnicodeData table (
data nvarchar(500)
)
insert into
#UnicodeData
values
(N'Horse�')
,(N'Dog')
,(N'Cat')
select
data
from
#UnicodeData
where
data collate LATIN1_GENERAL_BIN != cast(data as varchar(max))
Which works well for known columns.
For extra credit, I wrote this quick script to search all nvarchar columns in a given table for Unicode characters.
declare
#sql varchar(max) = ''
,#table sysname = 'mytable' -- enter your table here
;with ColumnData as (
select
RowId = row_number() over (order by c.COLUMN_NAME)
,c.COLUMN_NAME
,ColumnName = '[' + c.COLUMN_NAME + ']'
,TableName = '[' + c.TABLE_SCHEMA + '].[' + c.TABLE_NAME + ']'
from
INFORMATION_SCHEMA.COLUMNS c
where
c.DATA_TYPE = 'nvarchar'
and c.TABLE_NAME = #table
)
select
#sql = #sql + 'select FieldName = ''' + c.ColumnName + ''', InvalidCharacter = [' + c.COLUMN_NAME + '] from ' + c.TableName + ' where ' + c.ColumnName + ' collate LATIN1_GENERAL_BIN != cast(' + c.ColumnName + ' as varchar(max)) ' + case when c.RowId <> (select max(RowId) from ColumnData) then ' union all ' else '' end + char(13)
from
ColumnData c
-- check
-- print #sql
exec (#sql)
I'm not a fan of dynamic SQL but it does have its uses for exploratory queries like this.
try something like this:
DECLARE #YourTable table (PK int, col1 varchar(20), col2 varchar(20), col3 varchar(20));
INSERT #YourTable VALUES (1, 'ok','ok','ok');
INSERT #YourTable VALUES (2, 'BA'+char(182)+'D','ok','ok');
INSERT #YourTable VALUES (3, 'ok',char(182)+'BAD','ok');
INSERT #YourTable VALUES (4, 'ok','ok','B'+char(182)+'AD');
INSERT #YourTable VALUES (5, char(182)+'BAD','ok',char(182)+'BAD');
INSERT #YourTable VALUES (6, 'BAD'+char(182),'B'+char(182)+'AD','BAD'+char(182)+char(182)+char(182));
--if you have a Numbers table use that, other wise make one using a CTE
WITH AllNumbers AS
( SELECT 1 AS Number
UNION ALL
SELECT Number+1
FROM AllNumbers
WHERE Number<1000
)
SELECT
pk, 'Col1' BadValueColumn, CONVERT(varchar(20),col1) AS BadValue --make the XYZ in convert(varchar(XYZ), ...) the largest value of col1, col2, col3
FROM #YourTable y
INNER JOIN AllNumbers n ON n.Number <= LEN(y.col1)
WHERE ASCII(SUBSTRING(y.col1, n.Number, 1))<32 OR ASCII(SUBSTRING(y.col1, n.Number, 1))>127
UNION
SELECT
pk, 'Col2' BadValueColumn, CONVERT(varchar(20),col2) AS BadValue --make the XYZ in convert(varchar(XYZ), ...) the largest value of col1, col2, col3
FROM #YourTable y
INNER JOIN AllNumbers n ON n.Number <= LEN(y.col2)
WHERE ASCII(SUBSTRING(y.col2, n.Number, 1))<32 OR ASCII(SUBSTRING(y.col2, n.Number, 1))>127
UNION
SELECT
pk, 'Col3' BadValueColumn, CONVERT(varchar(20),col3) AS BadValue --make the XYZ in convert(varchar(XYZ), ...) the largest value of col1, col2, col3
FROM #YourTable y
INNER JOIN AllNumbers n ON n.Number <= LEN(y.col3)
WHERE ASCII(SUBSTRING(y.col3, n.Number, 1))<32 OR ASCII(SUBSTRING(y.col3, n.Number, 1))>127
order by 1
OPTION (MAXRECURSION 1000);
OUTPUT:
pk BadValueColumn BadValue
----------- -------------- --------------------
2 Col1 BA¶D
3 Col2 ¶BAD
4 Col3 B¶AD
5 Col1 ¶BAD
5 Col3 ¶BAD
6 Col1 BAD¶
6 Col2 B¶AD
6 Col3 BAD¶¶¶
(8 row(s) affected)
This script searches for non-ascii characters in one column. It generates a string of all valid characters, here code point 32 to 127. Then it searches for rows that don't match the list:
declare #str varchar(128);
declare #i int;
set #str = '';
set #i = 32;
while #i <= 127
begin
set #str = #str + '|' + char(#i);
set #i = #i + 1;
end;
select col1
from YourTable
where col1 like '%[^' + #str + ']%' escape '|';
running the various solutions on some real world data - 12M rows varchar length ~30, around 9k dodgy rows, no full text index in play, the patIndex solution is the fastest, and it also selects the most rows.
(pre-ran km. to set the cache to a known state, ran the 3 processes, and finally ran km again - the last 2 runs of km gave times within 2 seconds)
patindex solution by Gerhard Weiss -- Runtime 0:38, returns 9144 rows
select dodgyColumn from myTable fcc
WHERE patindex('%[^ !-~]%' COLLATE Latin1_General_BIN,dodgyColumn ) >0
the substring-numbers solution by MT. -- Runtime 1:16, returned 8996 rows
select dodgyColumn from myTable fcc
INNER JOIN dbo.Numbers32k dn ON dn.number<(len(fcc.dodgyColumn ))
WHERE ASCII(SUBSTRING(fcc.dodgyColumn , dn.Number, 1))<32
OR ASCII(SUBSTRING(fcc.dodgyColumn , dn.Number, 1))>127
udf solution by Deon Robertson -- Runtime 3:47, returns 7316 rows
select dodgyColumn
from myTable
where dbo.udf_test_ContainsNonASCIIChars(dodgyColumn , 1) = 1
There is a user defined function available on the web 'Parse Alphanumeric'. Google UDF parse alphanumeric and you should find the code for it. This user defined function removes all characters that doesn't fit between 0-9, a-z, and A-Z.
Select * from Staging.APARMRE1 ar
where udf_parsealpha(ar.last_name) <> ar.last_name
That should bring back any records that have a last_name with invalid chars for you...though your bonus points question is a bit more of a challenge, but I think a case statement could handle it. This is a bit psuedo code, I'm not entirely sure if it'd work.
Select id, case when udf_parsealpha(ar.last_name) <> ar.last_name then 'last name'
when udf_parsealpha(ar.first_name) <> ar.first_name then 'first name'
when udf_parsealpha(ar.Address1) <> ar.last_name then 'Address1'
end,
case when udf_parsealpha(ar.last_name) <> ar.last_name then ar.last_name
when udf_parsealpha(ar.first_name) <> ar.first_name then ar.first_name
when udf_parsealpha(ar.Address1) <> ar.last_name then ar.Address1
end
from Staging.APARMRE1 ar
where udf_parsealpha(ar.last_name) <> ar.last_name or
udf_parsealpha(ar.first_name) <> ar.first_name or
udf_parsealpha(ar.Address1) <> ar.last_name
I wrote this in the forum post box...so I'm not quite sure if that'll function as is, but it should be close. I'm not quite sure how it will behave if a single record has two fields with invalid chars either.
As an alternative, you should be able to change the from clause away from a single table and into a subquery that looks something like:
select id,fieldname,value from (
Select id,'last_name' as 'fieldname', last_name as 'value'
from Staging.APARMRE1 ar
Union
Select id,'first_name' as 'fieldname', first_name as 'value'
from Staging.APARMRE1 ar
---(and repeat unions for each field)
)
where udf_parsealpha(value) <> value
Benefit here is for every column you'll only need to extend the union statement here, while you need to put that comparisson three times for every column in the case statement version of this script
To find which field has invalid characters:
SELECT * FROM Staging.APARMRE1 FOR XML AUTO, TYPE
You can test it with this query:
SELECT top 1 'char 31: '+char(31)+' (hex 0x1F)' field
from sysobjects
FOR XML AUTO, TYPE
The result will be:
Msg 6841, Level 16, State 1, Line 3 FOR XML could not serialize the
data for node 'field' because it contains a character (0x001F) which
is not allowed in XML. To retrieve this data using FOR XML, convert it
to binary, varbinary or image data type and use the BINARY BASE64
directive.
It is very useful when you write xml files and get error of invalid characters when validate it.
Here is a UDF I built to detectc columns with extended ascii charaters. It is quick and you can extended the character set you want to check. The second parameter allows you to switch between checking anything outside the standard character set or allowing an extended set:
create function [dbo].[udf_ContainsNonASCIIChars]
(
#string nvarchar(4000),
#checkExtendedCharset bit
)
returns bit
as
begin
declare #pos int = 0;
declare #char varchar(1);
declare #return bit = 0;
while #pos < len(#string)
begin
select #char = substring(#string, #pos, 1)
if ascii(#char) < 32 or ascii(#char) > 126
begin
if #checkExtendedCharset = 1
begin
if ascii(#char) not in (9,124,130,138,142,146,150,154,158,160,170,176,180,181,183,184,185,186,192,193,194,195,196,197,199,200,201,202,203,204,205,206,207,209,210,211,212,213,214,216,217,218,219,220,221,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,248,249,250,251,252,253,254,255)
begin
select #return = 1;
select #pos = (len(#string) + 1)
end
else
begin
select #pos = #pos + 1
end
end
else
begin
select #return = 1;
select #pos = (len(#string) + 1)
end
end
else
begin
select #pos = #pos + 1
end
end
return #return;
end
USAGE:
select Address1
from PropertyFile_English
where udf_ContainsNonASCIIChars(Address1, 1) = 1