Related
I have a table named Product with following data for instance:
p_id
p_name
p_cat
1
shirt
null
2
null
null
3
cap
null
Suppose I don't know numbre of rows and columns in the table as well as I don't know which columns are compltely null (no non-null value in all of its rows). How to write a query to retrieve just the columns that have atleast one non-null value in its rows. My approach is as follows but not getting a corret output:
select
column_name
into #TempColumns
from information_schema.columns
where
table_name = 'Product'
and table_schema = 'DDB'
declare #CurrentColumn nvarchar(max) = '', #IsNull bit, #NonNullCols nvarchar(max) = ''
declare Cur cursor for
select column_name from #TempColumns
open Cur
while 1=1
begin
fetch next from Cur into #CurrentColumn
select #IsNull = case when count(#CurrentColumn) > 0 then 0 else 1 end
from Product
if #IsNull = 1
begin
set #NonNullCols = #NonNullCols + ',' + #CurrentColumn
end
if ##fetch_status <> 0 break
end
close Cur
deallocate Cur
select #NonNullCols as NullColumns
drop table #TempColumns
If there is any other approach or correction in my above (T-SQL) query. Thanks in advance.
First I just create a temporary table to store all the column names avaialbe in the Product table. Then I looped in this temporary table and feteched each row and checked it on the product table whether the column is comptely null or not using the count() function. The condition sets the bit variable 1 if the column is completely null and then that particular column name is stored in anothe variable which is then retrieved as null columns.
Here is a conceptual example for you.
It is using SQL Server XML and XQuery powers without dynamic SQL and cursors/loops.
The algorithm is very simple.
When we are converting each row into XML, columns that hold NULL value are missing from the XML.
SQL
USE tempdb;
GO
DROP TABLE IF EXISTS #tmpTable;
CREATE TABLE #tmpTable (
client_id int,
client_name varchar(500),
client_surname varchar(500),
city varchar(500),
state varchar(500));
INSERT #tmpTable VALUES
(1,'Miriam',NULL,'Las Vegas',NULL),
(2,'Astrid',NULL,'Chicago',NULL),
(3,'David',NULL,'Phoenix',NULL),
(4,'Hiroki',NULL,'Orlando',NULL);
SELECT DISTINCT x.value('local-name(.)', 'SYSNAME') AS NotNULLColumns
FROM #tmpTable AS t
CROSS APPLY (SELECT t.* FOR XML PATH(''), TYPE, ROOT('root')) AS t1(c)
CROSS APPLY c.nodes('/root/*') AS t2(x);
SQL #2
To handle edge cases.
SELECT DISTINCT x.value('local-name(.)', 'SYSNAME') AS NotNULLColumns
FROM #tmpTable AS t
CROSS APPLY (SELECT t.* FOR XML RAW, ELEMENTS, BINARY BASE64, TYPE, ROOT('root')) AS t1(c)
CROSS APPLY c.nodes('/root/row/*') AS t2(x);
Output
NotNULLColumns
city
client_id
client_name
I hope you are all well.
I would like your help on a data transformation task that I have.
I would like to convert the first row of a table to a column name
I am working on SQL Server Azure and I get daily data from another service.
This service loads a table that is of the same form.
and I would like to transform the data in the same manner
Do You have any idea how to do it ?
The way to solve this is by using a little dynamic SQL magic:
First, create and populate sample table (Please save us thus step in your future questions):
DECLARE #T As Table
(
Row_num int,
Line nvarchar(4000)
);
INSERT INTO #T (Row_Num, Line) VALUES
(1, 'Col1;Col2;Col3'),
(2, 'Val1;Val2;Val3'),
(3, 'Value1;Value2;Value1'),
(4, 'Val A; val B;Val A'),
(5, 'Value A; Value B;Value C');
Then, build a union all query that selects the values from every row but the first, replacing the semicolon (;) separator with a comma (,) surrounded by apostrophes ('). Add an apostrophe before and after the string (which means we are treating all the data as strings):
DECLARE #Sql nvarchar(max) = '';
SELECT #Sql += 'UNION ALL SELECT '''+ REPLACE(Line, ';', ''',''') + ''' '
FROM #T
WHERE Row_Num > 1;
Next, use stuff to replace the first UNION ALL with a common table expression declaration, specifying the column names in the declaration itself. Note that here we don't need the apostrophes anymore, just to replace the semicolon with a comma:
SELECT #Sql = STUFF(#Sql, 1, 10, 'WITH CTE('+ REPLACE(Line, ';', ',') +') AS (') + ') SELECT * FROM CTE'
FROM #T
WHERE Row_Num = 1;
Finally, execute the sql:
EXEC(#Sql)
Results:
Col1 Col2 Col3
Val1 Val2 Val3
Value1 Value2 Value1
Val A val B Val A
Value A Value B Value C
You can see a live demo on rextester.
Another possible approach is to transform your text data into valid JSON arrays and then use OPENJSON() with an explicit schema and dynamic statement.
Working example:
Input:
CREATE TABLE #Data (
RowNum int,
Line nvarchar(max)
)
INSERT INTO #Data
(RowNum, Line)
VALUES
(1, 'ColumnA;ColumnB;ColumnC'),
(2, 'ValueA1;ValueB1;ValueC1'),
(3, 'ValueA2;ValueB2;ValueC2'),
(4, 'ValueA3;ValueB3;ValueC3'),
(5, 'ValueA4;ValueB4;ValueC4'),
(6, 'ValueA5;ValueB5;ValueC5')
T-SQL:
-- Explicit schema generation
DECLARE #schema nvarchar(max)
SELECT #schema = STUFF((
SELECT CONCAT(N',', j.[value], N' nvarchar(max) ''$[', j.[key], N']''')
FROM #Data d
CROSS APPLY OPENJSON(CONCAT(N'["', REPLACE(d.Line, ';', '","'), N'"]')) j
WHERE d.RowNum = 1
FOR XML PATH('')
), 1, 1, N'')
-- Dymanic statement
DECLARE #stm nvarchar(max)
SET #stm = CONCAT(
N'SELECT j.* FROM #Data d ',
N'CROSS APPLY OPENJSON(CONCAT(N''[["'', REPLACE(d.Line, '';'', ''","''), N''"]]'')) ',
N'WITH (',
#schema,
N') j WHERE d.RowNum > 1'
)
-- Execution
EXEC sp_executesql #stm
Output:
-----------------------
ColumnA ColumnB ColumnC
-----------------------
ValueA1 ValueB1 ValueC1
ValueA2 ValueB2 ValueC2
ValueA3 ValueB3 ValueC3
ValueA4 ValueB4 ValueC4
ValueA5 ValueB5 ValueC5
Explanations:
The main part is to transform each row's data into valid JSON arrays. The count of the columns can be different.
Data from the first row will be used for explicit schema generation and values ColumnA;ColumnB;ColumnC are transformed into ["ColumnA","ColumnB","ColumnC"]. Values from subsequent rows ValueA1;ValueB1;ValueC1 are transformed into [["ValueA1","ValueB1","ValueC1"]].
Next simple examples demonstrate how OPENJSON() returns data with default and explicit schema:
With default schema:
DECLARE #json nvarchar(max)
SET #json = '["ValueA1", "ValueB1", "ValueC1"]'
SELECT *
FROM OPENJSON(#json)
Output for default schema:
----------------
key value type
----------------
0 ValueA1 1
1 ValueB1 1
2 ValueC1 1
With explicit schema:
SET #json = '[["ValueA1", "ValueB1", "ValueC1"]]'
SELECT *
FROM OPENJSON(#json)
WITH (
ColumnA nvarchar(max) '$[0]',
ColumnB nvarchar(max) '$[1]',
ColumnC nvarchar(max) '$[2]'
)
Output for explicit schema:
-----------------------
ColumnA ColumnB ColumnC
-----------------------
ValueA1 ValueB1 ValueC1
I need help in fine-tuning this code that I write. I am new to SQL Server and I believe there are better ways to do this or perhaps some of the following codes can be simplified or fine-tuned for performances or saving memory resources.
Basically, I have this XML data :
<table_result id="001" subj_cd="cdaaa" grade="b" name="Phua Chu Kang"/>
and I want to create a table which looks like this from that XML data
Please note on the following points :
SplitThis is not a built in function (check the code below).
The data can have space, but delimited by ". Do note as well that the XML data can have varying number of fields-data pairs for that particular given table - referred as #dummy in the following codes. i.e. example XML data above have 4 fields (id, subj_cd, grade, name) and the next XML data could have 5 fields (i.e. id, name, occupation, phone_no, address). In the following code, #table_result is created to match the example XML data for easier demonstration. In other words, the table structures are known..so I can ignore the field names from the XML data and focus on extracting the data itself.
The code ran well on SQL Server 2012 (you can copy and paste run code directly) and I am able to get as above. I just need to fine tune this, if possible. I have include line like this : - - test blabla. You can uncomment that and try. I could use enhancements such as in term of avoiding the number of temp tables used or any ways to replace the use of row_number() in the code.
/* remove all temp tables */
declare #sql varchar(5000)
SELECT #sql = isnull(#sql+';', '') + 'drop table ' + SUBSTRING(t.name, 1, CHARINDEX('___', t.name)-1)
FROM tempdb..sysobjects AS t
WHERE t.name LIKE '#%[_][_][_]%'
AND t.id =OBJECT_ID('tempdb..' + SUBSTRING(t.name, 1, CHARINDEX('___', t.name)-1));
exec (#sql)
/* end */
/* function */
drop function splitthis
go
create function splitthis(#separator char(1), #list varchar(max))
returns #returntable table(item nvarchar(max))
as
begin
declare #index int
declare #newtext varchar(max)
if #list = null
return
set #index = charindex(#separator, #list)
while not(#index = 0)
begin
set #newtext = rtrim(ltrim(left(#list, #index - 1)))
set #list = right(#list, len(#list) - #index)
insert into #returntable(item) values(#newtext)
set #index = charindex(#separator, #list)
end
insert into #returntable(item) values(rtrim(ltrim(#list)))
update #returntable set item='' where item is null
return
end
go
/* end of function */
/* create dummy tables */
create table #table_result
(id nvarchar(max), subj_cd nvarchar(max), grade nvarchar(max), name nvarchar(max))
create table #dummy (name nvarchar(max), data nvarchar(max))
insert into #dummy
values ('a', '<table_result id="001" subj_cd="cdaaa" grade="b" name="phua chu kang"/>');
--test : select * from #dummy
/* remove the fist non-data opening tag */
declare #record nvarchar(max)
select #record = data from #dummy where name = 'a'
select *, null as temp into #tempb from splitthis(' ',#record)
select *, row_number() over (order by temp) count into #tempc from #tempb
select item into #tempd from #tempc where #tempc.count>1
-- test : select * from #tempd
/* get the actual field & data into a single column table */
declare #temp varchar(max)
set #temp=''select #temp=#temp+' ' + item from #tempd
select *, null as temp into #tempe from splitthis('"',#temp)
select *, row_number() over (order by temp) count into #tempf from #tempe
select item, count into #tempg from #tempf
--test : select * from #tempg
/* prepare the data table */
select
case when #tempg.count % 2 = 0
then item
else null
end as data
into #temph
from #tempg
select data, null as temp into #tempi from #temph
select data, row_number() over (order by temp) count into #data from #tempi
where data is not null
--test : select * from #data
/* prepare the field table. */
select name, null as temp into #tempj
from tempdb.sys.columns where object_id=object_id('tempdb..#table_result');
select *, row_number() over (order by temp) count into #field from #tempj
--test : select * from #field
/* get the final table */
select a.name as field, b.data from #field a
left join #data b on a.count=b.count
This is - using XML methods - much easier!
Try this:
DECLARE #xml XML='<table_result id="001" subj_cd="cdaaa" grade="b" name="Phua Chu Kang"/>';
SELECT One.Attr.value('fn:local-name(.)','varchar(max)') AS field
,One.Attr.value('.','varchar(max)') AS data
FROM #xml.nodes('table_result/#*') AS One(Attr)
The result
field data
id 001
subj_cd cdaaa
grade b
name Phua Chu Kang
Now I try to imitate your table structure (I'd recommend to store the data as XML from the beginning! In this case you could omit the first CROSS APPLY with the CAST ... AS XML):
DECLARE #tbl TABLE(name VARCHAR(10),data VARCHAR(MAX));
INSERT INTO #tbl VALUES
('a','<table_result id="001" subj_cd="cdaaa" grade="b" name="Phua Chu Kang"/>')
,('b','<Another test="test data" test2="test2 data"/>')
,('c','<OneMore x="x data" y="y data" z="z data"/>');
SELECT tbl.name
,One.Attr.value('fn:local-name(..)','varchar(max)') AS element
,One.Attr.value('fn:local-name(.)','varchar(max)') AS field
,One.Attr.value('.','varchar(max)') AS data
FROM #tbl AS tbl
CROSS APPLY(SELECT CAST(tbl.data AS XML)) AS MyData(AsXml)
CROSS APPLY MyData.AsXml.nodes('*/#*') AS One(Attr)
The result
name element field data
a table_result id 001
a table_result subj_cd cdaaa
a table_result grade b
a table_result name Phua Chu Kang
b Another test test data
b Another test2 test2 data
c OneMore x x data
c OneMore y y data
c OneMore z z data
Now, I'm not at all very good with T-SQL XML, but can't you just do it like this:
create table #dummy (name nvarchar(max), data xml);
insert into #dummy
values ('a', '<table_result id="001" subj_cd="cdaaa" grade="b" name="phua chu kang"/>');
select 'id' "field",
elem.value('#id', 'nvarchar(50)') "data"
from #dummy
cross apply data.nodes('/table_result') tbl(elem)
union all
select 'subj_cd' "field",
elem.value('#subj_cd', 'nvarchar(50)') "data"
from #dummy
cross apply data.nodes('/table_result') tbl(elem)
union all
select 'grade' "field",
elem.value('#grade', 'nvarchar(50)') "data"
from #dummy
cross apply data.nodes('/table_result') tbl(elem)
union all
select 'name' "field",
elem.value('#name', 'nvarchar(50)') "data"
from #dummy
cross apply data.nodes('/table_result') tbl(elem);
Notice that I changed the data type for #dummy.data to be xml. That's required to be able to use the XML functions.
Scenario: I have a SSRS report which has a bunch of filters. One of those filters is for "Area", I get the areas like so:
SELECT DISTINCT Area FROM tblArea ORDER BY Area
I have a default value of "All" for the parameter that gets its available values from that, so the output would be like:
All
Area1
Area2
China
US
etc
In the report they could select "All" which when passed to the main stored procedure the parameter would look like:
All, Area1, Area2, China, US, etc
Now here is the issue: In the table which I query the Area column might have data like:
Area
--------
Area2,US
And the query is
SELECT * FROM tbl WHERE TPC IN (#Area) -- All,Area1,Area2,China,US,etc
And this would not find the record I am looking for.
In short, how do I compare each value in a comma separated column against a comma separated parameter?
Thanks in advance,
Your query is looking for a single value, that has a comma in it. The in doesn't parse the string.
You can do what you want using like:
where #Area = 'All' or
','+TPC+',' like '%,'+#Area+',%';
The use of comma is to prevent collisions, when one area is contained in another (say, "US" and "USSR").
In SQL, you must create a Table-valued function that will convert the comma-delimited string #Area into Table. You can then query using this table.
ALTER FUNCTION [dbo].[List_to_tbl] (#list nvarchar(MAX))
RETURNS #tbl TABLE (stringsTable nvarchar(10) NOT NULL) AS
BEGIN
DECLARE #pos int,
#nextpos int,
#valuelen int
SELECT #pos = 0, #nextpos = 1
WHILE #nextpos > 0
BEGIN
SELECT #nextpos = charindex(',', #list, #pos + 1)
SELECT #valuelen = CASE WHEN #nextpos > 0
THEN #nextpos
ELSE len(#list) + 1
END - #pos - 1
INSERT #tbl (stringsTable)
VALUES (substring(#list, #pos + 1, #valuelen))
SELECT #pos = #nextpos
END
RETURN
END
Now you can call this from you Stored procedure
Select * from TBL where TPC in (Select stringTable from List_to_tbl(#Area))
I think below example will solve your purpose
CREATE TABLE #TBL
([Id] int, [Areas] varchar(11))
;
INSERT INTO #TBL
([Id], [Areas])
VALUES
(1, 'abc,def,ghi'),
(2, 'abc,def'),
(3, 'abc')
;
DECLARE #Area VARCHAR(100) -- parameter
SET #Area = 'All,abc,def,ghi'
;with cte
as
(
select Areaxml.s.value('.','varchar(100)') as Area
from
(
select convert(xml,'<m>'+replace(#Area,',','</m><m>')+'</m>') as CArea
)t
cross apply
CArea.nodes('/m') as Areaxml(s)
)
select * FROM #TBL
where exists (select Area from cte where [Areas] LIKE '%'+ Area +'%')
DROP TABLe #TBL
I'm trying to create a procedure in SQL Server 2008 that inserts data from a temp table into an already existing table. I think I've pretty much figured it out, I'm just having an issue with a loop. I need the row count from the temp table to determine when the loop should finish.
I've tried using ##ROWCOUNT in two different ways; using it by itself in the WHILE statement, and creating a variable to try and hold the value when the first loop has finished (see code below).
Neither of these methods have worked, and I'm now at a loss as to what to do. Is it possible to use ##ROWCOUNT in this situation, or is there another method that would work better?
CREATE PROCEDURE InsertData(#KeywordList varchar(max))
AS
BEGIN
--create temp table to hold words and weights
CREATE TABLE #tempKeywords(ID int NOT NULL, keyword varchar(10) NOT NULL);
DECLARE #K varchar(10), #Num int, #ID int
SET #KeywordList= LTRIM(RTRIM(#KeywordList))+ ','
SET #Num = CHARINDEX(',', #KeywordList, 1)
SET #ID = 0
--Parse varchar and split IDs by comma into temp table
IF REPLACE(#KeywordList, ',', '') <> ''
BEGIN
WHILE #Num > 0
BEGIN
SET #K= LTRIM(RTRIM(LEFT(#KeywordList, #Num - 1)))
SET #ID = #ID + 1
IF #K <> ''
BEGIN
INSERT INTO #tempKeywords VALUES (#ID, #K)
END
SET #KeywordList = RIGHT(#KeywordList, LEN(#KeywordList) - #Num)
SET #Num = CHARINDEX(',', #KeywordList, 1)
--rowcount of temp table
SET #rowcount = ##ROWCOUNT
END
END
--declaring variables for loop
DECLARE #count INT
DECLARE #t_name varchar(30)
DECLARE #key varchar(30)
DECLARE #key_weight DECIMAL(18,2)
--setting count to start from first keyword
SET #count = 2
--setting the topic name as the first row in temp table
SET #t_name = (Select keyword from #tempKeywords where ID = 1)
--loop to insert data from temp table into Keyword table
WHILE(#count < #rowcount)
BEGIN
SET #key = (SELECT keyword FROM #tempKeywords where ID = #count)
SET #key_weight = (SELECT keyword FROM #tempKeywords where ID = #count+2)
INSERT INTO Keyword(Topic_Name,Keyword,K_Weight)
VALUES(#t_name,#key,#key_weight)
SET #count= #count +2
END
--End stored procedure
END
To solve the second part of your problem:
INSERT INTO Keyword(Topic_Name,Keyword,K_Weight)
SELECT tk1.keyword, tk2.keyword, tk3.keyword
FROM
#tempKeywords tk1
cross join
#tempKeywords tk2
inner join
#tempKeywords tk3
on
tk2.ID = tk3.ID - 1
WHERE
tk1.ID = 1 AND
tk2.ID % 2 = 0
(This code should replace everything in your current script from the --declaring variables for loop comment onwards)
You could change:
WHILE(#count < #rowcount)
to
WHILE(#count < (select count(*) from #tempKeywords))
But like marc_s commented, you should be able to do this without a while loop.
I'd look at reworking your query to see if you can do this in a set based way rather than row by row.
I'm not sure I follow exactly what you are trying to achieve, but I'd be tempted to look at the ROW_NUMBER() function to set the ID of your temp table. Used with a recursive CTE such as shown in this answer you could get an id for each of your non empty trimmed words. An example is something like;
DECLARE #KeywordList varchar(max) = 'TEST,WORD, ,,,LIST, SOME , WITH, SPACES'
CREATE TABLE #tempKeywords(ID int NOT NULL, keyword varchar(10) NOT NULL)
;WITH kws (ord, DataItem, Data) AS(
SELECT CAST(1 AS INT), LEFT(#KeywordList, CHARINDEX(',',#KeywordList+',')-1) ,
STUFF(#KeywordList, 1, CHARINDEX(',',#KeywordList+','), '')
union all
select ord + 1, LEFT(Data, CHARINDEX(',',Data+',')-1),
STUFF(Data, 1, CHARINDEX(',',Data+','), '')
from kws
where Data > ''
), trimKws(ord1, trimkw) AS (
SELECT ord, RTRIM(LTRIM(DataItem))
FROM kws
)
INSERT INTO #tempKeywords (ID, keyword)
SELECT ROW_NUMBER() OVER (ORDER BY ord1) as OrderedWithoutSpaces, trimkw
FROM trimKws WHERE trimkw <> ''
SELECT * FROM #tempKeywords
I don't fully understand what you are trying to acheive with the second part of your query , but but you could just build on this to get the remainder of it working. It certainly looks as though you could do what you are after without while statements at least.