I have a table called raw_data that contains a column with a large string of data fields formatted in fixed length sub-strings. I also have a table table_1 that specifies the column name and the data range in the string for each value. I need to create a SQL INSERT statement to move data from raw_data into a table called table_2 with all the columns. table_1 has about 600 rows, so I am wondering if I can loop through each record to create the SQL statement that inserts the data into table_2.
Table_1
Name Start Length
AAA 1 2
BBB 3 3
CCC 6 1
I haven't learned how to use cursors; the below query could be incorrect. There will be 3 tables involved in this task. table_1 to look up the name, start, length values. table_2 will be the table I need to insert the data into. The third table raw_data has the column with the sub-strings of each needed value.
DECLARE #SQL VARCHAR(200)
DECLARE #NAME VARCHAR(200)
DECLARE #START VARCHAR(200)
DECLARE #LENGTH VARCHAR(200)
SET #NAME = ''
DECLARE Col_Cursor CURSOR FOR
SELECT Name, Start, Length FROM ODS_SIEMENS_LAYOUT WHERE RecordType = '1'
OPEN Col_Cursor
FETCH NEXT FROM Col_Cursor INTO #NAME, #START, #LENGTH
WHILE ##FETCH_STATUS = 0
BEGIN
SET #SQL = #NAME + '=' + 'SUBSTRING(RAW_DATA,' + #START + ',' + #LENGTH + ')'
FETCH NEXT FROM Col_Cursor INTO #NAME, #START, #LENGTH
END
CLOSE Col_Cursor
DEALLOCATE Col_Cursor
I need to generate something like the below query:
INSERT INTO TABLE_2
'AAA' = SUBSTRING(RAW_DATA,1,2)
'BBB' = SUBSTRING(RAW_DATA,3,3)
'CCC' = SUBSTRING(RAW_DATA,5,2)
........
Can I loop through each column to form the SQL Statement instead of manually coding 600 columns?
At the risk of sounding like Clippy... it looks like you're trying to import a flat file. Is your RAW_DATA coming from a flat file somewhere? If so you might look into using bulk insert:
Use a Format File to Bulk Import Data
If you are just asking how can you build your sql statement using the data from your column definition table... then the code you have is very close. You want something like this:
DECLARE #COLUMNS varchar(max)
DECLARE #SUBCOLUMNS varchar(max)
DECLARE #NAME VARCHAR(200)
DECLARE #START VARCHAR(200)
DECLARE #LENGTH VARCHAR(200)
SET #NAME = ''
DECLARE Col_Cursor CURSOR FOR
SELECT Name, Start, Length FROM ODS_SIEMENS_LAYOUT WHERE RecordType = '1'
OPEN Col_Cursor
FETCH NEXT FROM Col_Cursor INTO #NAME, #START, #LENGTH
set #SUBCOLUMNS = ''
set #COLUMNS = ''
WHILE ##FETCH_STATUS = 0
BEGIN
SET #COLUMNS = #COLUMNS + #NAME + ','
SET #SUBCOLUMNS = #SUBCOLUMNS + 'SUBSTRING(RAW_DATA,' + #START + ',' + #LENGTH + '),'
FETCH NEXT FROM Col_Cursor INTO #NAME, #START, #LENGTH
END
CLOSE Col_Cursor
DEALLOCATE Col_Cursor
set #COLUMNS = LEFT(#COLUMNS, len(#COLUMNS)-1) --get rid of last comma
set #SUBCOLUMNS = LEFT(#SUBCOLUMNS, len(#SUBCOLUMNS)-1) --get rid of last comma
print 'INSERT INTO TABLE_2 ' + '(' + #COLUMNS + ') SELECT ' + #SUBCOLUMNS + ' FROM RawDataTable'
You can take the text that prints and insert that SQL statement into your procedure that does the actual inserts.
Ahh I think I am beginning to unravel what you are trying to do. There is no need for a cursor or dynamic sql here at all. You just need to use a select statement as the values for your insert. Something like this maybe??
INSERT INTO TABLE_2(AAA, BBB, CCC)
SELECT SUBSTRING(RAW_DATA,1,2)
, SUBSTRING(RAW_DATA,3,3)
, SUBSTRING(RAW_DATA,5,2)
FROM ODS_SIEMENS_LAYOUT
WHERE RecordType = '1'
Related
I'm trying to build a stored procedure that will query multiple database depending on the databases required.
For example:
SP_Users takes a list of #DATABASES as parameters.
For each database it needs to run the same query and union the results together.
I believe a CTE could be my best bet so I have something like this at the moment.
SET #DATABASES = 'DB_1, DB_2' -- Two databases in a string listed
-- I have a split string function that will extract each database
SET #CURRENT_DB = 'DB_1'
WITH UsersCTE (Name, Email)
AS (SELECT Name, Email
FROM [#CURRENT_DB].[dbo].Users),
SELECT #DATABASE as DB, Name, Email
FROM UsersCTE
What I don't want to do is hard code the databases in the query. The steps I image are:
Split the parameter #DATABASES to extract and set the #CURRENT_DB Variable
Iterate through the query with a Recursive CTE until all the #DATABASES have been processed
Union all results together and return the data.
Not sure if this is the right approach to tackling this problem.
Using #databases:
As mentioned in the comments to your question, variables cant be used to dynamically select a database. Dynamic sql is indicated. You can start by building your template sql statement:
declare #sql nvarchar(max) =
'union all ' +
'select ''#db'' as db, name, email ' +
'from [#db].dbo.users ';
Since you have sql server 2016, you can split using the string_split function, with your #databases variable as input. This will result in a table with 'value' as the column name, which holds the database names.
Use the replace function to replace #db in the template with value. This will result in one sql statement for each database you passed into #databases. Then, concatenate the statements back together. Unfortunately, in version 2016, there's no built in function to do that. So we have to use the famous for xml trick to join the statements, then we use .value to convert it to a string, and finally we use stuff to get rid of the leading union all statement.
Take the results of the concatenated output, and overwrite the #sql variable. It is ready to go at this point, so execute it.
I do all that is described in this code:
declare #databases nvarchar(max) = 'db_1,db_2';
set #sql = stuff(
(
select replace(#sql, '#db', value)
from string_split(#databases, ',')
for xml path(''), type
).value('.[1]', 'nvarchar(max)')
, 1, 9, '');
exec(#sql);
Untested, of course, but if you print instead of execute, it seems to give the proper sql statement for your needs.
Using msForEachDB:
Now, if you didn't want to have to know which databases had 'users', such as if you're in an environment where you have a different database for every client, you can use sp_msForEachDb and check the structure first to make sure it has a 'users' table with 'name' and 'email' columns. If so, execute the appropriate statement. If not, execute a dummy statement. I won't describe this one, I'll just give the code:
declare #aggregator table (
db sysname,
name int,
email nvarchar(255)
);
insert #aggregator
exec sp_msforeachdb '
declare #sql nvarchar(max) = ''select db = '''''''', name = '''''''', email = '''''''' where 1 = 2'';
select #sql = ''select db = ''''?'''', name, email from ['' + table_catalog + ''].dbo.users''
from [?].information_schema.columns
where table_schema = ''dbo''
and table_name = ''users''
and column_name in (''name'', ''email'')
group by table_catalog
having count(*) = 2
exec (#sql);
';
select *
from #aggregator
I took the valid advice from others here and went with this which works great for what I need:
I decided to use a loop to build the query up. Hope this helps someone else looking to do something similar.
CREATE PROCEDURE [dbo].[SP_Users](
#DATABASES VARCHAR(MAX) = NULL,
#PARAM1 VARCHAR(250),
#PARAM2 VARCHAR(250)
)
BEGIN
SET NOCOUNT ON;
--Local variables
DECLARE
#COUNTER INT = 0,
#SQL NVARCHAR(MAX) = '',
#CURRENTDB VARCHAR(50) = NULL,
#MAX INT = 0,
#ERRORMSG VARCHAR(MAX)
--Check we have databases entered
IF #DATABASES IS NULL
BEGIN
RAISERROR('ERROR: No Databases Provided,
Please Provide a list of databases to execute procedure. See stored procedure:
[SP_Users]', 16, 1)
RETURN
END
-- SET Number of iterations based on number of returned databases
SET #MAX = (SELECT COUNT(*) FROM
(SELECT ROW_NUMBER() OVER (ORDER BY i.value) AS RowNumber, i.value
FROM dbo.udf_SplitVariable(#DATABASES, ',') AS i)X)
-- Build SQL Statement
WHILE #COUNTER < #MAX
BEGIN
--Set the current database
SET #CURRENTDB = (SELECT X.Value FROM
(SELECT ROW_NUMBER() OVER (ORDER BY i.value) AS RowNumber, i.value
FROM dbo.udf_SplitVariable(#DATABASES, ',') AS i
ORDER BY RowNumber OFFSET #COUNTER
ROWS FETCH NEXT 1 ROWS ONLY) X);
SET #SQL = #SQL + N'
(
SELECT Name, Email
FROM [' + #CURRENTDB + '].[dbo].Users
WHERE
(Name = #PARAM1 OR #PARAM1 IS NULL)
(Email = #PARAM2 OR #PARAM2 IS NULL)
) '
+ N' UNION ALL '
END
PRINT #CURRENTDB
PRINT #SQL
SET #COUNTER = #COUNTER + 1
END
-- remove last N' UNION ALL '
IF LEN(#SQL) > 11
SET #SQL = LEFT(#SQL, LEN(#SQL) - 11)
EXEC sp_executesql #SQL, N'#CURRENTDB VARCHAR(50),
#PARAM1 VARCHAR(250),
#PARAM2 VARCHAR(250)',
#CURRENTDB,
#PARAM1 ,
#PARAM2
END
Split Variable Function
CREATE FUNCTION [dbo].[udf_SplitVariable]
(
#List varchar(8000),
#SplitOn varchar(5) = ','
)
RETURNS #RtnValue TABLE
(
Id INT IDENTITY(1,1),
Value VARCHAR(8000)
)
AS
BEGIN
--Account for ticks
SET #List = (REPLACE(#List, '''', ''))
--Account for 'emptynull'
IF LTRIM(RTRIM(#List)) = 'emptynull'
BEGIN
SET #List = ''
END
--Loop through all of the items in the string and add records for each item
WHILE (CHARINDEX(#SplitOn,#List)>0)
BEGIN
INSERT INTO #RtnValue (value)
SELECT Value = LTRIM(RTRIM(SUBSTRING(#List, 1, CHARINDEX(#SplitOn, #List)-1)))
SET #List = SUBSTRING(#List, CHARINDEX(#SplitOn,#List) + LEN(#SplitOn), LEN(#List))
END
INSERT INTO #RtnValue (Value)
SELECT Value = LTRIM(RTRIM(#List))
RETURN
END
I''m trying to use a varchar variable in my select, but its registering as a string and outputting just the string itself instead of treating it like an actual column.
Not very many solutions online, as it seems like this isn't a problem very many run into.
Declare #counter INT = 0
Declare #totalcol INT
Declare #col VARCHAR(50)
select #totalcol = count(*)
FROM [Loyalty_DW].information_schema.columns
WHERE table_name = 'Transactions'
while (#counter < #totalcol)
begin
select #col = COLUMN_NAME
from [Loyalty_DW].information_schema.columns
where table_name = 'Transactions'
order by (select null)
offset #counter rows
fetch next 1 rows only
select distinct(#col)
from [Loyalty_DW].dbo.Transactions
set #counter += 1
end
The output is just a string with no actual data returned. The same as if I were to say select 'asdf' from tablename where ... it would just output 'asdf'.
You can use CURSOR statement rather than WHILE statement, as Gordon says you need to use dynamic SQL:
DECLARE #columName AS NVARCHAR(50);
DECLARE #sqlText AS NVARCHAR(1000);
DECLARE cursor_name CURSOR FOR
SELECT COLUMN_NAME
FROM [Loyalty_DW].INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'Transactions'
ORDER BY(SELECT NULL)
OPEN cursor_name
FETCH NEXT FROM cursor_name INTO #columName
WHILE ##FETCH_STATUS = 0
BEGIN
SET #sqlText = N'SELECT DISTINCT ' + #columName + ' FROM [Loyalty_DW].dbo.Transactions'
EXEC (#sqlText)
FETCH NEXT FROM cursor_name INTO #columName
END
CLOSE cursor_name
DEALLOCATE cursor_name
I'm looking to set up an array of data quality scripts on a new set of data that my group is getting. One of the issues I'd like to keep track of is maintaining a list of fields that contain nulls. Some fields that's ok, others it is not, although we don't want to block the bulk insert because we've not allowed certain fields to null.
Example:
SELECT [columns]
FROM [TABLE] T
WHERE [columns] CONTAINS NULLS
You could use something like this in a function to loop thru all of the columns and see if there is a count of null values in the table. The code is kind of brute force but it would work.
declare #tablename nvarchar(255) = 'schema.tablename'
declare #results table (ColumnName nvarchar(255))
/* Declare the Variables to be used in the cusor*/
declare #column_name nvarchar(255)
/* Declare the cursor and the value set that will be used*/
declare tbl_Crawler cursor
for SELECT name FROM sys.columns
WHERE object_id = OBJECT_ID(#tablename)
/* Make the Cursor Available*/
open tbl_Crawler
/*Load the first row into the variables (must match sequence in cursor select)*/
fetch next from tbl_Crawler
into #column_name
/*Creates While loop that will run until the curor is empty*/
while ##fetch_status = 0
begin
/*statement to be run every loop of the cursor*/
DECLARE #sqlCommand nvarchar(1000)
declare #counts int
SET #sqlCommand = 'SELECT #cnt=COUNT(*) FROM '+#tablename+' WHERE '+ #column_name + ' is null'
EXECUTE sp_executesql #sqlCommand, N'#cnt int OUTPUT', #cnt=#counts OUTPUT
if (isnull(#counts,0) > 0)
insert into #results
select #column_name
/*Loads the next row of records into the variables (must match sequence in cursor select)*/
fetch next from tbl_Crawler
into #column_name
end
/*Release the cursor so that it is not retained in memory*/
close tbl_Crawler
deallocate tbl_Crawler
/* if in function return #results*/
select * from #results
This is slow and ugly, but short of buying a tool from RedGate (et al.), you might be able to modify it with an ISNULL(#column,'PICK ME') to search for your nulls...
DECLARE #Query NVARCHAR(MAX), #Column NVARCHAR(100), #Table NVARCHAR(100)
DECLARE #Search NVARCHAR(100)
SET #Search = 'PICK ME'
IF OBJECT_ID('tempdb.dbo.#Results2','U') IS NOT NULL
DROP TABLE tempdb.dbo.#Results2
CREATE TABLE #Results2(Table_Name VARCHAR(100), Column_Name VARCHAR(100))
DECLARE Col CURSOR FOR
SELECT Table_Name, Column_Name
FROM INFORMATION_SCHEMA.COLUMNS
WHERE COLLATION_NAME IS NOT NULL
ORDER BY TABLE_NAME, ORDINAL_POSITION
OPEN Col
FETCH NEXT FROM Col INTO #Table, #Column
WHILE ##FETCH_STATUS = 0
BEGIN
SET #Query = 'IF EXISTS (SELECT * FROM '+QUOTENAME(#Table)+' WHERE '+QUOTENAME(#Column)+'='''+#Search+''')
SELECT '''+#Table+''','''+#Column+''''
INSERT INTO #Results2
EXEC sp_executesql #Query
FETCH NEXT FROM Col INTO #Table, #Column
END
CLOSE Col
DEALLOCATE Col
SELECT * FROM #Results2
We suffered some kind of invasion in our SQL Server.
I'm trying to find in every database, in every table, every column the word abortion and cheat.
I can do this with this query, but in a single database.
-- Store results in a local temp table so that. I'm using a
-- local temp table so that I can access it in SP_EXECUTESQL.
create table #tmp
(
db varchar(max),
tbl nvarchar(max),
col nvarchar(max),
val nvarchar(max),
);
declare #db nvarchar(max);
declare #tbl nvarchar(max);
declare #col nvarchar(max);
declare #q nvarchar(max);
declare #search nvarchar(max) = 'abortion';
-- Create a cursor on all columns in the database
declare c cursor for
SELECT
DB_NAME(DB_ID()) as DBName, tbls.TABLE_NAME, cols.COLUMN_NAME
FROM INFORMATION_SCHEMA.TABLES AS tbls
JOIN INFORMATION_SCHEMA.COLUMNS AS cols ON tbls.TABLE_NAME = cols.TABLE_NAME
-- For each table and column pair, see if the search value exists.
open c
fetch next from c into #db, #tbl, #col
while ##FETCH_STATUS = 0
begin
-- Look for the search key in current table column and if found add it to the results.
SET #q = 'INSERT INTO #tmp SELECT ''' +#db+''',''' + #tbl + ''', ''' + #col + ''', ' + #col + ' FROM ' + #tbl + ' WHERE ' + #col + ' LIKE ''%' + #search + '%'''
EXEC SP_EXECUTESQL #q
fetch next from c into #db, #tbl, #col
end
close c
deallocate c
-- Get results
select distinct db,tbl,col from #tmp
-- Remove local temp table.
drop table #tmp
How can I find these strings? The result set should be:
DATABASE | TABLE | COLUMN
I don't need the result ( text field ), and I need to select distinct for tables and columns, because it will be a lot of abortion in the same table/column.
While the use of the undocumented sp_msforeachdb is generally not encouraged, my instinct would be to send your existing code to this procedure like this:
exec sp_MSforeachdb 'USE [?];
-- Store results in a local temp table so that. I'm using a
-- local temp table so that I can access it in SP_EXECUTESQL.
create table #tmp (
db varchar(max) ,
tbl nvarchar(max),
col nvarchar(max),
val nvarchar(max),
);
declare #db nvarchar(max);
declare #tbl nvarchar(max);
declare #col nvarchar(max);
declare #q nvarchar(max);
--------------------------------------------------------------------------------------------
declare #search nvarchar(max) = ''abortion'';
--------------------------------------------------------------------------------------------
-- Create a cursor on all columns in the database
declare c cursor for
SELECT DB_NAME(DB_ID()) as DBName,tbls.TABLE_NAME, cols.COLUMN_NAME FROM INFORMATION_SCHEMA.TABLES AS tbls
JOIN INFORMATION_SCHEMA.COLUMNS AS cols
ON tbls.TABLE_NAME = cols.TABLE_NAME
-- For each table and column pair, see if the search value exists.
open c
fetch next from c into #db, #tbl, #col
while ##FETCH_STATUS = 0
begin
-- Look for the search key in current table column and if found add it to the results.
SET #q = ''INSERT INTO #tmp SELECT '''''' +#db+'''''','''''' + #tbl + '''''', '''''' + #col + '''''', '' + #col + '' FROM '' + #tbl + '' WHERE '' + #col + '' LIKE ''''%'' + #search + ''%''''''
EXEC SP_EXECUTESQL #q
fetch next from c into #db, #tbl, #col
end
close c
deallocate c;'
The only added code here is the first line, for the rest of the code just make sure to replace ' with ''. The ? in USE [?] is a special character meaning the currently active database in the loop sp_MSforeachdb executes.
I need to create about 300 columns for a table and I don't want to to it manually.
How can I do this?
I want to have columns like
Bigint1..to..Bigint60
Text1 ..to..Text60
and so on.
IF (NOT EXISTS (SELECT *
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA = 'dbo'
AND TABLE_NAME = 'mytbl'))
begin
create table OBJ_AttributeValues(
ObjectID numeric(18,0) not null
);
end
else
begin
DECLARE #A INT
set #A = 1;
WHILE(#A <=60)
BEGIN
alter table OBJ_AttributeValues
add ...............................
set #A = #A+1;
END
end
What should I write instead of "..."?
You will need to use dynamic SQL for that, something like
DECLARE #SSQL VARCHAR(1000)
DECLARE #A INT
set #A = 1;
WHILE(#A <=60)
BEGIN
SET #SSQL = 'alter table OBJ_AttributeValues add Bigint' + CAST(#A as varchar) + ' bigint'
EXEC (#SSQL)
set #A = #A+1;
END
This isn't really a good idea, you should take the time to write the sql or just copy-paste the columns from Excel or something like that. You also shouldn't be using the TEXT data type, is deprecated and filled with restriction (use VARCHAR(MAX) instead if you need). That said, here is a way using dynamic SQL:
DECLARE #BigintCols NVARCHAR(MAX), #TextCols NVARCHAR(MAX)
DECLARE #Query NVARCHAR(MAX)
SET #BigintCols = ''
SET #TextCols = ''
SELECT #BigintCols = #BigintCols + N'Bigint' + CAST(number AS NVARCHAR(2)) + N' BIGINT,',
#TextCols = #TextCols + N'Text' + CAST(number AS NVARCHAR(2)) + N' TEXT,'
FROM master..spt_values
WHERE type = 'P'
AND number BETWEEN 1 AND 60
ORDER BY number
SET #Query = '
CREATE TABLE OBJ_AttributeValues(ObjectID numeric(18,0) not null,'+#BigintCols+
LEFT(#TextCols,LEN(#TextCols)-1)+')'
EXECUTE sp_executesql #Query
Oh, you should probably read about dynamic sql first.