I am trying to figure out how to use sp_msforeachtable to perform an action on all tables and variables that match variable/table names stored in another table
IE
I have a table that has 3 columns : table, variable, action
and I am trying to use sp_MSforeachtable to see which tables and variables match, and if match, perform that action on the table.
How do you call variable names in the sp_MSforeachtable statement? I know to use ? for the table name, but not sure how I would say if variable name=variable name then do X
Is there another way to do this without using this undocumented SP?
Ill try to explain better:
I am trying to clean personal info from a bunch of tables... I have a table that looks like this (not sure how to format a table, so imagine each entry is a seperate row, so the first row is Name, A, and set to '')
Variable
Name
Phone Number
Name
Table
A
A
B
Action
Set to ''
Set to '555-555-5555'
Set to ''
etc.
I then have a database full of tables....on table A, I would want my code to set all rows of variable 'Name'
to '' (blank)
, and Phone Number to '555-555-5555'
etc.and then move on to table B and do the same and so on
I would use a cursor and dynamic SQL:
--Set up for test:
CREATE TABLE #DataTable (column1 nvarchar(128) NOT NULL, column2 int NOT NULL); --Create global temp table so it can be accessed from dynamic SQL.
CREATE TABLE ##ActionTable ([table] nvarchar(128) NOT NULL, variable nvarchar(MAX) NOT NULL, [action] nvarchar(MAX) NOT NULL);
INSERT INTO ##ActionTable ([table], variable, [action])
VALUES
('#DataTable', '1', 'INSERT INTO #table (column1, column2) VALUES (''#variable_1'', #variable);'),
('#DataTable', '2', 'INSERT INTO #table (column1, column2) VALUES (''#variable_1'', #variable);'),
('#DataTable', '3', 'INSERT INTO #table (column1, column2) VALUES (''#variable_1'', #variable);'),
('#DataTable', '4', 'INSERT INTO #table (column1, column2) VALUES (''#variable_1'', #variable);');
--Code:
DECLARE #action nvarchar(MAX);
DECLARE #table nvarchar(128);
DECLARE #variable nvarchar(MAX);
DECLARE rowCurser CURSOR FOR SELECT [table], variable, [action] FROM ##ActionTable;
OPEN rowCurser;
FETCH rowCurser INTO #table, #variable, #action
WHILE ##FETCH_STATUS = 0
BEGIN
--Execute the code (pick one of the two. Option 2 is safer and can be cached (faster), but it does not work with my example because the parameters are left as variables).
-- Option 1:
SET #action = REPLACE(REPLACE(#action, '#table', #Table), '#variable', #variable);
EXECUTE(#action);
-- Option 2:
EXECUTE sp_executesql #stmt = N'INSERT INTO #DataTable (column1, column2) VALUES (CAST(#variable as nvarchar(128)) + N''_2'', #variable);', #params = N'#variable nvarchar(MAX)', #variable = #variable;
--Setup for next iteration
FETCH rowCurser INTO #table, #variable, #action
END
CLOSE rowCurser;
DEALLOCATE rowCurser;
--Check and cleanup from test
SELECT * FROM #DataTable;
DROP TABLE #DataTable;
DROP TABLE ##ActionTable;
Note: There are security concerns with what you are trying to do, since anyone who can add to your table will have the same access as the account which runs the script. You could reduce these concerns by defining the actions in another table which can only be edited by the administrator, then referencing the action in your existing table.
Note: It is best to have the data types of #action, #table, and #variable match their source columns. The variables an be any data type in your database (as long as it is not a local temp type). You will notice that there are two places in the code above where the types are defined, first where the variables are declared at the top, and second where the arguments for sp_executesql are defined in the string near the bottom.
Note: if #stmt and #params are assigned with a constant instead of a variable, make sure to prefix the constant with N so it will be read as a Unicode string.
Related
I'm using a search object query (found on the internet, wish I could give credit to the developer) to search database for the columns needed when I write queries. The output search object query allows me to enter the type of table to look in (partial name) as well as the column name (partial name) I'm trying to find. I've been attempting to modify the search object query so it returns the 1st value (Top 1) it finds. This would help me to easily see at a glance if the column has the particular type of data I'm looking for.
I've attempted to write it both as a stored procedure that I could pass two parameters (partial table and partial column name) and I've also tried using dynamic SQL (my first attempt at using it, so I'm a novice when it comes to use it). I had moderate success with the use of dynamic SQL, but can only get it to produce one result rather than be called multiple times for all the results in my search object output. The code I used is shown here:
-- This is the search object query found on internet
Use masterdb
Select a.name, b.name
From sysobjects a
Inner Join syscolumns b On a.id = b.id
Where b.name like '%Result%'
And a.name like '%Lab%'
Order By a.name, b.name
-- This is a separate query I used to test calling the data with dynamic SQL
DECLARE #value VARCHAR(100), #tablename VARCHAR(100)
SET #value = 'Result'
SET #tablename = 'LabSpecimen'
DECLARE #sqlText NVARCHAR(1000);
SET #sqlText = N'SELECT Top 1 ' + #value + ' FROM testndb.dbo.' + #tablename
EXEC (#sqlText)
If I use the search object query and search for tables that have lab and column names that have result, I might get output like this:
LabMain,ResultID
LabSpecimen,ResultCategory
LabSpecimen,ResultDate
LabSpecimen,Results
I would like to have the search object query pull data from the table in the first column and the column name in the 2nd column and return the first value it finds to give me a sample output for the given column name/table. Output would look like this:
LabMain,ResultID,E201812310001
LabSpecimen,ResultCategory,ExampleCategory
LabSpecimen,ResultDate,20181231
LabSpecimen,Results,34.20
Okay, I really didn't want to have to post an answer to this, but here goes.
So, the first, really-really-huge thing is: SQL Injection. SQL Injection is the #1 security vulnerability for something like a dozen years running, per OWASP. Basically, SQL Injection is where you use dynamic SQL that has any fragment of the sql command being populated by a user. So in the OP's case, this section here:
SET #value = 'Result'
SET #tablename = 'LabSpecimen'
DECLARE #sqlText NVARCHAR(1000);
SET #sqlText = N'SELECT Top 1 ' + #value + ' FROM testndb.dbo.' + #tablename
EXEC (#sqlText)
... if the end incarnation would be that #tableName and #value are populated by the user as part of their search? Then the user can do a 'search' that ends up injecting sql statements that the server runs directly; for a cheap example, imagine this for #value:
3' ; drop table #tableName --
... which would go ahead and drop every table that matches the #tablename you passed in.
Anyway, so, as we go through this problem, we're going to keep SQL Injection in mind at every step.
Problem #1: How to get the tables/columns that match.
You pretty much already nailed this. The only thing missing is to put it into a temp table so that you can loop through it (and limit it down to U-types, since otherwise you'll get stored procs and system tables.) I went ahead and had it also hit the Schema information - that way, if you have tables in different schemas, it'll still be able to get the results.
declare #tableNameFragment varchar(100) -- note: these top 4 lines will eventually
declare #columnNameFragment varchar(100) -- be changed to stored proc args
set #tableNameFragment = 'Performance' -- and populated by the user calling
set #columnNameFragment = 'status' -- the proc (instead of hard-coded.)
declare #entityMatches TABLE (TableName varchar(200), ColName varchar(128))
insert into #entityMatches
Select sch.TABLE_SCHEMA + '.' + sysobj.name as TableName, syscol.name as ColName
From sysobjects sysobj
Join syscolumns syscol On sysobj.id = syscol.id
Join INFORMATION_SCHEMA.TABLES sch on sch.TABLE_NAME = sysobj.name
where sysobj.xtype = 'U'
and (sysobj.name like '%' + isnull(#tableNameFragment,'') + '%')
and (syscol.name like '%' + isnull(#columnNameFragment,'') + '%')
Now, notice that while #tableNameFragment and #columnNameFragment are used, they're not used in a dynamic query. It doesn't matter if the user puts in something malicious into those values
Problem #2 - How to loop through your table
Basically, you're going to need a cursor. I hate cursors, but sometimes (like this one), they're necessary.
Problem #3 - How to actually do a dynamic query and get a result back
This is actually trickier than it looks. You can't do a raw EXEC() for a return value, nor can you simply have the cmd you're executing populating a variable - because EXEC (and SP_ExecuteSql operate in a different context, so they can't populate variables outside in your script.)
You need to use SP_ExecuteSQL, but specify a return variable getting populated by the interior sql command. For example:
declare #sqlCmd nvarchar(max)
declare #dynamicReturn varchar(max)
set #sqlCmd = 'select #retVal=1'
EXEC Sp_executesql #sqlCmd,
N'#retVal varchar(max) output',
#dynamicReturn output
select #dynamicReturn
Problem #4 - How to write your Dynamic command
Here's where things get dicey, since it's where we're using a dynamic SQL command. The important thing here is: you cannot use anything the user provided as an input. Which means, you can't use the variables #tableNameFragment or #columnNameFragment. You can use the values in the #entityMatches table, though. Why? Because the user didn't populate them. They got populated by the data in the sys tables - it doesn't matter if the user puts something nefarious in the input variables, that #entityMatches data simply holds the existing table/column names that match.
Also important: When you're working on code that could be a problem if a future dev down the line tweaks or copies/pastes - you should put comment warnings to illuminate the issue.
So, putting it all together? You'll have something that looks like this:
declare #tableNameFragment varchar(100) -- note: these top 4 lines will eventually
declare #columnNameFragment varchar(100) -- be changed to stored proc args
set #tableNameFragment = 'Performance' -- and populated by the user calling
set #columnNameFragment = 'status' -- the proc (instead of hard-coded.)
declare #entityMatches TABLE (TableName varchar(200), ColName varchar(128))
insert into #entityMatches
Select sch.TABLE_SCHEMA + '.' + sysobj.name as TableName, syscol.name as ColName
From sysobjects sysobj
Join syscolumns syscol On sysobj.id = syscol.id
Join INFORMATION_SCHEMA.TABLES sch on sch.TABLE_NAME = sysobj.name
where sysobj.xtype = 'U'
and (sysobj.name like '%' + isnull(#tableNameFragment,'') + '%')
and (syscol.name like '%' + isnull(#columnNameFragment,'') + '%')
declare #returnResults TABLE (TableName varchar(200), ColName varchar(128), FirstValue varchar(max))
declare Cur Cursor For select TableName,ColName from #entityMatches
declare #cursorTable varchar(200), #cursorColumn varchar(128)
open Cur
fetch Next from cur into #cursorTable,#cursorColumn
while ##FETCH_STATUS = 0
begin
-- Note: the variables #cursorTable, #cursorColumn are NOT user populated
-- but instead are populated from the Sys tables. Because of this,
-- this dynamic sql below is not SQL-Injection vulnerable (the entries
-- are not populated from user entry of any sort.)
-- Be very careful modifying the lines below to make sure you don't
-- introduce a vulnerability.
declare #sqlCmd nvarchar(max)
declare #dynamicReturn varchar(max)
set #sqlCmd = 'select top 1 #retVal=[' + #cursorColumn + '] from ' + #cursorTable
EXEC Sp_executesql #sqlCmd,
N'#retVal varchar(max) output',
#dynamicReturn output
insert into #returnResults values (#cursorTable, #cursorColumn, #dynamicReturn)
fetch Next from cur into #cursorTable,#cursorColumn
End
close cur
deallocate cur
select * from #returnResults
Create a stored procedure like below mention stored procedure.
Get the table and column name from sysobject & syscolumn and add it in hash table on the base of parameter of stored procedure. After that declare a cursor and in loop of cursor create a dynamic query of column and table name and get first row of current column from table of cursor loop. After that execute the query and update the result in the hash table. At the end of lookup select the Record from hash table. Check the below stored procedure. I hope that its helpful for you.
Create procedure Sp_GetSampleData
#TName varchar(200) = ''
as
Select
a.name TableName, b.name ColumnName,
CAST('' as varchar(max)) as SampleValue
into
#Tbl
from
sysobjects a
inner join
syscolumns b on a.id = b.id
where
(#TName='' or a.name = #TName)
order ny
a.name, b.name
declare #TableName varchar(200), #ColumnName varchar(200),
#sqlText nvarchar(max), #Val varchar(max)
declare Cur Cursor For
select TableName, ColumnName
from #Tbl
open Cur
fetch Next from cur into #TableName,#ColumnName
while ##FETCH_STATUS =0
begin
set #sqlText=''
set #Val=''
SET #sqlText = N'SELECT Top 1 #Val=[' + #ColumnName + '] FROM testndb.dbo.' + #TableName
EXEC Sp_executesql
#sqlText,
N'#Val varchar(max) output',
#Val output
print #sqlText
update #Tbl set SampleValue=#Val where TableName=#TableName and ColumnName =#ColumnName
fetch Next from cur into #TableName,#ColumnName
End
close cur
deallocate cur
select * from #Tbl
I have a stored procedure with dynamic sql that i have embedded as below:
delete from #temp_table
begin tran
set #sql = 'select * into #temp_table from sometable'
exec (#sql)
commit tran
begin
set #sql = 'alter table #temp_table add column1 float'
exec(#sql)
end
update #temp_table
set column1 = column1*100
select *
into Primary_Table
from #temp_table
However, I noticed that all the statements work but the alter does not. When run the procedure, I get an error message: "Invalid Column name column1"
What am I doing wrong here?
EDIT: Realized I didn't mention that the first insert is a dynamic sql as well. Updated it.
Alternate approach tried but throws same error:
delete from #temp_table
begin tran
set #sql = 'select * into #temp_table from sometable'
exec (#sql)
commit tran
alter table #temp_table add column1 float
update #temp_table set column1 = column1*100
Local temporary tables exhibit something like dynamic scope. When you create a local temporary table inside a call to exec it goes out of scope and existence on the return from exec.
EXEC (N'create table #x (c int)')
GO
SELECT * FROM #x
Msg 208, Level 16, State 0, Line 4
Invalid object name '#x'.
The select is parsed after the dynamic SQL to create #x is ran. But #x is not there because dropped on exit from exec.
Update
Depending on the situation there are different ways to work around the issue.
Put everything into the same string:
DECLARE #Sql NVARCHAR(MAX) = N'SELECT 1 AS source INTO #table_name;
ALTER TABLE #table_name ADD TARGET float;
UPDATE #table_name SET Target = 100 * source;';
EXEC (#Sql);
Create the table ahead of the dynamic sql that populates it.
CREATE TABLE #table_name (source INT);
EXEC (N'insert into #table_name (source) select 1;');
ALTER TABLE #table_name ADD target FLOAT;
UPDATE #table_name SET target = 100 * source;
In this option, the alter table statement can be removed by adding the additional column to the create table statement.' Note also that the alter table and update statements could be in separate invocations of dynamic SQL, if that was beneficial to your context.
1) It should be ALTER TABLE #temp... Not ALTER #temp.
2) Even if #1 weren't an issue, you're adding column1, as a NULLable column with no default value and, in the next statement setting it's value to itself * 100...
NULL * 100 = NULL
3) Why are you using dynamic sql to alter the #temp table? It can just as easily be done with a regular ALTER TABLE script... or, better yet, can be included in the original table definition.
This is because the #temp_table reference in the outer batch is a different temp table than the one created in dynamic SQL. Consider:
use tempdb
drop table if exists sometable
drop table if exists #temp_table
go
create table sometable(id int, a int)
create table #temp_table(id int, b int)
exec( 'select * into #temp_table from sometable; select * from #temp_table;' )
select * from #temp_table
Outputs
id a
----------- -----------
(0 rows affected)
id b
----------- -----------
(0 rows affected)
A temp table created in a nested batch is scoped to the nested batch and automatically dropped after. A "nested batch" is either a dynamic SQL query or a stored procedure. This behavior is explained here CREATE TABLE, but it only mentions stored procedures. Dynamic SQL behaves the same.
If you create the temp table in a top level batch, you can access it in dynamic SQL, you just can't create a new temp table in dynamic SQL and see it in the outer batch or in subsequent same-level dynamic SQL. So try to use INSERT INTO instead of SELECT INTO.
I am trying to code a stored procedure in SQL that does the following
Takes 2 inputs (BatchType and "Column Name").
Searches database and gives the batchdate and the data in the column = "Column name"
Code is as give below
ALTER PROCEDURE [dbo].[chartmilldata]
-- Add the parameters for the stored procedure here
(#BatchType nvarchar (50),
#Data nvarchar(50))
AS
BEGIN
-- Insert statements for procedure here
SELECT BatchDate,#Data FROM --Database-- WHERE BatchType = #BatchType
END
I am trying to select column from the database based on operator input. But I am not getting the output. It would be great if someone can give me a direction.
You may want to build out your SELECT statement as a string then execute it using sp_executesql.
See this page for more info:
https://msdn.microsoft.com/en-us/library/ms188001.aspx
This will allow you to set your query to substitute in your column name via your variable and then execute the statement. Be sure to sanitize your inputs though!
You'd need to use dynamic SQL, HOWEVER I would not recommend this solution, I don't think there is anything I can add as to why I wouldn't recommend it that isn't explained better in Erland Sommarskog in The Curse and Blessings of Dynamic SQL.
Nonetheless, if you had to do it in a stored procedure you could use something like:
ALTER PROCEDURE [dbo].[chartmilldata]
-- Add the parameters for the stored procedure here
(#BatchType nvarchar (50),
#Data nvarchar(50))
AS
BEGIN
-- DECLARE AND SET SQL TO EXECUTE
DECLARE #SQL NVARCHAR(MAX) = N'SELECT BatchDate = NULL, ' +
QUOTENAME(#Data) + N' = NULL;';
-- CHECK COLUMN IS VALID IN THE TABLE
IF EXISTS
( SELECT 1
FROM sys.columns
WHERE name = #Data
AND object_id = OBJECT_ID('dbo.YourTable', 'U')
)
BEGIN
SET #SQL = 'SELECT BatchDate, ' + QUOTENAME(#Data) +
' FROM dbo.YourTable WHERE BatchType = #BatchType;';
END
EXECUTE sp_executesql #SQL, N'#BatchType NVARCHAR(50)', #BatchType;
END
It would probably be advisable to change your input parameter #Data to be NVARCHAR(128) (or the alias SYSNAME) though, since this is the maximum for column names.
In Sql server, i write a procedure and i use one tem table and a cursor and dynamically add one column to that temporary table but it is giving erro :
(10 row(s) affected)
Msg 213, Level 16, State 1, Procedure USP_F_Roll_AllIndia_Report, Line 27
Column name or number of supplied values does not match table definition.
This is my proc :
alter procedure USP_F_Roll_AllIndia_Report
(#segcode int,#rollplanyear int)
as
begin
declare #cfcode varchar(10)
declare #cfname varchar(30)
declare #SQl nvarchar(max)
create table #TEP (productcode varchar(10) collate database_default,proddesc varchar(100))
declare db_cursor cursor for
select distinct canfm.CFCode, SUBSTRING (CANFM.CFName,4,5)as CFName from Tbl_F_CandF_M CANFM left outer join Tbl_F_Org_CandF_T CT on CANFM.CFCode = ct.CFCode where CANFM .status =1 and ct.Status =1 order by canfm.cfcode
open db_cursor
fetch next from db_cursor into #cfcode, #cfname
while ##FETCH_STATUS =0
begin
set #SQL ='alter table #TEP add '+#cfname+' float'
exec sp_executesql #Sql
--exec ( #Sql)
insert into #TEP
select pd.productcode,PM.productdesc,convert(varchar,sum(isnull(AmendedQty,isnull(Quantity,0))))as quantity from Tbl_F_Roll_PlanDetails_T pd left outer join Tbl_F_ProductMaster_M PM on
pd.ProductCode =pm.ProductCode left outer join Tbl_F_CandF_M CANDF on pd.CandFLocation =CANDF.CFCode where pd. RollPlanYear =#rollplanyear and pd.CandFLocation =#cfcode and pd.ProductCode in (
select ProductCode from Tbl_F_Segment_Product_t where SegCode =#segcode ) group by pd.ProductCode,pm.ProductDesc
fetch next from db_cursor into #cfcode, #cfname
end
close db_cursor
deallocate db_cursor
select * from #TEP
end
this will not work . if you are adding column then your select query in insert statement must be a dynamic one since it will keep adding column .
you can create a dynamic query for insert and in select as well.
you will also need to specify columns names like
insert into #TEP (col1,clo2,col3..)
there might be better ways for your requiremnet if you specify them .
cursor and adding column is not good logic.
This will not work as you're trying to. SQL Server is trying to compile the entire batch as early as possible - it compiles the insert once your create table has executed, and at that time, there are 2 columns in the table, but 3 in the insert.
But, stop and think about it further - even if it worked the first time through the loop, what happens the next time through your loop? At that point, there are 4 columns in your table, but still only 3 in the insert. I can't remember if that will fail completely, or just insert into the first additional column you've added to the table, but either way, I'm almost certain it's not what you want.
It looks like you're trying to do some form of pivot with an unknown number of columns - there are plenty of questions and answers on SO about doing that already. You have to go down the route of dynamic SQL (even more so than what you've attempted), and it's never pretty. I'd almost always recommend returning a normal result set (e.g. fixed number of columns) to another system (code, report generator, etc) which is better suited to do that kind of mucking about with formatting.
I think the problem is that you are adding a new column on each loop, but the insert statement has a static number of columns.
i don't know why this isn't working for you but
I'm Running SQL SERVER 2008 R2 AND the following is working fine with me
CREATE TABLE #temp (ID int)
DECLARE #Sql as varchar(250)
DECLARE #colName varchar(50)
SET #colName = 'name'
SET #Sql = 'ALTER TABLE #temp ADD [' + #colName + '] VARCHAR(50)'
EXEC (#Sql)
SELECT * FROM #temp
DROP TABLE #temp
SQL Fiddle
Declare #colname varchar(max)='col1,col2,col3',#sqlq varchar(max)=''
Declare #tblname varchar(max)='tbl1'
set #sqlq='Select '+#colname+' from table tbl_DefaultPermission '
EXECUTE (#sqlq)
I have a stored procedure that uses sp_executesql to generate a result set, the number of columns in the result can vary but will be in the form of Col1 Col2 Col3 etc.
I need to get the result into a temp table or table variable so I can work with it. The problem is I need to define the columns of the temp table, which I cant do dynamically using sp_executesql as the scope of the temp table is lost after the command is executed.
I have toyed with the idea of using Global Temp tables, as the scope allows it to be created dynamically, however, there is a very good chance the Global Temps would get updated by the concurrent executions of this process.
Any ideas?
I have found a solution that works for me with the help of #SQLMenace in this post T-SQL Dynamic SQL and Temp Tables
In short, I need to create a #temp table in normal SQL first, then I can alter the structure using further dynamic SQL statements. In this example #colcount is set to 6. This will be determined by another stored proc when I implement this.
IF object_id('tempdb..#myTemp') IS NOT NULL
DROP TABLE #myTemp
CREATE TABLE #myTemp (id int IDENTITY(1,1) )
DECLARE #cmd nvarchar(max)
DECLARE #colcount int
SET #colcount = 6
DECLARE #counter int
SET #counter = 0
WHILE #counter < #colcount
BEGIN
SET #counter = #counter + 1
SET #cmd = 'ALTER TABLE #myTemp ADD col' + CAST(#counter AS varchar(4)) + ' NVARCHAR(MAX)'
EXEC(#cmd)
END
INSERT INTO #myTemp
EXEC myProc #param1, #param2, #param3
SELECT * FROM #myTemp
IS there any reason you can't do something like:
SELECT *
INTO #MyTempTable
FROM MyResultSet
SELECT INTO doesn't require an explicit field list.
You can use global temp tables whose names are 'uniquified' by the SPID of the creating process. This can allow you to avoid stomping on other global temp tables created by other connections.
Just make sure to clean them up when you're done... :)