We are using SQL Server 2014 Enterprise with many databases. I have to execute query and get reports / data from every database with EXACT SAME Schema and database starts with Cab
When a new company is added in our ERP project a new database is created with exact schema starting with Cab and incremented number is assigned to it like:
Cab1
Cab2
Cab3
Cab5
Cab10
I can get the database names as:
SELECT name
FROM master.sys.databases
where [name] like 'Cab%' order by [name]
I have to create a Stored Procedure to get data from tables of every database.
How to do that using a Stored Procedure as the databases are created dynamically starting with Cab?
You can use EXEC(#Statement) or EXEC SP_EXECUTESQL if you have to pass parameters.
CREATE OR ALTER PROCEDURE dbo.GetDataFromAllDatabases
AS
BEGIN
DECLARE #T TABLE (id INT NOT NULL IDENTITY(1, 1), dbName VARCHAR(256) NOT NULL)
INSERT INTO #T
SELECT NAME FROM MASTER.SYS.DATABASES WHERE [NAME] LIKE 'Cab%' ORDER BY [NAME]
CREATE TABLE #AllData (......)
DECLARE #Id INT, #DbName VARCHAR(128)
SELECT #Id = MIN(Id) FROM #T
WHILE #Id IS NOT NULL
BEGIN
SELECT #DbName = dbName FROM #T WHERE Id = #Id
DECLARE #Statement NVARCHAR(MAX)
SET #Statement = CONCAT(N'INSERT INTO #AllData (...) SELECT .... FROM ', #DbName, '.dbo.[TableName]')
EXEC(#Statement);
--YOU CAN USE BELOW LINE TOO IF YOU NEED TO PASS VARIABLE
--EXEC SP_EXECUTESQL #Statement, '#Value INT', #Value = 128
SET #Id = (SELECT MIN(Id) FROM #T WHERE Id > #Id)
END
END
A quick and easy dynamic SQL solution would be something like this:
DECLARE #Sql nvarchar(max);
SET #Sql = STUFF((
SELECT ' UNION ALL SELECT [ColumnsList], '''+ [name] + ''' As SourceDb FROM '+ QUOTENAME([name]) + '.[SchemaName].[TableName]' + char(10)
FROM master.sys.databases
WHERE [name] LIKE 'Cab%'
FOR XML PATH('')
), 1, 10, '');
--When dealing with dynamic SQL, print is your best friend...
PRINT #Sql
-- Once the #Sql is printed and you can see it looks OK, you can run it.
--EXEC(#Sql)
Notes:
Use quotename to protect against "funny" chars in identifiers names.
Replace [ColumnsList] with the actual list of columns you need.
There's no need for loops of any kind, just a simple stuff + for xml to mimic string_agg (which was only introduced in 2017).
I've thrown in the source database name as a "bonus", if you don't want it that's fine.
The Order by clause in the query that generates the dynamic SQL is meaningless for the final query, so I've removed it.
Related
I have a system that takes in Revit models and loads all the data in the model to a 2016 SQL Server. Unfortunately, the way the system works it created a new database for each model that is loaded. All the databases start with an identical schema because there is a template database that the system uses to build any new ones.
I need to build a view that can query data from all databases on the server but can automatically add new databases as they are created. The table names and associated columns will be identical across all databases, including data types.
Is there a way to pull a list of current database names using:
SELECT [name] FROM sys.databases
and then use the results to UNION the results from a basic SELECT query like this:
SELECT
[col1]
,[col2]
,[col3]
FROM [database].[dbo].[table]
Somehow replace the [database] part with the results of the sys.databases query?
The goal would be for the results to look as if I did this:
SELECT
[col1]
,[col2]
,[col3]
FROM [database1].[dbo].[table]
UNION
SELECT
[col1]
,[col2]
,[col3]
FROM [database2].[dbo].[table]
but dynamically for all databases on the server and without future management from me.
Thanks in advance for the assistance!
***Added Info: A couple suggestions using STRING_AGG have been made, but that function is not available in 2016.
Try this. It will automatically detect and include new databases with the specified table name. If a database is dropped it will automatically exclude it.
I updated the TSQL. STRING_AGG concatenates the string with each database. Without it it only returns the last database. STRING_AGG is more secure than += which also concatenates. I changed the code so it generates and executes the query. In SQL 2019 the query is all in one line using +=. I don't have SQL 2016. It may format it better in SQL 2016. You can uncomment --SELECT #SQL3 to see what the query looks like. Please mark as answer if this is what you need.
DECLARE #TblName TABLE
(
TblName VARCHAR(100)
)
Declare #SQL VARCHAR(MAX),
#SQL3 VARCHAR(MAX),
#DBName VARCHAR(50),
#Count Int,
#LoopCount Int
Declare #SQL2 VARCHAR(MAX) = ''
Select Identity(int,1,1) ID, name AS DBName into #Temp from sys.databases
Select #Count = ##RowCount
Set #LoopCount = 1
While #LoopCount <= #Count
Begin
SET #DBName = (SELECT DBName FROM #Temp Where ID = #LoopCount)
SET #SQL =
' USE ' + #DBName +
' SELECT TABLE_CATALOG FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = ''table'''
INSERT INTO #TblName (TblName)
EXEC (#SQL)
Set #LoopCount=#LoopCount + 1
End
SELECT #SQL2 +=
' SELECT ' + char(10) +
' [col1] ' + char(10) +
' ,[col2] ' + char(10) +
' ,[col3] ' + char(10) +
' FROM [' + TblName + '].[dbo].[table] ' + char(10) +
' UNION '
FROM #TblName
DROP TABLE #Temp
SET #SQL3 = (SELECT SUBSTRING(#SQL2, 1, LEN(#SQL2) - 5))
--SELECT #SQL3
EXEC (#SQL3)
I'm using a search object query (found on the internet, wish I could give credit to the developer) to search database for the columns needed when I write queries. The output search object query allows me to enter the type of table to look in (partial name) as well as the column name (partial name) I'm trying to find. I've been attempting to modify the search object query so it returns the 1st value (Top 1) it finds. This would help me to easily see at a glance if the column has the particular type of data I'm looking for.
I've attempted to write it both as a stored procedure that I could pass two parameters (partial table and partial column name) and I've also tried using dynamic SQL (my first attempt at using it, so I'm a novice when it comes to use it). I had moderate success with the use of dynamic SQL, but can only get it to produce one result rather than be called multiple times for all the results in my search object output. The code I used is shown here:
-- This is the search object query found on internet
Use masterdb
Select a.name, b.name
From sysobjects a
Inner Join syscolumns b On a.id = b.id
Where b.name like '%Result%'
And a.name like '%Lab%'
Order By a.name, b.name
-- This is a separate query I used to test calling the data with dynamic SQL
DECLARE #value VARCHAR(100), #tablename VARCHAR(100)
SET #value = 'Result'
SET #tablename = 'LabSpecimen'
DECLARE #sqlText NVARCHAR(1000);
SET #sqlText = N'SELECT Top 1 ' + #value + ' FROM testndb.dbo.' + #tablename
EXEC (#sqlText)
If I use the search object query and search for tables that have lab and column names that have result, I might get output like this:
LabMain,ResultID
LabSpecimen,ResultCategory
LabSpecimen,ResultDate
LabSpecimen,Results
I would like to have the search object query pull data from the table in the first column and the column name in the 2nd column and return the first value it finds to give me a sample output for the given column name/table. Output would look like this:
LabMain,ResultID,E201812310001
LabSpecimen,ResultCategory,ExampleCategory
LabSpecimen,ResultDate,20181231
LabSpecimen,Results,34.20
Okay, I really didn't want to have to post an answer to this, but here goes.
So, the first, really-really-huge thing is: SQL Injection. SQL Injection is the #1 security vulnerability for something like a dozen years running, per OWASP. Basically, SQL Injection is where you use dynamic SQL that has any fragment of the sql command being populated by a user. So in the OP's case, this section here:
SET #value = 'Result'
SET #tablename = 'LabSpecimen'
DECLARE #sqlText NVARCHAR(1000);
SET #sqlText = N'SELECT Top 1 ' + #value + ' FROM testndb.dbo.' + #tablename
EXEC (#sqlText)
... if the end incarnation would be that #tableName and #value are populated by the user as part of their search? Then the user can do a 'search' that ends up injecting sql statements that the server runs directly; for a cheap example, imagine this for #value:
3' ; drop table #tableName --
... which would go ahead and drop every table that matches the #tablename you passed in.
Anyway, so, as we go through this problem, we're going to keep SQL Injection in mind at every step.
Problem #1: How to get the tables/columns that match.
You pretty much already nailed this. The only thing missing is to put it into a temp table so that you can loop through it (and limit it down to U-types, since otherwise you'll get stored procs and system tables.) I went ahead and had it also hit the Schema information - that way, if you have tables in different schemas, it'll still be able to get the results.
declare #tableNameFragment varchar(100) -- note: these top 4 lines will eventually
declare #columnNameFragment varchar(100) -- be changed to stored proc args
set #tableNameFragment = 'Performance' -- and populated by the user calling
set #columnNameFragment = 'status' -- the proc (instead of hard-coded.)
declare #entityMatches TABLE (TableName varchar(200), ColName varchar(128))
insert into #entityMatches
Select sch.TABLE_SCHEMA + '.' + sysobj.name as TableName, syscol.name as ColName
From sysobjects sysobj
Join syscolumns syscol On sysobj.id = syscol.id
Join INFORMATION_SCHEMA.TABLES sch on sch.TABLE_NAME = sysobj.name
where sysobj.xtype = 'U'
and (sysobj.name like '%' + isnull(#tableNameFragment,'') + '%')
and (syscol.name like '%' + isnull(#columnNameFragment,'') + '%')
Now, notice that while #tableNameFragment and #columnNameFragment are used, they're not used in a dynamic query. It doesn't matter if the user puts in something malicious into those values
Problem #2 - How to loop through your table
Basically, you're going to need a cursor. I hate cursors, but sometimes (like this one), they're necessary.
Problem #3 - How to actually do a dynamic query and get a result back
This is actually trickier than it looks. You can't do a raw EXEC() for a return value, nor can you simply have the cmd you're executing populating a variable - because EXEC (and SP_ExecuteSql operate in a different context, so they can't populate variables outside in your script.)
You need to use SP_ExecuteSQL, but specify a return variable getting populated by the interior sql command. For example:
declare #sqlCmd nvarchar(max)
declare #dynamicReturn varchar(max)
set #sqlCmd = 'select #retVal=1'
EXEC Sp_executesql #sqlCmd,
N'#retVal varchar(max) output',
#dynamicReturn output
select #dynamicReturn
Problem #4 - How to write your Dynamic command
Here's where things get dicey, since it's where we're using a dynamic SQL command. The important thing here is: you cannot use anything the user provided as an input. Which means, you can't use the variables #tableNameFragment or #columnNameFragment. You can use the values in the #entityMatches table, though. Why? Because the user didn't populate them. They got populated by the data in the sys tables - it doesn't matter if the user puts something nefarious in the input variables, that #entityMatches data simply holds the existing table/column names that match.
Also important: When you're working on code that could be a problem if a future dev down the line tweaks or copies/pastes - you should put comment warnings to illuminate the issue.
So, putting it all together? You'll have something that looks like this:
declare #tableNameFragment varchar(100) -- note: these top 4 lines will eventually
declare #columnNameFragment varchar(100) -- be changed to stored proc args
set #tableNameFragment = 'Performance' -- and populated by the user calling
set #columnNameFragment = 'status' -- the proc (instead of hard-coded.)
declare #entityMatches TABLE (TableName varchar(200), ColName varchar(128))
insert into #entityMatches
Select sch.TABLE_SCHEMA + '.' + sysobj.name as TableName, syscol.name as ColName
From sysobjects sysobj
Join syscolumns syscol On sysobj.id = syscol.id
Join INFORMATION_SCHEMA.TABLES sch on sch.TABLE_NAME = sysobj.name
where sysobj.xtype = 'U'
and (sysobj.name like '%' + isnull(#tableNameFragment,'') + '%')
and (syscol.name like '%' + isnull(#columnNameFragment,'') + '%')
declare #returnResults TABLE (TableName varchar(200), ColName varchar(128), FirstValue varchar(max))
declare Cur Cursor For select TableName,ColName from #entityMatches
declare #cursorTable varchar(200), #cursorColumn varchar(128)
open Cur
fetch Next from cur into #cursorTable,#cursorColumn
while ##FETCH_STATUS = 0
begin
-- Note: the variables #cursorTable, #cursorColumn are NOT user populated
-- but instead are populated from the Sys tables. Because of this,
-- this dynamic sql below is not SQL-Injection vulnerable (the entries
-- are not populated from user entry of any sort.)
-- Be very careful modifying the lines below to make sure you don't
-- introduce a vulnerability.
declare #sqlCmd nvarchar(max)
declare #dynamicReturn varchar(max)
set #sqlCmd = 'select top 1 #retVal=[' + #cursorColumn + '] from ' + #cursorTable
EXEC Sp_executesql #sqlCmd,
N'#retVal varchar(max) output',
#dynamicReturn output
insert into #returnResults values (#cursorTable, #cursorColumn, #dynamicReturn)
fetch Next from cur into #cursorTable,#cursorColumn
End
close cur
deallocate cur
select * from #returnResults
Create a stored procedure like below mention stored procedure.
Get the table and column name from sysobject & syscolumn and add it in hash table on the base of parameter of stored procedure. After that declare a cursor and in loop of cursor create a dynamic query of column and table name and get first row of current column from table of cursor loop. After that execute the query and update the result in the hash table. At the end of lookup select the Record from hash table. Check the below stored procedure. I hope that its helpful for you.
Create procedure Sp_GetSampleData
#TName varchar(200) = ''
as
Select
a.name TableName, b.name ColumnName,
CAST('' as varchar(max)) as SampleValue
into
#Tbl
from
sysobjects a
inner join
syscolumns b on a.id = b.id
where
(#TName='' or a.name = #TName)
order ny
a.name, b.name
declare #TableName varchar(200), #ColumnName varchar(200),
#sqlText nvarchar(max), #Val varchar(max)
declare Cur Cursor For
select TableName, ColumnName
from #Tbl
open Cur
fetch Next from cur into #TableName,#ColumnName
while ##FETCH_STATUS =0
begin
set #sqlText=''
set #Val=''
SET #sqlText = N'SELECT Top 1 #Val=[' + #ColumnName + '] FROM testndb.dbo.' + #TableName
EXEC Sp_executesql
#sqlText,
N'#Val varchar(max) output',
#Val output
print #sqlText
update #Tbl set SampleValue=#Val where TableName=#TableName and ColumnName =#ColumnName
fetch Next from cur into #TableName,#ColumnName
End
close cur
deallocate cur
select * from #Tbl
I want to pass a param value into a function in sql server.
CREATE FUNCTION dbo.MyFunc(#param varchar(30))
RETURNS TABLE
AS
RETURN(
SELECT #param FROM MyTable
)
GO;
Can somebody help me?
I need do it using a function.
Thanks
I have considered the solution procedure and exec, however need to be a function not a procedure, because then I'd like to do joins between the results of the function and other tables.
Something like:
SELECT * FROM MyFunction ('column1, column2, column3, columnN, etc') MyTable INNER JOIN ON (....);
A procedure does not work for me Joins other tables.
In addition,,,, not always be three values to select, so I do not set the table structure.
Sometimes it will be:
id column1 column2
1 value1 value1.2
2 value2 value2.2
Sometimes it will be:
id column1 column2 column3
1 value1 value1.2 value1.3
2 value2 value2.2 value2.3
You can't parameterize column names (or table names, for that matter), So the only way to do such a thing is using dynamic Sql.
This also means that you can't use a user defined table function to do this, since that requires a hard-coded well-defined table to be returned.
You can, however, use a stored procedure to do such a thing:
CREATE PROCEDURE SELECT_FROM_TABLE
(
#ColumnName sysname,
#TableName sysname
)
AS
DECLARE #Sql varchar(max)
SELECT #Sql = 'SELECT '+ #ColumnName +' FROM '+ #TableName
EXEC(#Sql)
GO
Note: This kind of procedure should NEVER be exposed outside of your database. the reason for that is that it's a HUGH risk for Sql injection attacks.
Should you decide to create a procedure like this, you MUST verify the inputs inside the stored procedure itself. Something like this should get you started:
CREATE PROCEDURE SELECT_FROM_TABLE
(
#ColumnName sysname,
#TableName sysname
)
AS
IF EXISTS(
SELECT 1
FROM INFORMATION_SCHEMA.COLUMNS
WHERE COLUMN_NAME = #ColumnName
AND TABLE_NAME = #TableName
)
BEGIN
DECLARE #Sql varchar(max) = 'SELECT '+ #ColumnName +' FROM '+ #TableName
EXEC(#Sql)
END
GO
This way, you can be sure that the column name and table name passed to the procedure actually exists in your database.
I have a SQL Server table called "tblProducts".
Sometimes I backup this table by making a copy of it with this simple query:
SELECT *
INTO [test01].[dbo].[tblProducts_20141206]
FROM [test01].[dbo].[tblProducts]
Every time when making a backup, the date is included in the table name.
I would like to create a SQL Job that runs this kind of query once every week.
Is it possible to maybe in a stored procedure or declaring a variable to achieve this that allows the backed-up table name to be named like [tblProducts_todaysDate]?
Thanks.
If you are using a SP, you can do something like:
CREATE PROC sp_createATable
#name VARCHAR(20) AS
CREATE TABLE #name
...
do your insert
Or, if you want to, w/o SP:
DECLARE #name varchar(20)
SET #name = 'tblName' + SELECT CONVERT(VARCHAR(8), GETDATE(), 112) AS [YYYYMMDD]
CREATE TABLE #name
...
do your insert
You need Dynamic SQL to create the tables names appended with date.
CREATE PROC usp_createtable( #tablename VARCHAR(20),
#Dbname VARCHAR(20),
#SchemaName VARCHAR(20))
AS
BEGIN
DECLARE #sql NVARCHAR(max)
SET #sql =' SELECT * INTO '+#Dbname+'.'+#SchemaName+'.'+#tablename+'CONVERT(VARCHAR(8), GETDATE(), 112) FROM '+#Dbname+'.'+#SchemaName+'.'+#tablename''
EXEC sp_executesql
#sql
END
I need to write a query engine on a web app, what needs to be accomplish is that a user can enter any SELECT statement into a textbox and then the results should be created into a new table.
This is my function I have created but it only support SQL Server 2012 and I want similar to this function but only it should support SQL Server 2005 and above:
CREATE FUNCTION [dbo].[CustomQueryTableCreation]
(
#TableName varchar(max),
#sql NVARCHAR(MAX)
)
RETURNS
#TableBuilder TABLE
(
DS varchar(max)
)
BEGIN
INSERT INTO #TableBuilder
SELECT 'CREATE TABLE dbo.' + #TableName+'(';
INSERT INTO #TableBuilder
SELECT
CASE column_ordinal
WHEN 1 THEN '' ELSE ',' END
+ name + ' ' + system_type_name + CASE is_nullable
WHEN 0 THEN ' not null' ELSE '' END
FROM
sys.dm_exec_describe_first_result_set
(
#sql, NULL, 0
) AS f
ORDER BY
column_ordinal;
INSERT INTO #TableBuilder
SELECT ');';
RETURN
END
What I want to do now is that I want to search through my query and replace the FIRST FROM with INTO NewTable FROM.
The query can contain multiple joins.
Should I control this with SQL or C#?
I had a similar problem with the 2005 Environment. If you save the Select query to a table, and use the following built in procedure to execute the query:
EXECUTE sp_executesql #Query
Here is the MS docs:
http://msdn.microsoft.com/en-us/library/ms188001%28v=sql.90%29.aspx
Edit
Keeping this in mind, can take the SQL dumps and Create OpenRowset Queries to take the SQL and dump them into a TempTable, and from the Temp Table to a permanent table if required.
I created the following SP's to assist with getting the info to a permanent table.
First the procedure to execute the specific SQL Statement
CREATE PROCEDURE [dbo].[spExecuteRowset]
(
#Query NVARCHAR(MAX)
)
AS
BEGIN
--Execute SQL Statement
EXECUTE sp_executesql #Query
END
Then the OpenRowset SP:
CREATE PROCEDURE [dbo].[spCustomquery]
(
#ProQuery NVARCHAR(MAX),
#Tablename NVARCHAR(MAX)
)
AS
BEGIN
--Insert the info into a Specidied Table
DECLARE #Query NVARCHAR(max)
SET #Query = 'SELECT * INTO #MyTempTable FROM OPENROWSET(''SQLNCLI'', ''Server=localhost;Trusted_Connection=yes;'','' EXEC [YOUR DATABASE].dbo.spExecuteRowset' +''''+#ProQuery+''''') SELECT * INTO '+ #Tablename +' FROM #MyTempTable'
--FOR DEBUG ONLY!!!!
PRINT #Query
EXEC [YourDatabase].dbo.spExecuteRowset #Query
END
This takes it from the #tempTable to A Physical Table.
Here are some docs on OpenRowset.
You have no guarantee that the first from in a query will accept an into, because you can have a subselect in the select statement. In addition, you could have a field name like datefrom that throws things off too.
But, assuming you have "simple" SQL statements, you can do it as:
select stuff(#query, charindex('from ', #query), 0, 'into '+#Table+' ')
from t;
EDIT:
The following is what you really want to do:
select *
into #Table
from (#query) q;
Using the subquery solves the problem.
This is a well-known problem. String concatenation is usually a bad/limited solution.
The more recommended solution is to let some other mechanism to return you the result set (openquery etc.), and then insert it to a table.
For example:
SELECT *
INTO YourTable
FROM OPENQUERY([LinkedServer],your query...)