Automating SQL view exports to Excel - sql-server

I have a view written in SQL Server. I need to export the data to Excel so that there are separate Excel files for every possible two letter country code. Files are named "Report - Country Date.xslx", for example "Report - Australia 06082018.xlsx".
So far I have just connected the view to Excel and repeated following steps for each country:
Change the country code in the SQL view
Save the Excel file with a name matching the country in question
There are some 100 different countries, so the work is quite repetitive and tedious. There are also too many rows in the query results to export several countries' data at the same time. Is there a way to automate this process, even just some part of it?

Okay, normally i would recommend SSIS to do this. Its an excellent tool to handle your question. However if you REALLY want to do it in SQL you can do something like this:
NB's
I could only get it to work by using xls excel files.
It will give you a warning about its not trusted source however you can just click OK
I havent tested it deeply so there can be errors or fixes that needs to be made :)
Prestep: Load sample data
CREATE TABLE [dbo].[TestExcel](
[CountryCode] [varchar](50) NULL,
[Names] [varchar](50) NULL,
[Sales] [int] NULL
)
Insert into dbo.TestExcel (CountryCode,Names,Sales)
Values
('DA','Thomas', 10),
('DA','Jens', 20),
('DA','John', 40),
('EN','Mark', 10),
('EN','Adam', 5 )
Step 1:
Create a stored procedure
Create procedure proc_generate_excel_with_columns
(
#db_name varchar(100),
#table_name varchar(100),
#file_name varchar(100),
#CountryCode varchar(10)
)
as
--Generate column names as a recordset
declare #columns varchar(8000), #sql varchar(8000), #data_file varchar(100)
select
#columns=coalesce(#columns+',','')+column_name+' as '+column_name
from
information_schema.columns
where
table_name=#table_name
select #columns=''''''+replace(replace(#columns,' as ',''''' as '),',',',''''')
PRINT 'Headers'
PRINT #columns
PRINT 'Create a dummy file to have actual data'
select #data_file=substring(#file_name,1,len(#file_name)-charindex('\',reverse(#file_name)))+'\data_file_'+cast(cast(GETDATE() as date) as nvarchar)+'.xls'
print #data_file
print 'Generate column names in the passed EXCEL file'
set #sql='exec master..xp_cmdshell ''bcp " select * from (select '+#columns+') as t" queryout "'+#file_name+'_'+#CountryCode+cast(cast(GETDATE() as date) as nvarchar)+'.xls" -S "EGC25199\SQL2016" -T -c'''
print #sql
exec(#sql)
print 'Generate data in the dummy file'
set #sql='exec master..xp_cmdshell ''bcp "select * from '+#db_name+'..'+#table_name+' where CountryCode = '''''+#CountryCode+'''''" queryout "'+#data_file+'" -S "EGC25199\SQL2016" -T -c'''
print #sql
exec(#sql)
--Copy dummy file to passed EXCEL file
set #sql= 'exec master..xp_cmdshell ''type '+#data_file+' >> "'+#file_name+'_'+#CountryCode+cast(cast(GETDATE() as date) as nvarchar)+'.xls"'''
exec(#sql)
--Delete dummy file
set #sql= 'exec master..xp_cmdshell ''del '+#data_file+''''
exec(#sql)
Step 2: Execute stored procedure
USE [LegOgSpass]
GO
DECLARE #RC int
DECLARE #db_name varchar(100) = 'LegOgSpass'
DECLARE #table_name varchar(100) = 'TestExcel'
DECLARE #file_name varchar(100) = 'D:\Test\TestExcel'
DECLARE #CountryCode varchar(10) = 'EN'
-- TODO: Set parameter values here.
EXECUTE #RC = [dbo].[proc_generate_excel_with_columns]
#db_name
,#table_name
,#file_name
,#CountryCode
Result - Picture of Folder
Result of DA
Result of EN

Python can also help in automating SQL data ->Excel. This sample code creates 4 excel sheets using pyexcelerate.
#•pyexcelerate
import pyexcelerate as px
from datetime import date
wb = px.Workbook()
ws = wb.new_sheet("Report")
n = date.today()
ReportDate = "{:02.0f}{:02.0f}{}".format(n.day,n.month,n.year)
ws.range("B2", "C3").value = [[1, 2], [3, 4]]
Countrylist = ['USA', 'Australia','China','Nigeria']
#"Report - Australia 06082018.xlsx".
for c in countrylist:
wb.save("Report -"+c+' '+str(ReportDate)+".xlsx")

Related

Create view that selects from identical tables from all databases on the server and can handle any time a new database is added to the server

I have a system that takes in Revit models and loads all the data in the model to a 2016 SQL Server. Unfortunately, the way the system works it created a new database for each model that is loaded. All the databases start with an identical schema because there is a template database that the system uses to build any new ones.
I need to build a view that can query data from all databases on the server but can automatically add new databases as they are created. The table names and associated columns will be identical across all databases, including data types.
Is there a way to pull a list of current database names using:
SELECT [name] FROM sys.databases
and then use the results to UNION the results from a basic SELECT query like this:
SELECT
[col1]
,[col2]
,[col3]
FROM [database].[dbo].[table]
Somehow replace the [database] part with the results of the sys.databases query?
The goal would be for the results to look as if I did this:
SELECT
[col1]
,[col2]
,[col3]
FROM [database1].[dbo].[table]
UNION
SELECT
[col1]
,[col2]
,[col3]
FROM [database2].[dbo].[table]
but dynamically for all databases on the server and without future management from me.
Thanks in advance for the assistance!
***Added Info: A couple suggestions using STRING_AGG have been made, but that function is not available in 2016.
Try this. It will automatically detect and include new databases with the specified table name. If a database is dropped it will automatically exclude it.
I updated the TSQL. STRING_AGG concatenates the string with each database. Without it it only returns the last database. STRING_AGG is more secure than += which also concatenates. I changed the code so it generates and executes the query. In SQL 2019 the query is all in one line using +=. I don't have SQL 2016. It may format it better in SQL 2016. You can uncomment --SELECT #SQL3 to see what the query looks like. Please mark as answer if this is what you need.
DECLARE #TblName TABLE
(
TblName VARCHAR(100)
)
Declare #SQL VARCHAR(MAX),
#SQL3 VARCHAR(MAX),
#DBName VARCHAR(50),
#Count Int,
#LoopCount Int
Declare #SQL2 VARCHAR(MAX) = ''
Select Identity(int,1,1) ID, name AS DBName into #Temp from sys.databases
Select #Count = ##RowCount
Set #LoopCount = 1
While #LoopCount <= #Count
Begin
SET #DBName = (SELECT DBName FROM #Temp Where ID = #LoopCount)
SET #SQL =
' USE ' + #DBName +
' SELECT TABLE_CATALOG FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = ''table'''
INSERT INTO #TblName (TblName)
EXEC (#SQL)
Set #LoopCount=#LoopCount + 1
End
SELECT #SQL2 +=
' SELECT ' + char(10) + 
' [col1] ' + char(10) + 
' ,[col2] ' + char(10) + 
' ,[col3] ' + char(10) + 
' FROM [' + TblName + '].[dbo].[table] ' + char(10) + 
' UNION '
FROM #TblName
DROP TABLE #Temp
SET #SQL3 = (SELECT SUBSTRING(#SQL2, 1, LEN(#SQL2) - 5))
--SELECT #SQL3
EXEC (#SQL3)

Get data from many databases - dynamic database

We are using SQL Server 2014 Enterprise with many databases. I have to execute query and get reports / data from every database with EXACT SAME Schema and database starts with Cab
When a new company is added in our ERP project a new database is created with exact schema starting with Cab and incremented number is assigned to it like:
Cab1
Cab2
Cab3
Cab5
Cab10
I can get the database names as:
SELECT name
FROM master.sys.databases
where [name] like 'Cab%' order by [name]
I have to create a Stored Procedure to get data from tables of every database.
How to do that using a Stored Procedure as the databases are created dynamically starting with Cab?
You can use EXEC(#Statement) or EXEC SP_EXECUTESQL if you have to pass parameters.
CREATE OR ALTER PROCEDURE dbo.GetDataFromAllDatabases
AS
BEGIN
DECLARE #T TABLE (id INT NOT NULL IDENTITY(1, 1), dbName VARCHAR(256) NOT NULL)
INSERT INTO #T
SELECT NAME FROM MASTER.SYS.DATABASES WHERE [NAME] LIKE 'Cab%' ORDER BY [NAME]
CREATE TABLE #AllData (......)
DECLARE #Id INT, #DbName VARCHAR(128)
SELECT #Id = MIN(Id) FROM #T
WHILE #Id IS NOT NULL
BEGIN
SELECT #DbName = dbName FROM #T WHERE Id = #Id
DECLARE #Statement NVARCHAR(MAX)
SET #Statement = CONCAT(N'INSERT INTO #AllData (...) SELECT .... FROM ', #DbName, '.dbo.[TableName]')
EXEC(#Statement);
--YOU CAN USE BELOW LINE TOO IF YOU NEED TO PASS VARIABLE
--EXEC SP_EXECUTESQL #Statement, '#Value INT', #Value = 128
SET #Id = (SELECT MIN(Id) FROM #T WHERE Id > #Id)
END
END
A quick and easy dynamic SQL solution would be something like this:
DECLARE #Sql nvarchar(max);
SET #Sql = STUFF((
SELECT ' UNION ALL SELECT [ColumnsList], '''+ [name] + ''' As SourceDb FROM '+ QUOTENAME([name]) + '.[SchemaName].[TableName]' + char(10)
FROM master.sys.databases
WHERE [name] LIKE 'Cab%'
FOR XML PATH('')
), 1, 10, '');
--When dealing with dynamic SQL, print is your best friend...
PRINT #Sql
-- Once the #Sql is printed and you can see it looks OK, you can run it.
--EXEC(#Sql)
Notes:
Use quotename to protect against "funny" chars in identifiers names.
Replace [ColumnsList] with the actual list of columns you need.
There's no need for loops of any kind, just a simple stuff + for xml to mimic string_agg (which was only introduced in 2017).
I've thrown in the source database name as a "bonus", if you don't want it that's fine.
The Order by clause in the query that generates the dynamic SQL is meaningless for the final query, so I've removed it.

Trigger MSSQL - Output to File (XML)

we have a very old ERP system which is badly supported.
Now the warehouse want´s to buy a new "store system" for our goods. It´s a fully automatic store system which need´s data from our ERP system. The support of your ERP system can´t help us, so we have to build a solution of our own.
The idea was to "move" the items for the new storage system to a special storage place called (SHUT1) and output the "part number" and "quantity" to a file (xml) which can be read by the new software.
We can´t change anything in the software of our ERP system, so we have to do it on the SQL Server itself.
(I know, a trigger is not the "best" thing to do, but I have have no other choice)
CREATE TRIGGER tr_LagerShut ON dbo.Lagerverwaltung
AFTER INSERT
AS
BEGIN
IF (SELECT [Lagerort] from Inserted) = 'SHUT1'
BEGIN
DECLARE #Cmd VARCHAR(2000) ;
DECLARE #FormatDate4File VARCHAR(200);
SET #FormatDate4File = (SELECT(SYSUTCDATETIME()));
SET #FormatDate4File = (SELECT REPLACE(#FormatDate4File,' ','-'));
SET #FormatDate4File = (SELECT REPLACE(#FormatDate4File,':','-'));
SET #FormatDate4File = (SELECT REPLACE(#FormatDate4File,'.','-'));
SET #Cmd = ( SELECT [Artikelnummer],[Menge] FROM inserted FOR XML PATH('')) ;
SET #Cmd = 'Echo "' + #Cmd + '" >>"C:\Temp\' + #FormatDate4File +'.xml"' ;
EXEC xp_cmdshell #Cmd ;
END;
END;
The trigger "installs" fine, but if I change a storage place to a new one, the ERP system stalls with "ERROR" (there is no error description :(
If I drop the trigger the system is just running fine again. So I think there is a error in the trigger, but I can´t find it.
Can anybody help please?
Aleks.
Don't know what ERP system stalls with "ERROR" looks like... Frozend GUI? Timeout? Just no file created?
My magic glass bulb tells me the following: You are inserting more than one row at once. If so, this statement will break, because a comparison like this is only valid against a scalar value. If there is more than one row in inserted, this will not work:
IF (SELECT [Lagerort] from Inserted) = 'SHUT1'.
Your trigger can be simplified, but I doubt, that you will like the result. Check this with special characters (like üöä) and check for enclosing "-characters...
CREATE TRIGGER tr_LagerShut ON dbo.Lagerverwaltung
AFTER INSERT
AS
BEGIN
IF EXISTS(SELECT 1 FROM inserted WHERE [Lagerort]='SHUT1')
BEGIN
DECLARE #FileName VARCHAR(255) =REPLACE(REPLACE(REPLACE(SYSUTCDATETIME(),' ','-'),':','-'),'.','-');
DECLARE #Content XML=
(
SELECT [Artikelnummer],[Menge] FROM inserted WHERE [Lagerort]='SHUT1' FOR XML AUTO,ELEMENTS
);
DECLARE #Cmd VARCHAR(4000) = 'Echo "' + CAST(#Content AS VARCHAR(MAX)) + '" >>"c:\temp\' + #FileName + '.xml"' ;
PRINT #cmd;
EXEC xp_cmdshell #Cmd ;
END
END;
This might better be done with BCP.
UPDATE Your comment...
First you should check, if this works at all:
DECLARE #FileName VARCHAR(255) =REPLACE(REPLACE(REPLACE(SYSUTCDATETIME(),' ','-'),':','-'),'.','-');
DECLARE #Content XML=
(
SELECT TOP 5 * FROM sys.objects FOR XML AUTO,ELEMENTS
);
DECLARE #Cmd VARCHAR(4000) = 'Echo "' + CAST(#Content AS VARCHAR(MAX)) + '" >>"c:\temp\' + #FileName + '.xml"' ;
PRINT #cmd;
EXEC xp_cmdshell #Cmd ;
If you find no file in c:\temp\: Are you aware, that SQL-Server will always write in its own context? Might be, that you are awaiting a file in your local c-drive, but the file is written to the Server's machine acutally.
If this works isolatedly, it should work within a trigger too. You might pack the call into BEGIN TRY ... END TRY and add an appropriate CATCH block.
So okay, the "simple" trigger could be really a problem. Now I have this idea:
(one more info: time stamp is not inserted into table "Lagerverwaltung" when a new row is inserted)
Pseudo Code on:
Trigger on Table "Lagerverwaltung"
Check if the storage place(s) is "SHUT1"
If "yes"
DECLARE #FileName VARCHAR(255) =REPLACE(REPLACE(REPLACE(SYSUTCDATETIME(),' ','-'),':','-'),'.','-');
create a new table with name dbo + '.' + #filename
Insert all the data (inserted) + row(SYSUTCDATETIME()) AS TimeStamp where 'Lagerort' = 'SHUT1' into new table named 'dbo.' + #filename
DECLARE #Cmd varchar(4000) = 'bcp "select [Artikelnummer],[Menge],[TimeStamp] FROM [wwsbautest].[dbo].#filename WHERE [Lagerort]=''SHUT1'' AND [Menge] > ''0'' FOR XML AUTO, ELEMENTS" queryout "C:\temp\' + #FileName + '.xml" -c -T';
EXEC xp_cmdshell #Cmd;
Drop table dbo.#filename;
Could somthing like that work?

Replace first FROM in sql query

I need to write a query engine on a web app, what needs to be accomplish is that a user can enter any SELECT statement into a textbox and then the results should be created into a new table.
This is my function I have created but it only support SQL Server 2012 and I want similar to this function but only it should support SQL Server 2005 and above:
CREATE FUNCTION [dbo].[CustomQueryTableCreation]
(
#TableName varchar(max),
#sql NVARCHAR(MAX)
)
RETURNS
#TableBuilder TABLE
(
DS varchar(max)
)
BEGIN
INSERT INTO #TableBuilder
SELECT 'CREATE TABLE dbo.' + #TableName+'(';
INSERT INTO #TableBuilder
SELECT
CASE column_ordinal
WHEN 1 THEN '' ELSE ',' END
+ name + ' ' + system_type_name + CASE is_nullable
WHEN 0 THEN ' not null' ELSE '' END
FROM
sys.dm_exec_describe_first_result_set
(
#sql, NULL, 0
) AS f
ORDER BY
column_ordinal;
INSERT INTO #TableBuilder
SELECT ');';
RETURN
END
What I want to do now is that I want to search through my query and replace the FIRST FROM with INTO NewTable FROM.
The query can contain multiple joins.
Should I control this with SQL or C#?
I had a similar problem with the 2005 Environment. If you save the Select query to a table, and use the following built in procedure to execute the query:
EXECUTE sp_executesql #Query
Here is the MS docs:
http://msdn.microsoft.com/en-us/library/ms188001%28v=sql.90%29.aspx
Edit
Keeping this in mind, can take the SQL dumps and Create OpenRowset Queries to take the SQL and dump them into a TempTable, and from the Temp Table to a permanent table if required.
I created the following SP's to assist with getting the info to a permanent table.
First the procedure to execute the specific SQL Statement
CREATE PROCEDURE [dbo].[spExecuteRowset]
(
#Query NVARCHAR(MAX)
)
AS
BEGIN
--Execute SQL Statement
EXECUTE sp_executesql #Query
END
Then the OpenRowset SP:
CREATE PROCEDURE [dbo].[spCustomquery]
(
#ProQuery NVARCHAR(MAX),
#Tablename NVARCHAR(MAX)
)
AS
BEGIN
--Insert the info into a Specidied Table
DECLARE #Query NVARCHAR(max)
SET #Query = 'SELECT * INTO #MyTempTable FROM OPENROWSET(''SQLNCLI'', ''Server=localhost;Trusted_Connection=yes;'','' EXEC [YOUR DATABASE].dbo.spExecuteRowset' +''''+#ProQuery+''''') SELECT * INTO '+ #Tablename +' FROM #MyTempTable'
--FOR DEBUG ONLY!!!!
PRINT #Query
EXEC [YourDatabase].dbo.spExecuteRowset #Query
END
This takes it from the #tempTable to A Physical Table.
Here are some docs on OpenRowset.
You have no guarantee that the first from in a query will accept an into, because you can have a subselect in the select statement. In addition, you could have a field name like datefrom that throws things off too.
But, assuming you have "simple" SQL statements, you can do it as:
select stuff(#query, charindex('from ', #query), 0, 'into '+#Table+' ')
from t;
EDIT:
The following is what you really want to do:
select *
into #Table
from (#query) q;
Using the subquery solves the problem.
This is a well-known problem. String concatenation is usually a bad/limited solution.
The more recommended solution is to let some other mechanism to return you the result set (openquery etc.), and then insert it to a table.
For example:
SELECT *
INTO YourTable
FROM OPENQUERY([LinkedServer],your query...)

How to create text files of database rows?

I have a database table with a column named File Content, and many rows. What I need is to create a text file for each row of File Content column.
Example:
Sr. File Name File Content
1. FN1 Hello
2. FN2 Good Morning
3. FN3 How are you?
4. FN4 Where are you?
Suppose I have 4 rows, then 4 text files should be created (maybe with any name which we want)
File1.txt should have text "hello" in it.
File2.txt should have text "Good Morning" in it.
File3.txt should have text "How are you?" in it.
File4.txt should have text "Where are you?" in it
Although you said you said you need to do it in TSQL, I wouldn't do it that way if possible. Ram has shown you one solution, but it has the disadvantages that you need to use xp_cmdshell and the SQL Server service account needs permission to access the file system in whatever location you want to have the files.
My suggestion would be to write a script or small program in your preferred language (PowerShell, Perl, Python, C#, whatever) and use that instead. TSQL as a language is simply badly suited for manipulating files or handling anything outside the database. It is obviously possible (CLR procedures are another way), but you often run into problems with permissions, encodings and other issues that are much easier to deal with in an external language.
This can be done with BCP OUT syntax of SQL server.
For the setup: just make sure that you have xp_cmdshell exec permissions on the server. This can be checked from master.sys.configurations table. Also change filelocation path corresponding to your server or network share. I checked and was able to generate 4 files as there are 4 records in the table.
use master
go
declare #DSQL Nvarchar(max)
declare #counter int
declare #maxrows int
declare #filename Nvarchar(30)
select #counter=1, #maxrows = 0
create table t1 (
sno int identity(1,1) not null,
filename varchar(5),
filecontent varchar(100)
)
insert into t1
select 'FN1', 'Hello'
UNION
select 'FN2', 'Good Morning'
UNION
select 'FN3', 'How are you?'
UNION
select 'FN14', 'Where are you?'
select #maxrows = count(*) from t1
--SELECT * FROM T1
while (#counter <= #maxrows)
begin
select #filename = filename from t1
where sno = #counter
select #DSQL = N'exec xp_cmdshell' + ' ''bcp "select filecontent from master.dbo.T1 where sno = ' + cast(#counter as nvarchar(10)) + '" queryout "d:\temp\' + #filename + '.txt" -T -c -S home-e93994b54f'''
print #dsql
exec sp_executesql #DSQL
select #counter = #counter + 1
end
drop table t1

Resources