I am using SQL Server 2005. I have a few SSIS packages located here: C:\SSIS
This code below is used to execute all the packages but I still need to place each package name in a table called Packages.
Can I execute all the packages without having to save the package name? I just want to supply the path of where they all are sitting and want SQL to do the rest.
DECLARE #package_name varchar(200)
Declare #PackageCount int
Declare #X int
Set #X = 1
Set #PackageCount = (Select COUNT(*) from Packages)
set #FilePath = 'C:\SSIS'
While (#X <= #PackageCount)
Begin
With PackageList as
(
Select PackageName, Row_Number() Over(Order by PackageName) as Rownum
From Packages
)
SELECT #package_name = PackageName
FROM PackageList
Where Rownum = #X
select #cmd = 'DTExec /F "' + #FilePath + #Package_name + '"'
print #cmd
Set #X = #X + 1
exec master..xp_cmdshell #cmd
End
you would need to use xp_cmdshell to execute a loop on the folder and get the file names.
Here is an example on how to do it. Of course, you'll need to clean up the result and get only the rows that matter
SET NOCOUNT ON
DECLARE #Command VARCHAR(100)
SET #Command = 'dir C:\test'
DECLARE #Folder VARCHAR(100)
SET #Folder = 'C:\test'
DECLARE #FilesInAFolder TABLE (FileNamesWithFolder VARCHAR(500))
INSERT INTO #FilesInAFolder
EXEC MASTER..xp_cmdshell #Command
select * from #FilesInAFolder
How to enable xp_cmdshel:
EXEC sp_configure 'show advanced options', 1
GO
RECONFIGURE
GO
EXEC sp_configure 'xp_cmdshell', 1
GO
RECONFIGURE
GO
Why not just use SSIS to do this? Drop a Foreach Loop container onto a new package. Grab the fully qualified path for all .dtsx files. Inside the foreach loop, have an Execute Package Task and assign it the current package path.
This would reduce your problem to run a single package which you can solve through a host of measures (.NET, SQL Agent, windows scheduler, etc)
Related
I have an MS Access database that we're converting to a SQL Server backend. This database has an Attachments table with a few simple columns:
PK, FK to MainTable.RecordID, Description, filename
Attachments are stored in a fileshare. VBA code uses a hardcoded filepath and ShellExecute to save attachments to a directory, under a RecordID subfolder.
We're moving to store attachments in SQL Server using filestream.
I need to move these attachments from fileshare, to SQL Server, while maintaining RecordID integrity. SQL Server tables and columns are already set up.
These attachments vary in extensions (.msg, .doc, .xlsx, .pdf)
I've been looking into "OPENROWSET" but every example I've seen uses only one file.
I've been looking into SSMA but can't find what I'm looking for.
Any references/reference articles or code resources I can use/repurpose would be greatly appreciated.
Sounds like you want to write an SQL stored procedure that will find all files in a given file path, iterate over those files, and insert the file into a table.
This article will help in general: https://www.mssqltips.com/sqlservertip/5432/stored-procedure-to-import-files-into-a-sql-server-filestream-enabled-table/
This article is about xp_dirtree: https://www.sqlservercentral.com/blogs/how-to-use-xp_dirtree-to-list-all-files-in-a-folder
Here's sample code to read the file system from SQL. THIS IS UNTESTED CODE, you'll need to modify to your needs but it gives you some idea of how to do the loops and read in files.
--You will need xm_cmdshell enabled on SQL server if not already.
USE master
GO
EXEC sp_configure 'show advanced option',1
RECONFIGURE WITH OVERRIDE
EXEC sp_configure 'xp_cmdshell',1
RECONFIGURE WITH OVERRIDE
GO
--Create a variable to hold the pickup folder.
DECLARE #PickupDirectory nvarchar(512) = '\\folder_containing_files_or_folders\';
--Create a temp table to hold the files found in the pickup directory.
PRINT 'Parsing directory to identify most recent file.';
DECLARE #DirTree TABLE (
id int IDENTITY(1,1)
, subdirectory nvarchar(512)
, depth int
, isfile bit
);
--Enumerate the pickup directory.
INSERT #DirTree
EXEC master.sys.xp_dirtree #PickupDirectory,1,1 --Second variable is depth.
--Create variables to loop through folders and files.
DECLARE #folderCount int;
DECLARE #folderName nvarchar(max);
DECLARE #folderPath nvarchar(max);
DECLARE #i int = 0;
DECLARE #fileCount int;
DECLARE #fileName NVARCHAR(max);
DECLARE #filePath varchar(max);
DECLARE #j int = 0;
DECLARE #RecordID nvarchar(50);
DECLARE #SQLText NVARCHAR(max);
SET #folderCount = (SELECT Count(*) FROM #DirTree WHERE isfile = 0);
WHILE ( #i < #folderCount )
BEGIN
--Get the next folder to process.
SET #folderName = (
SELECT TOP 1 subdirectory
FROM #DirTree as dt
LEFT OUTER JOIN #processedFolders as pf
on pf.folder_name = dt.subdirectory
WHERE isfile = 0
AND pf.folder_name IS NULL
);
--Get the recordID from folder name.
SET #recordID = #folderName; --Edit this to get the RecordID from your folder structure.
--Concat root path and new folder to get files from.
SET #folderPath = #PickupDirectory + #folderName + '\';
--Enumerate the this subdirectory to process files from.
INSERT #filesToProcess
EXEC master.sys.xp_dirtree #folderPath,1,1
--Get count of files to loop through.
SET #fileCount = (SELECT COUNT(*) FROM #filesToProcess WHERE isfile = 1);
WHILE (#j < #fileCount)
BEGIN
--Get next filename.
SET #fileName = (SELECT TOP 1 subdirectory FROM #filesToProcess WHERE isfile = 1);
--Concat the whole file path.
SET #filePath = #folderPath + #fileName;
SET #SQLText = '
INSERT INTO [table_name](RecordID,[filename],[filestreamCol])
SELECT
''' + #RecordID + '''
, ''' + #fileName + '''
, BulkColumn
FROM OPENROWSET(Bulk ''' + #filePath + ''', Single_Blob) as tb'
EXEC Sp_executesql #SQLText
DELETE FROM #filesToProcess
WHERE subdirectory = #fileName;
SET #j = #j + 1;
END
INSERT INTO #processedFolders (folder_name)
SELECT #folderName;
PRINT 'Folder complete: ' + #folderName;
SET #i = #i + 1
END
I think you want to parse just a root directory with the xp_dirtree command above. That will display all the subdirectories which should contain the "RecordID". Read the RecordID into a variable, then parse each of those subdirectories to get the actual files. If you want more detailed code, you'll have to show some examples of the directory structure and the destination table.
I have this stored procedure; I am printing the value of the variable in SSMS.
Instead, I want to store this result in a .txt file.
NOTE: I don't want to do it using SSMS options of right clicking on the result and then saving the result as. I want it to be done using any SQL code/built-in function directly in the stored procedure itself.
CREATE PROCEDURE [dbo].[usp_printresulttofile]
AS
BEGIN
DECLARE #var NVARCHAR(MAX) = ''
SET #var = 'print this data in txt file'
PRINT 'Data is : ' + #var
/* SQL query here to store result of Print statement in text file */
END
EXEC [dbo].[usp_printresulttofile]
Sharing the updated working SP here so that it might be useful to someone with a similar requirement Thanks #David Browne - Microsoft
ALTER PROCEDURE [dbo].[usp_printresulttofile]
AS
BEGIN
DECLARE #fileTimeStamp varchar(200) = convert(varchar,getDate(), 112 )+'_'+ Replace(convert(varchar,getDate(), 114 ),':','') -- select convert(varchar, getdate(), 121)
DECLARE #fileExtension varchar(5) = 'txt'
DECLARE #var NVARCHAR(MAX) = ''
SET #var = 'print this data in txt file'
PRINT 'Data is : ' + #var
declare #fn varchar(500) = 'c:/log/SP_output_'+#fileTimeStamp+'.'+#fileExtension;
declare #cmd varchar(8000) = concat('echo ', #var, ' > "', #fn, '"');
print #cmd
exec xp_cmdshell #cmd, no_output
set #cmd = concat('type "', #fn, '"');
print #cmd
exec xp_cmdshell #cmd;
END
GO
As the comments and other answers indicate, this is not usually a good idea. But here's how to do it anyway, assuming you're a sysadmin on SQL Server. :)
-- To allow advanced options to be changed.
EXEC sp_configure 'show advanced options', 1;
GO
-- To update the currently configured value for advanced options.
RECONFIGURE;
GO
-- To enable the feature.
EXEC sp_configure 'xp_cmdshell', 1;
GO
-- To update the currently configured value for this feature.
RECONFIGURE;
GO
CREATE OR ALTER PROCEDURE [dbo].[usp_printresulttofile]
AS
BEGIN
DECLARE #var NVARCHAR(MAX) = ''
SET #var = 'print this data in txt file'
PRINT 'Data is : ' + #var
declare #fn varchar(200) = 'c:\temp\out.txt';
declare #cmd varchar(8000) = concat('echo ', #var, ' > "', #fn, '"');
print #cmd
exec xp_cmdshell #cmd, no_output
set #cmd = concat('type "', #fn, '"');
print #cmd
exec xp_cmdshell #cmd;
END
go
EXEC [dbo].[usp_printresulttofile]
I might recommend creating a batch file to accomplish this. You can simply do this:
sqlcmd -i ExecSQLProc.sql > Results.txt
Save your EXEC command in a file named ExecSQLProc.sql and then create a batch script with the line above. You can also attach this to the scheduler if needed so that you can regularly generate these scripts. Also you can create a process to output whatever your process produces into an email using only SQL server and see the results this way as well.
Generally I have found it better to use the Operating System to manipulate individual files, if you want to append results you have options:
sqlcmd -i ExecSQLProc.sql >> CumulativeResults.txt
Update your stored process to keep track of your data in the database that is a large container - a blob for instance, and then when you execute your script it will generate a file that has everything in it.
Use bcp command to copy data into any format. You just need to mention the format in which you want. Like .text
bcp 'select * from table' queryout c:\sql\bcp.txt -c -T
Example and explaination in below link :-
https://www.mssqltips.com/sqlservertip/4353/export-sql-server-records-into-individual-text-files/
Just remember to be careful with the string going to the text file. If it has quotes / double quotes, etc. it will affect its ability to save. I automatically put double quotes around my outgoing string and it typically helps but if your string is more complicated (i.e. a SQL query with literal comparisons) then you likely will have more work to do.
I have files in my computer's folder with following names:
XXX.IN.txt
YYY.TEST.NUM.txt
ABC.AA.Z100.X.E999567777.Y001.txt
ABC.AA.Z100.X.E999568888.Y002.txt
ABC.AA.Z100.X.E999568888.Y003.txt
I want to write a SQL statement that would insert files that have the above described structure into a table, so I can later on write some logic on them.
I already used the command line statement inside of my stored proc to check if files exist:
EXEC master.dbo.xp_fileexist #fullPath, #exist OUTPUT
SET #exist = CAST(#exist AS BIT)
Now, I need to find certain files containing string in their names. I have the statement to do that:
DECLARE #cmdLine VARCHAR(200)
DECLARE #fullPath VARCHAR(900) = '\\my_network_path\MyDir\'
DECLARE #filter VARCHAR(100) = 'ABC.AA.Z100.X.*.txt'
SET #cmdLine = 'dir "' + #fullPath + '"'
EXEC master..xp_cmdshell #cmdLine
Above command should give me the following files:
ABC.AA.Z100.X.E999567777.Y001.txt
ABC.AA.Z100.X.E999568888.Y002.txt
ABC.AA.Z100.X.E999568888.Y003.txt
CREATE TABLE #FileDetails
(
data VARCHAR(MAX)
)
INSERT #FileDetails(data) EXEC master..xp_cmdshell #cmdLine
But it lists all the .txt files in the folder
How would I do list only those files that I need
First of all, the #cmdline should be set higher than #fullpath since it's supposed to fit all of it in the end.
Second of all, unless I am looking at it wrong or you didn't correct it here, the #filter variable isn't being used, so it would show every file regardless of extension.
My Code:
DECLARE #cmdLine VARCHAR(2000)
DECLARE #fullPath VARCHAR(1000) = '\\my_network_path\MyDir\'
DECLARE #filter VARCHAR(100) = 'ABC.AA.Z100.X.*.txt'
SET #cmdLine = 'dir "' + #fullPath + #filter + '"'
EXEC master..xp_cmdshell #cmdLine
My Output (keep in mind I created a Test.txt in the same folder):
10-10-2017 12:17 0 ABC.AA.Z100.X.Y001.txt
10-10-2017 12:18 0 ABC.AA.Z100.X.Y002.txt
10-10-2017 12:18 0 ABC.AA.Z100.X.Y003.txt
I would do this with either a CLR proc or an SSIS Script that uses the FileSystemObject to iterate through the files, filter for the ones you want and build a SQL String and execute it.
I don't know any way to do what you want with just straight TSQL.
we have a very old ERP system which is badly supported.
Now the warehouse want´s to buy a new "store system" for our goods. It´s a fully automatic store system which need´s data from our ERP system. The support of your ERP system can´t help us, so we have to build a solution of our own.
The idea was to "move" the items for the new storage system to a special storage place called (SHUT1) and output the "part number" and "quantity" to a file (xml) which can be read by the new software.
We can´t change anything in the software of our ERP system, so we have to do it on the SQL Server itself.
(I know, a trigger is not the "best" thing to do, but I have have no other choice)
CREATE TRIGGER tr_LagerShut ON dbo.Lagerverwaltung
AFTER INSERT
AS
BEGIN
IF (SELECT [Lagerort] from Inserted) = 'SHUT1'
BEGIN
DECLARE #Cmd VARCHAR(2000) ;
DECLARE #FormatDate4File VARCHAR(200);
SET #FormatDate4File = (SELECT(SYSUTCDATETIME()));
SET #FormatDate4File = (SELECT REPLACE(#FormatDate4File,' ','-'));
SET #FormatDate4File = (SELECT REPLACE(#FormatDate4File,':','-'));
SET #FormatDate4File = (SELECT REPLACE(#FormatDate4File,'.','-'));
SET #Cmd = ( SELECT [Artikelnummer],[Menge] FROM inserted FOR XML PATH('')) ;
SET #Cmd = 'Echo "' + #Cmd + '" >>"C:\Temp\' + #FormatDate4File +'.xml"' ;
EXEC xp_cmdshell #Cmd ;
END;
END;
The trigger "installs" fine, but if I change a storage place to a new one, the ERP system stalls with "ERROR" (there is no error description :(
If I drop the trigger the system is just running fine again. So I think there is a error in the trigger, but I can´t find it.
Can anybody help please?
Aleks.
Don't know what ERP system stalls with "ERROR" looks like... Frozend GUI? Timeout? Just no file created?
My magic glass bulb tells me the following: You are inserting more than one row at once. If so, this statement will break, because a comparison like this is only valid against a scalar value. If there is more than one row in inserted, this will not work:
IF (SELECT [Lagerort] from Inserted) = 'SHUT1'.
Your trigger can be simplified, but I doubt, that you will like the result. Check this with special characters (like üöä) and check for enclosing "-characters...
CREATE TRIGGER tr_LagerShut ON dbo.Lagerverwaltung
AFTER INSERT
AS
BEGIN
IF EXISTS(SELECT 1 FROM inserted WHERE [Lagerort]='SHUT1')
BEGIN
DECLARE #FileName VARCHAR(255) =REPLACE(REPLACE(REPLACE(SYSUTCDATETIME(),' ','-'),':','-'),'.','-');
DECLARE #Content XML=
(
SELECT [Artikelnummer],[Menge] FROM inserted WHERE [Lagerort]='SHUT1' FOR XML AUTO,ELEMENTS
);
DECLARE #Cmd VARCHAR(4000) = 'Echo "' + CAST(#Content AS VARCHAR(MAX)) + '" >>"c:\temp\' + #FileName + '.xml"' ;
PRINT #cmd;
EXEC xp_cmdshell #Cmd ;
END
END;
This might better be done with BCP.
UPDATE Your comment...
First you should check, if this works at all:
DECLARE #FileName VARCHAR(255) =REPLACE(REPLACE(REPLACE(SYSUTCDATETIME(),' ','-'),':','-'),'.','-');
DECLARE #Content XML=
(
SELECT TOP 5 * FROM sys.objects FOR XML AUTO,ELEMENTS
);
DECLARE #Cmd VARCHAR(4000) = 'Echo "' + CAST(#Content AS VARCHAR(MAX)) + '" >>"c:\temp\' + #FileName + '.xml"' ;
PRINT #cmd;
EXEC xp_cmdshell #Cmd ;
If you find no file in c:\temp\: Are you aware, that SQL-Server will always write in its own context? Might be, that you are awaiting a file in your local c-drive, but the file is written to the Server's machine acutally.
If this works isolatedly, it should work within a trigger too. You might pack the call into BEGIN TRY ... END TRY and add an appropriate CATCH block.
So okay, the "simple" trigger could be really a problem. Now I have this idea:
(one more info: time stamp is not inserted into table "Lagerverwaltung" when a new row is inserted)
Pseudo Code on:
Trigger on Table "Lagerverwaltung"
Check if the storage place(s) is "SHUT1"
If "yes"
DECLARE #FileName VARCHAR(255) =REPLACE(REPLACE(REPLACE(SYSUTCDATETIME(),' ','-'),':','-'),'.','-');
create a new table with name dbo + '.' + #filename
Insert all the data (inserted) + row(SYSUTCDATETIME()) AS TimeStamp where 'Lagerort' = 'SHUT1' into new table named 'dbo.' + #filename
DECLARE #Cmd varchar(4000) = 'bcp "select [Artikelnummer],[Menge],[TimeStamp] FROM [wwsbautest].[dbo].#filename WHERE [Lagerort]=''SHUT1'' AND [Menge] > ''0'' FOR XML AUTO, ELEMENTS" queryout "C:\temp\' + #FileName + '.xml" -c -T';
EXEC xp_cmdshell #Cmd;
Drop table dbo.#filename;
Could somthing like that work?
How can I delete files which already exists in a database table as Filename.
Example On Drive C:\Data there are 100 Word documents and 70 of these documents will be found in the database DMS.Filename.
If directory.filename=table.filename then the File should be deleted. In this case we have to delete 70 Word documents. The procedure should run as daily task an check new files against the database.
How can I check and delete the files ?
Here new code:
you can't delete in cmd Files with space or blanks in filename. I think this the msg what i get.
Could Not Find C:\Data\Integration
Could Not Find C:\Windows\system32\Lettre
DECLARE #image_files TABLE (file_path VARCHAR(MAX))
DECLARE #file_path VARCHAR(MAX), #cmd VARCHAR(MAX)
INSERT INTO #image_files (file_path)
EXEC xp_cmdshell 'C:\Data\*.doc /b /s /x'
DECLARE file_cursor CURSOR FOR
SELECT file_path FROM #image_files
WHERE file_path IN
(
select 'C:\Data\' + QC_DESCRIPTION +'.doc' from tbl_France where QC_DESCRIPTION is not null
)
OPEN file_cursor
FETCH NEXT FROM file_cursor INTO #file_path
WHILE (##FETCH_STATUS = 0)
BEGIN
SET #cmd = 'EXEC xp_cmdshell ''del ' + #file_path + ''''
EXEC(#cmd)
FETCH NEXT FROM file_cursor INTO #file_path
END
CLOSE file_cursor
DEALLOCATE file_cursor
Haven't tested this code, but it should gives you a start: xp_cmdshell allows you to execute shell command from SQL Server:
You first need to enable it (credit pero):
-- To allow advanced options to be changed.
EXEC sp_configure 'show advanced options', 1
GO
-- To update the currently configured value for advanced options.
RECONFIGURE
GO
-- To enable the feature.
EXEC sp_configure 'xp_cmdshell', 1
GO
-- To update the currently configured value for this feature.
RECONFIGURE
GO
Then use a cursor to go through your table and delete files:
DECLARE #FileName varchar(200)
DECLARE #Command varchar(300)
DECLARE FileName_Cursor CURSOR FOR
SELECT [FileName] FROM MyTable
OPEN FileName_Cursor
FETCH NEXT FROM FileName_Cursor INTO #FileName
WHILE ##FETCH_STATUS <> 0
BEGIN
SET #Command = 'del "' + #FileName + '"'
EXEC xp_cmdshell #Command
FETCH NEXT FROM FileName_Cursor INTO #FileName
END
CLOSE FileName_Cursor
DEALLOCATE FileName_Cursor
Note that there are security risks with this approach. It does not handle escape characters, or double quotes in the FileName. You will face problems like shellshock. it's best to use SSIS to read your table and delete files, or do it through application code.