I can't get my script to save my file to a specified folder that I have permissions to write to. Here is the script.
DECLARE #sql VARCHAR (8000)
SET #SQL = 'bcp SELECT * FROM LMP_DAILY.dbo.vw_DirDailyActual ORDER by Date, DirCOde queryout
T:\Shared information\SD3\dirdailyactual.csv -T -c';
PRINT #SQL
exec master.xp_cmdshel #sql;
I have sqladmin privileges on the DB and the server. I also, have read/write permissions on the path where I'm try to send the file.
The job runs and completes but not file is being created on my specified path.
Related
I have an SSIS package which logs into my client's SFTP site and uploads a file using commands sent to WinSCP.
This package fails quite often and is difficult to work with. Visual Studio crashes so frequently that my team refuses to use it any more. We are migrating all automated tasks to Stored Procedures and SQL Agent Jobs, as they work a lot better than SSIS does.
However, this leaves us with no good solution for automating FTP uploads of report files produced on SQL Server.
I have been doing it using PSFTP and batch files called from VB scripts. The script checks a particular folder to see if a file is there and uploads it if it is. This works, but is inelegant and I have to schedule these scripts to look for a file in the folder every 15 minutes, rather than just call an "upload" command from my stored procedure on SQL Server. There's potential for things to go horribly wrong if someone renames a file in that folder, or puts two files in there, etc.
I can easily create a .CSV file as part of a stored procedure into a particular folder using BCP, e.g.:
--Create Client Update .CSV File if there are any rows in the CLIENT_UPDATE table
IF EXISTS(SELECT 1 FROM CLIENT_UPDATE)
BEGIN
DECLARE #ColumnHeader VARCHAR(8000)
SET #FileName = #DataPath + '\Output File.csv'
SELECT #ColumnHeader = COALESCE(#ColumnHeader+',' ,'')+''''+column_name+'''' FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME='CLIENT_UPDATE'
SET #BCPCommand = 'bcp "SELECT '+ #ColumnHeader +' UNION ALL SELECT * FROM CLIENT_UPDATE" queryout "' + #FileName + '" -c -t , -r \n -S . -T'
EXEC master..xp_cmdshell #bcpCommand
END
I'd like the next step in that stored procedure to be "Upload .CSV file to client SFTP site". How would I accomplish this?
Success!
I created a stored proc on my database named sp_UploadFileToSFTP, which uses PSFTP.EXE to send files.
-- ================================================================================================================================
-- Description: Upload File to SFTP
-- - Creates "SFTPUploadScript.bat" and "SFTPUploadScript.txt" scripting files to control psftp.exe, then executes the batch file
-- - Requires a local #ScriptFolder on the SQL Server to create files inside, and which contains psftp.exe
-- - "SFTPUploadScript.bat" needs to be run once manually from the SQL Server desktop to capture the keypair of the SFTP site
-- After this, the script will work automatically
-- ================================================================================================================================
CREATE PROCEDURE sp_UploadFileToSFTP
#FilePathToUpload varchar(1000),
#SFTPSiteAddress varchar(1000),
#SFTPLogin varchar(1000),
#SFTPPassword varchar(1000),
#SFTPRemoteFolder varchar(1000)
AS
BEGIN
SET NOCOUNT ON;
--Declare Variables
DECLARE #ScriptFolder varchar(1000)
SET #ScriptFolder = 'C:\SFTPBatchFiles\'
DECLARE #CommandString AS varchar(8000)
--Delete Existing files if they exist
SET #CommandString = 'del "'+ #ScriptFolder + 'SFTPUploadScript.txt"'
EXEC master..xp_cmdshell #CommandString
SET #CommandString = 'del "'+ #ScriptFolder + 'SFTPUploadScript.bat"'
EXEC master..xp_cmdshell #CommandString
--Create Batch fle with login credentials
SET #CommandString = 'echo "'+ #ScriptFolder +'psftp.exe" ' + #SFTPSiteAddress + ' -l ' + #SFTPLogin + ' -pw ' + #SFTPPassword + ' -b "' + #ScriptFolder + 'SFTPUploadScript.txt" > "' + #ScriptFolder + 'SFTPUploadScript.bat"'
EXEC master..xp_cmdshell #CommandString
--Create SFTP upload script file
SET #CommandString = 'echo cd "' + #SFTPRemoteFolder + '" > "' + #ScriptFolder + 'SFTPUploadScript.txt"'
EXEC master..xp_cmdshell #CommandString
SET #CommandString = 'echo put "' + #FilePathToUpload + '" >> "' + #ScriptFolder + 'SFTPUploadScript.txt"'
EXEC master..xp_cmdshell #CommandString
--Run the Batch File
SET #CommandString = #ScriptFolder + 'SFTPUploadScript.bat'
EXEC master..xp_cmdshell #CommandString
END
GO
I can then call it from inside another stored proc like so:
sp_UploadFileToSFTP 'C:\FileToUpload\Test.txt','sftp.mysite.com','Login#domain.com','Password1234','UploadFolder/In here/'
All works perfectly, takes about a quarter of a second to run, a very neat and stable solution.
Compared to what I was doing before which was using SQL Server to drop a file into a folder, then checking that folder for new files every 15 minutes with my Visual Basic SFTP upload script, it's miles better :)
If you want to try this yourself, all you need is a folder on the C:\ drive of your SQL Server named "SFTPBatchFiles", and in there you put psftp.exe, which is part of PuTTY.
You will also need a trusting server admin who will allow you to run psftp commands and your own batch files using xp_cmdshell, and they'll probably need to open a route in your firewall so you can connect to your remote site directly from the SQL Server.
Finally, you will also need to be able to remote desktop into your SQL Server, since you'll need to run the batch file it creates manually once and press "Y" when psftp asks you to accept the new keypair. After that, it's all plain sailing.
No SSIS, no WINSCP, no Visual Studio, no hours upon hours of crashes and debugging!
The only issue right now is that if the FTP transfer fails for whatever reason, the stored procedure still reports success, I'm not bringing the psftp error messages back into SQL Server.
My next task will be to trap the status and error messages as they get generated from xp_cmdshell and give the user some detailed feedback and generate errors if the process fails.
UPDATE:
Trapping errors and feedback from the remote FTP site
--Run the Batch File
DROP TABLE IF EXISTS #CommandOutput
CREATE TABLE #CommandOutput (ReadLine varchar(1000))
SET #CommandString = #ScriptFolder + 'SFTPUploadScript.bat'
INSERT #CommandOutput
EXEC master..xp_cmdshell #CommandString
--Check for Invalid Password error
IF EXISTS (SELECT ReadLine
FROM #CommandOutput
WHERE ReadLine LIKE '%Invalid Password%')
BEGIN
PRINT 'Invalid Password Error!'
END
All of the output from the batch file (what would normally be shown on the screen if you were to run it manually) is now inserted line-by-line into a temporary table named #CommandOutput as the batch file runs.
After the batch file runs, you can then query #CommandOutput to see what's happened with your transfer.
I've added a simple check for the words "Invalid Password" anywhere in the output, as this is a common error you might have. You could add as many of these checks as you liked, or you could import the entire table into an email which gets sent to you every time the job runs, log the table to an FTP event logging table, or any number of other things.
I am getting an error Access is denied when I try deleting the csv file from a folder using xp_cmdshell. however I can delete .csv.gpg file successfully from the same location using xp_cmdshell.
My query goes as:
--delete the csv file from local folder
SELECT #Delete2 = 'del ' + 'C:\Akshay\files\testfile.csv'
EXEC master..Xp_cmdshell #Delete2
I would say you need to user declare and set assuming you know what the file name is. I just tried the statement you have and it erred because of the select.
Here is my solution with the information I have:
DECLARE #delete VARCHAR(50)
SELECT #delete = 'del B:\test.txt'
EXEC xp_cmdshell #delete
As I am a beginner in SQL Server and my scripting is not very polished yet. I need suggestions on the below issue.
I receive files from a remote server to my machine (around 700/day) as follows :
ABCD.100.1601310200
ABCD.101.1601310210
ABCD.102.1601310215
Naming Convention:
Here the first part 'ABCD' remains the same, middle part is a sequence id which is in incremental order for every file. The last part is time stamp.
File structure
The file does not have any specific extension but can be opened with notepad/excel. Therefore can be called as flat file. Each files consist of 95 columns and 20000 rows fixed with some garbage value on top 4 and bottom 4 rows of column 1.
Now, I need to make a database in SQL server where I can import data from these flat files using a scheduler. Suggestion needed.
There are probably other ways of doing this, but this is one way:
Create a format file for your tables. You only need to create it once. Use this file in the import script in step 2.
Create an import script based on OPENROWSET(BULK '<file_name>', FORMATFILE='<format_file>'
Schedule the script from step 2 in SQL Server to run against the database you want the data imported in
Create the format file
This creates a format file to be used in the next step. The following script creates a format file in C:\Temp\imp.fmt based on an existing table (replace TEST_TT with the database you are importing to). This creates such a format file with a , as field seperator. If the files have tab as seperator, remove the -t, switch.
DECLARE #cmd VARCHAR(8000);
SET #cmd='BCP TEST_TT.dbo.[ABCD.100.1601310200] format nul -f "C:\Temp\imp.fmt" -c -t, -T -S ' + (SELECT ##SERVERNAME);
EXEC master..xp_cmdshell #cmd;
Before executing this you will to reconfigure SQL Server to allow the xp_cmdshell stored procedure. You only need to do this once.
EXEC sp_configure 'show advanced options', 1
GO
RECONFIGURE
GO
EXEC sp_configure 'xp_cmdshell', 1
GO
RECONFIGURE
GO
import script
This script assumes:
The files need to be imported to separate tables
The files are located in C:\Temp
The format file is C:\Temp\imp.fmt (generated in the previous step)
SET NOCOUNT ON;
DECLARE #store_path VARCHAR(256)='C:\Temp';
DECLARE #files TABLE(fn NVARCHAR(256));
DECLARE #list_cmd VARCHAR(256)='DIR ' + #store_path + '\ABCD.* /B';
INSERT INTO #files EXEC master..xp_cmdshell #list_cmd;
DECLARE #fullcmd NVARCHAR(MAX);
SET #fullcmd=(
SELECT
'IF OBJECT_ID('''+QUOTENAME(fn)+''',''U'') IS NOT NULL DROP TABLE '+QUOTENAME(fn)+';'+
'SELECT * INTO '+QUOTENAME(fn)+' '+
'FROM OPENROWSET(BULK '''+#store_path+'\'+fn+''',FORMATFILE=''C:\Temp\imp.fmt'') AS tt;'
FROM
#files
WHERE
fn IS NOT NULL
FOR XML PATH('')
);
EXEC sp_executesql #fullcmd;
i am facing an issue while executing the BCP command in the remote server.
i have the following command in a batch file which executes a script file.
The script file process a temporary table and writes the records in temporary table to a file.
SQLCMD -SQA_Server236 -dtestdb -Usa -PPassword1 -i"D:\script\Writing to Files\Write to CSV.sql" -o"D:\script\script_logs.log"
--script files contains...
declare #table NVARCHAR(255)
declare #filename VARCHAR(100)
set #filename='C:\TextFile\abcd.csv'
set #table ='##Indexes_Add'
IF OBJECT_ID('tempdb..#Indexes_Add') IS NOT NULL
BEGIN
DROP TABLE ##Indexes_Add
END
CREATE TABLE ##Indexes_Add
(
id int IDENTITY(1,1) PRIMARY KEY,
alter_command NVARCHAR(MAX),
successfully_readded BIT NULL,
)
insert into ##Indexes_Add select 'a',0
insert into ##Indexes_Add select 'a',0
SET NOCOUNT ON;
IF OBJECT_ID('tempdb..'+#table) IS NOT NULL
BEGIN
DECLARE
#sql NVARCHAR(MAX),
#cols NVARCHAR(MAX) = N'';
SELECT #cols += ',' + name
FROM tempdb.sys.columns
WHERE [object_id] = OBJECT_ID('tempdb..'+#table)
ORDER BY column_id;
SELECT #cols = STUFF(#cols, 1, 1, '');
SET #sql = N'EXEC master..xp_cmdshell ''bcp "SELECT '''''
+ REPLACE(#cols, ',', ''''',''''') + ''''' UNION ALL SELECT '
+ 'RTRIM(' + REPLACE(#cols, ',', '),RTRIM(') + ') FROM '
+ 'tempdb.dbo.'+#table + '" queryout "' + #filename + '" -c -t, -SQA_Server236 -Usa -PPassword1''';
EXEC sp_executesql #sql;
print #sql
END
My problem-:
When i run the above batch command in a batch file and i give my local server other than server name "QA_Server236" , i am getting the file "abcd.csv" created in my system but when i give the server name as "QA_Server236" the file is created in the remote machine i.e QA_Server236. But i want the file to be created in my system if the given server is a remote server say "QA_Server236"
Can anyone help me in this issue. i am not getting any method to do so.
If I'm right BCP does not allow saving result sets and/or logs to remote machines. But you could try to mount a folder on remote PC to a shared folder on your local machine and set the output location there, or maybe try using network path "\[YOURPC]....".As I'm not sure it works (actually, I think it won't), here's the only solution I can think of: add a line to your batch which moves the file(s) from the remote machine to your PC (xcopy or similar) after BCP finished executing (in batch, do this as "Call bcp.exe [params]" instead of just "bcp.exe ...")Hope this helps!
I'm trying to save a XML file from SQL. I'm doing something like:
DECLARE #FileName VARCHAR(100)
DECLARE #db VARCHAR(100)
DECLARE #SQLCmd VARCHAR(1000)
SELECT #FileName = 'D:\ProductsList.xml'
SELECT #db='MyBase.dbo.'
SELECT #SQLCmd = 'bcp "SELECT '+#db+'ArticleName FROM '+#db+' ARTICLES' +
' FOR XML PATH(''Product''),ELEMENTS, ROOT(''Products''), TYPE "' +
' queryout '+#FileName +' -w -T -S' + ##SERVERNAME
EXECUTE master..xp_cmdshell #SQLCmd
It works okay for me one one sql server but fails one another. I get message "Error = [Microsoft][SQL Server Native Client 11.0]Unable to open BCP host data-file".
Bcp,queryout, cmdshell are unfortunately all new for me.
I'm reading a lot of websites and all they have some interesting information but I really don't know how to begin.
They say - add permissions for the user running the script. But who is this user if not me? ;) Is there some "automatic" Windows user created for running SQL Server or what? Where can I see this user and configure his rights so it can save my file?
Or maybe there are some other issues with bcp and cmdshell that I should to know?
Big thanks for any help. I'm really stuck :(