I am trying to automate a call to a remote server using WinSCP which is a redundant everyday task. I want the batch file to call remote machine - provide username and password, and later check for if files exist.
So far I am able to start a WinSCP server but still the batch file does not consume username and password - it is still asking for them and/or providing an error regarding too many arguments.
chdir /d D:\Program Files\WinSCP\
winscp.com 172.18.186.39 username password
Your syntax does not even remotely resemble WinSCP command-line syntax for scripting.
See WinSCP article on Checking file existence using scripting:
#echo off
set REMOTE_PATH=/home/user/test.txt
winscp.com /command ^
"open ftp://username:password#ftp.example.com/" ^
"stat %REMOTE_PATH%" ^
"exit"
if %ERRORLEVEL% neq 0 goto error
echo File %REMOTE_PATH% exists
rem Do something
exit /b 0
:error
echo Error or file %REMOTE_PATH% not exists
exit /b 1
The above example is for FTP protocol. You didn't tell use what protocol are you using. If not FTP, you have to change the open command accordingly.
You can have WinSCP GUI generate the open command for you.
Once you have the batch file ready, use Windows Scheduler to call it regularly.
Related
I have an issue where the files I am uploading to a vendor are being consumed before they are finished uploading. To get around it the vendor has provided me with a location underneath where I can upload the files first then requested I move them once the upload is done.
Stripping out sensitive information, I am referencing this file in my sftp command in batch myfile.scr
cd /from_client/payments/upload
lcd E:/uploadfiles
mput uploadfiles.xml.*
quit
That would upload the files to the remote directory upload. I know need to get them into the level above. The theory is this will then process as normal and the files will already exist so the application won't consume them before they are done uploading.
I would assume I need to do a job to build a list of the files and then a 2nd to loop through them to do a rename on the remote folder to get them to go up a level.
Any ideas on how this could work? The vendor is unable to do anything here.
already tried various solutions to see if I could use mv but thats not an option as its not using Winscp. We can only use batch scripts and then native sftp commands.
You can generate put and rename commands for each file using a batch file like this:
set SOURCE=E:\uploadfiles
set SCRIPT=myfile.scr
(
echo cd /from_client/payments/upload
echo lcd %SOURCE%
for %%F in (%SOURCE%\*) do (
echo put %%~nF
echo rename %%~nF /final/path/%%~nF
)
echo quit
) > %SCRIPT%
sftp user#host -b %SCRIPT%
It should generate a script like this:
cd /from_client/payments/upload
lcd E:\uploadfiles
put one.txt
rename one.txt /final/path/one.txt
put two.txt
rename two.txt /final/path/two.txt
put three.txt
rename three.txt /final/path/three.txt
quit
Though with a more capable client, it would be easier. For example my WinSCP supports wildcards with mv command, so you can simply do:
WinSCP.com /ini=nul /log=winscp.log /command ^
"open sftp://user:password#host/ -hostkey=""...""" ^
"put E:\uploadfiles\* /from_client/payments/upload/" ^
"mv /from_client/payments/upload/* /final/path/" ^
"exit"
I'm aware that you wrote that you cannot use other clients, because:
"It's not using WinSCP" – What "it"? It's you! Use the best tool for your task.
"We can only use ... native sftp commands" – There are no "native sftp commands". There are only commands of the client you are using. Currently you are using OpenSSH sftp client. But that's not set in stone. It's easy to switch (you do not even to read it, everything is already in my code above).
I currently have batch scripts on different servers that transfer a csv file to an FTP server at a different location. My script looks similar to this:
echo user ftp_user> ftpcmd.dat
echo password>> ftpcmd.dat
echo put c:\directory\%1-export-%date%.csv>> ftpcmd.dat
echo quit>> ftpcmd.dat
ftp -n -s:ftpcmd.dat ftp.MyFTPSite.com
del ftpcmd.dat
If I wanted to require a secure transmission, is how would my script be updated?
Thanks.
First, make sure you understand, if you need to use Secure FTP (=FTPS, as per your text) or SFTP (as per tag you have used).
Neither is supported by Windows command-line ftp.exe. As you have suggested, you can use WinSCP. It supports both FTPS and SFTP.
Using WinSCP, your batch file would look like (for SFTP):
echo open sftp://ftp_user:password#ftp.MyFTPSite.com -hostkey="..." >> ftpcmd.dat
echo put c:\directory\%1-export-%date%.csv >> ftpcmd.dat
echo exit >> ftpcmd.dat
winscp.com /script=ftpcmd.dat
del ftpcmd.dat
And the batch file:
winscp.com /log=ftpcmd.log /script=ftpcmd.dat /parameter %1 %date%
Though using all capabilities of WinSCP (particularly providing commands directly on command-line and the %TIMESTAMP% syntax), the batch file simplifies to:
winscp.com /log=ftpcmd.log /command ^
"open sftp://ftp_user:password#ftp.MyFTPSite.com -hostkey=""...""" ^
"put c:\directory\%1-export-%%TIMESTAMP#yyyymmdd%%.csv" ^
"exit"
For the purpose of -hostkey switch, see verifying the host key in script.
Easier than assembling the script/batch file manually is to setup and test the connection settings in WinSCP GUI and then have it generate the script or batch file for you:
All you need to tweak is the source file name (use the %TIMESTAMP% syntax as shown previously) and the path to the log file.
For FTPS, replace the sftp:// in the open command with ftpes:// (explicit TLS/SSL) or ftps:// (implicit TLS/SSL). And remove the -hostkey switch.
winscp.com /log=ftpcmd.log /command ^
"open ftps://ftp_user:password#ftp.MyFTPSite.com -explicit" ^
"put c:\directory\%1-export-%%TIMESTAMP#yyyymmdd%%.csv" ^
"exit"
You may need to add the -certificate switch, if your server's certificate is not issued by a trusted authority.
Again, as with the SFTP, easier is to setup and test the connection settings in WinSCP GUI and then have it generate the script or batch file for you.
See a complete conversion guide from ftp.exe to WinSCP.
You should also read the Guide to automating file transfers to FTP server or SFTP server.
Note to using %TIMESTAMP#yyyymmdd% instead of %date%: A format of %date% variable value is locale-specific. So make sure you test the script on the same locale you are actually going to use the script on. For example on my Czech locale the %date% resolves to čt 06. 11. 2014, what might be problematic when used as a part of a file name.
For this reason WinSCP supports (locale-neutral) timestamp formatting natively. For example %TIMESTAMP#yyyymmdd% resolves to 20170515 on any locale.
(I'm the author of WinSCP)
The built in FTP command doesn't have a facility for security. Use cUrl instead. It's scriptable, far more robust and has FTP security.
ftps -a -z -e:on -pfxfile:"S-PID.p12" -pfxpwfile:"S-PID.p12.pwd" -user:<S-PID number> -s:script <RemoteServerName> 2121
S-PID.p12 => certificate file name ;
S-PID.p12.pwd => certificate password file name ;
RemoteServerName => abcd123 ;
2121 => port number ;
ftps => command is part of ftps client software ;
I can't try to reconnect with a different password after authentication failed. ERRORLEVEL seems to not be working in my code. I have no clue what's going wrong. The code is in batch file on Windows 10.
#echo off
echo use USERNAME> ftpcmd.dat
echo PASSWORD>> ftpcmd.dat
ftp -n -s:ftpcmd.dat files.000webhost.com
if %ERRORLEVEL% neq 0 (
goto TEST
)
:TEST
echo user USERNAME> ftpcmd.dat
echo DIFFERENTPASSWORD>> ftpcmd.dat
echo prompt>>ftpcmd.dat
echo cd Test>>ftpcmd.dat
echo mput FILE PATH>> ftpcmd.dat
ftp -n -s:ftpcmd.dat FTPTHING
The Windows ftp.exe always returns 0, no matter what.
All you can possibly do, is to parse its output and look for errors. But that's pretty unreliable.
If you want to take this approach anyway, see
how to capture error conditions in windows ftp scripts? or
Batch command to check if FTP connection is successful.
Though, you better use some better FTP client. Sooner or later you will run into troubles with the ftp.exe anyway, as it does not support an encryption, the passive mode or recursive transfers, among many other limitations.
If you are familiar with PowerShell, use the FtpWebRequest or WebClient from a Powershell script. There are numerous examples here, for example Upload files with FTP using PowerShell.
If not, use some 3rd party command-line FTP client.
For example with WinSCP FTP client, your batch file may look like:
#echo off
winscp.com /ini=nul /log=script.log /command ^
"open ftp://USERNAME:PASSWORD#ftp.example.com/" ^
"put C:\LOCAL\FILE /REMOTE/PATH/" ^
"exit"
if ERRORLEVEL 1 (
winscp.com /ini=nul /log=script2.log /command ^
"open ftp://USERNAME:DIFFERENTPASSWORD#ftp.example.com/" ^
"put C:\LOCAL\FILE /REMOTE/PATH/" ^
"exit"
)
References:
Automate file transfers to FTP server;
How do I know that script completed successfully?;
Guide to converting Windows FTP script to WinSCP SFTP script.
WinSCP can also be used from PowerShell, if you want even more robust implementation.
(I'm the author of WinSCP)
I have a batch script which generates a WinSCP upload script to upload a file to an SFTP location. Now when I run the batch file via command prompt - it runs successfully and loads it. I called the same thru SSIS Execute process task - it runs successfully and loads it. Now when I put the same on SQL Agent - I tried the following two options:
Using Operating System (CmdExec) - cmd.exe /c "\.bat"
Added the SSIS package to SSISDB and added it as a job step.
With both the above options the job showed a successful run. However the file is not uploaded! Any ideas on what is happening?
Here's my batch script:
echo off
SET winscp=C:\"Program Files (x86)"\WinSCP\WinSCP.com
SET stagingDirectory=\\<staging path>\
SET scriptPath=\\<ScriptPath>\UploadScript.txt
SET ftpHost=xx.xx.xx.xx
SET ftpUser=user
SET ftpPass=password
SET fileName=Test.xlsx
SET ftpFlags=
#REM ftpFlags: -explicit
echo deleting uploadScript if it already exists
IF EXIST %scriptPath% del /F %scriptPath%
IF EXIST %scriptPath% exit 1
echo Generating WINSCP Upload Script
>>%scriptPath% echo option batch abort
>>%scriptPath% echo option confirm off
>>%scriptPath% echo open sftp://%ftpUser%:%ftpPass%#%ftpHost% %ftpFlags%
>>%scriptPath% echo option transfer binary
>>%scriptPath% echo put %stagingDirectory%%fileName% /
>>%scriptPath% echo close
>>%scriptPath% echo exit
echo Launching WINSCP upload
start /wait %winscp% /console /script=%scriptPath%
As you start the WinSCP via the start (why?), the exit code is not propagated to the SSIS. So, you never learn, if the script fails. And it most probably fails.
You also should enable logging, so that you can see what's wrong.
You should use this code to propagate the WinSCP exit code to SSIS and to enable logging:
%winscp% /log=\\<ScriptPath>\UploadScript.log /script=%scriptPath%
exit /b %ERRORLEVEL%
(Note that the winscp.com does not have the /console parameter)
Anyway, one clear problem is that you do not specify an expected SSH host key in the script. When you run the script manually, you probably have the key cached in the registry of your Windows account. But under the SSIS a different account is used, and its host key cache is likely empty. You should add the -hostkey switch to the open command in the script to make the script independent on the cache. See Where do I get SSH host key fingerprint to authorize the server?
When testing the script, add the /ini=nul parameter to isolate the script from your configuration.
For this and other hints, when debugging WinSCP running under SSIS, see My script works fine when executed manually, but fails or hangs when run by Windows Scheduler, SSIS or other automation service. What am I doing wrong?
And finally, see WinSCP SFTP Task for SSIS.
Your variable seems set incorrectly. To manage with a space in the path and into the variable you have to put in quotes the whole path or the whole variable.
i.e.
set "winscp=C:\Program Files (x86)\WinSCP\WinSCP.com"
echo start "%winscp%"
:: output: start "C:\Program Files (x86)\WinSCP\WinSCP.com"
or
set winscp="C:\Program Files (x86)\WinSCP\WinSCP.com"
echo start %winscp%
:: output: start "C:\Program Files (x86)\WinSCP\WinSCP.com"
Another point, you have to check this file: UploadScript.txt because your script adds new lines rather than remake the file.
change this line to >%scriptPath% echo option batch abort instead of >>%...
Ah, I did not pay attention to the IF EXIST.
I want to use a SOCKS5 connection with a generic proxy to upload files to FTP.
I'm working on Windows server. My .txt for my .bat actually looks like:
open proxyhost
proxyuser
proxypassword
user ftpuser#ftphost.de
ftppassword
bin
mput \\upload\path\to\*.pdf
quit
I'm opening the .txt file with:
ftp -i -s:"\\path\to\mytxt.txt" >"\\path\to\ftp.log"
find "226" "\\path\to\ftp.log" && goto finished || goto error
:finished
exit 0
:error
exit 1
Now the question: Is there a way to use SOCKS5 in this .txt?
I found nothing on Google.
Maybe you can help me out. Thanks a lot.
Best regards
The Windows built-in ftp.exe FTP client does not support any kind of proxies (the FTP proxy you are using in your script is a transparent proxy, the FTP client does not even know it exists).
You have to use another command-line FTP client.
For example with WinSCP scripting:
winscp.com /log="\\path\to\ftp.log" /command ^
"open ftp://ftpuser:ftppassword#ftphost.de/ -rawsettings ProxyMethod=2 ProxyHost=proxyhost ProxyUsername=proxyuser ProxyPassword=proxypassword" ^
"put \\upload\path\to\*.pdf" ^
"exit"
if errorlevel 1 goto error
exit 0
:error
exit 1
For details see:
the guide for converting Windows FTP script to WinSCP.
documentation for the "raw session settings" for the proxy.
(I'm the author of WinSCP)