I am writing a batch file that downloads files from an FTP server.
Now, I want these files to be checked against the existing files, so that they are not being downloaded again. There are multiple files being downloaded and I do not know the name of the files that may be present in the FTP server.
Thanks
Just use any Window command-line FTP client that supports synchronization.
For example with WinSCP FTP client, you can use a batch file like:
#echo off
winscp.com /ini=nul /log=synchronize.log /command ^
"open ftp://username:password#ftp.example.com/" ^
"synchronize local C:\local\path /remote/path" ^
"exit"
References:
Guide to automating synchronization from FTP server;
Scripting command synchronize.
You can have WinSCP GUI generate a batch file template for you.
(I'm the author of WinSCP)
Related
I need to write a WinSCP script or batch file that will download the contents of a remote SFTP folder, delete contents and then upload to a mapped network folder. I know how to write the login part of the script. But I am confused as to what the specific command line should look like. From what I've read the command script to download and delete the file is something like this:
get -delete -transfer=binary *
The part that I'm not getting is the upload to the network folder command. I understand put is the WinSCP upload command operator. But how do I use it in conjunction with the network folder location files should be uploaded to? Also, can this be scripted out in a batch file or do the batch file and script need to be written separately? Thanks!
An "upload" to a mapped network folder is not a job for WinSCP. A mapped network folder behaves as a local drive. A plain Windows copy command will do. So first do your WinSCP download and then copy (it's not really called an upload) the files to the network drive.
winscp /ini=nul /command ^
"open sftp://user:password#example.com/" ^
"get -delete /sftp/path/* c:\local\drive\path\" ^
"exit"
copy c:\local\drive\path\* n:\mapped\network\drive\path\
Though, if you do not need the files locally, you can have WinSCP download the files from the SFTP folder directly to the mapped network drive:
winscp /ini=nul /command ^
"open sftp://user:password#example.com/" ^
"get -delete /sftp/path/* n:\mapped\network\drive\path\" ^
"exit"
I have an ETL (SSIS) job which creates a file in a folder named as TEST_20170505.csv before it sends the file to an SFTP server.
There would be multiple file in the folder such as, TEST_20170504.csv, TEST_20170503.csv. Currently I am using following sftp script file (File.txt) in a batch file.
lcd E:\localpath\
cd \sftpserverpath\
ascii
put *.csv
bye
This is my .bat file.
sftp -oIdentityFile=E:\sftp\filepath\ssh.ppk -B E:\sftp\filepath\File.txt username#ipaddress
But this will upload all the files in the local path to SFTP server instead of taking the latest/last modified/current date file.
OpenSSH sftp cannot do this on its own. But you can use some fancy batch file construct to select the latest file and then generate an ad-hoc sftp upload script.
Some references:
How do I write a Windows batch script to copy the newest file from a directory?
Windows batch file - Upload only latest file to FTP
Or use some more advanced Windows command-line SFTP client.
For example with WinSCP scripting, it's as easy as using -latest switch in the put command:
open sftp://username#example.com/ -privatekey=ssh.ppk
lcd E:\localpath\
cd \sftpserverpath\
ascii
put -transfer=ascii -latest *.csv
exit
Run the script (upload.txt) like:
winscp.com /script=upload.txt /ini=nul /log=upload.log
You can even have WinSCP generate the script/batch file for you (you just need to add the -latest switch manually).
References:
Uploading the most recent file with WinSCP
Converting OpenSSH SFTP script to WinSCP script
Note that WinSCP uses .ppk format of private key files. While OpenSSH sftp uses PEM format. Even though your key file is named .ppk, it cannot be real .ppk file, otherwise the sftp would reject it. You have probably converted original .ppk file to PEM, but incorrectly kept an original extension. You have to use the original .ppk file with WinSCP (if you do not have it, you can convert the PEM back to .ppk using PuTTYgen).
Also note that WinSCP actually supports the text/ascii mode (-transfer=ascii), while sftp does not (there's no ascii command).
(I'm the author of WinSCP)
The solution is instead of "put *.csv" in your script file, you specify the actual filename you want to upload
I am using the following code to transfer files from my FTP server to my local machine which works fine.
"C:\Program Files (x86)\WinSCP\WinSCP.com" /command ^
"open ftp://rnandipati:J13#Files8.cyberlynk.net/kgptel/" ^
"lcd ""C:\\rnandipati\KGP\File History""" ^
"get *.xls>1D" ^
"rm *.xls<1D" ^
"exit"
Now, I access my server using this path
\\fs01\\Reporting\KGP\File History
When I put this path in place of my local directory path, it shows an error that the system could not find the file specified and error changing directory.
Thanks.
A UNC path cannot be a working directory in Windows.
But you can use it as a target path in the get command:
get *.xls>1D "\\fs01\Reporting\KGP\File History\"
A full command for a batch file will be:
"C:\Program Files (x86)\WinSCP\WinSCP.com" /command ^
"open ftp://rnandipati:J13#Files8.cyberlynk.net/kgptel/" ^
"get *.xls>1D ""\\fs01\Reporting\KGP\File History\""" ^
"rm *.xls<1D" ^
"exit"
(not that I understand a logic of the get *.xls>1D and rm *.xls<1D)
For a similar question, see Get file from FTP server and copy it to UNC directory.
If you need to authenticate to the file server, see:
Include credentials of shared folder in WinSCP script file
How to give credentials in a batch script that copies files to a network location?
I'm trying to download the latest folder from an FTP server. This folder contains several folders in which contains several CSV files.
The issue I have is that the folders are created each day and each time I run the script I only want it to download the latest folder at that location.
I have not the foggiest idea of how to specify this, or even download an entire folder structure from an FTP using batch file.
Please let me know if any additional information is required and I will provide it immediately, thanks again for your help.
Kind Regards,
Lewis
There's no easy way to select the most recent folder with the built-in Windows FTP client, the ftp.exe. You would have more luck with a PowerShell script and the FtpWebRequest.
But even if you manage to select the most recent directory, neither the ftp.exe nor the FtpWebRequest support recursive downloads anyway.
You better use some more powerful 3rd party FTP client.
For example with WinSCP FTP client you can download the latest file or folder, using the -latest switch of the get command (WinSCP 5.9 and newer):
winscp.com /command ^
"open ftp://username:password#ftp.example.com/" ^
"cd /remote/path" ^
"lcd c:\local\path" ^
"get -latest *" ^
"exit"
See also the guide to downloading the most recent file with WinSCP.
(I'm the author of WinSCP)
I'm looking for a FTP client that I can use to upload new files from a local development machine to a remote web-server. I only want to upload the newly edited files though.
Is there a command line utility that can do this, that I can add into an automated process? Is there a GUI client available that can do this? Would be nice to have it cross-platform too. Any ideas?
The Mercurial FTP Extension should do this for you, although I haven't tried it myself.
There is a 'backup' program called SyncBack that does this.
You can find out more about it here: http://www.2brightsparks.com
You have two options:
Schedule a frequent synchronization of a local folder against a remote folder (or moving all files from local folder to a remote folder, if that's more appropriate)
Use a tool that can watch for changes in a local directory and reflect them on a remote directory
You can implement both these options with WinSCP FTP client.
Scheduling
To synchronize changes in a local directory to a remote directory, use the WinSCP synchronize script command from a batch file like:
winscp.com /ini=nul /log=c:\writable\path\to\synchronize.log /command ^
"open ftp://username:password#ftp.example.com/" ^
"synchronize remote C:\local\path /remote/path" ^
"exit"
And schedule the batch file to be run frequently using Windows scheduler.
If you do not want to keep a local copy of the images, just move them to the FTP server, instead of synchronizing them. For that, replace the
"synchronize remote C:\local\path /remote/path" ^
with the put -delete command like:
"put -delete C:\local\path\* /remote/path/" ^
For details see also a guide to automating file transfers (or synchronization) to FTP server.
Watching for changes
Use "Keep remote directory up to date" function of WinSCP.
It can be used both in command-line/console mode using the keepuptodate command, like:
winscp.com /ini=nul /log=c:\writable\path\to\synchronize.log /command ^
"open ftp://username:password#ftp.example.com/" ^
"keepuptodate C:\local\path /remote/path" ^
"exit"
Or in a graphical/GUI mode. You can launch the graphical mode in WinSCP GUI (after logging in) or from a command-line using the /keepuptodate switch like:
winscp.exe ftp://username:password#ftp.example.com/ /keepuptodate C:\local\path /remote/path
(I'm the author of WinSCP)