I am currently creating a small process that will download any new or updated file from an FTP(not SFTP) location to a local drive. I have written a small batch script along with WinSCP, but it takes FAR too long. There are around 200k files and 16k folders on the FTP. It takes just under an hour to check every file and folder, not including any time downloading any new/updated file. From what I understand, WinSCP compared file modification dates of every single file and folder, which is why it takes so long. I'm not sure where to go from here.
I am using WINDOWS 10.
Is there a better way to go about synchronizing these? Possibly check FTP drive if anything has been changed, if so, find the change and download it? Will post pics of batch if needed.
Thank you!
Related
I am trying to develop a batch script which copies a list of files from the a source directory to another directory and does some functions. Once the copy has been done, the files needs to be deleted from the source directory.
In the source directory, we are receiving files on real time basis 24x7 from an external source.
How can I make sure, the copy or the delete from the script does not impact any files which are currently being sent (incomplete files which are in relay at that moment) from the external source in the source directory, whilst the script is doing its own job?
The script needs to be ran every 5 minutes.
Due to some challenges, we cannot xxcopy for our functionality.
Please can you advise.
Instead of COPY& DELETE use MOVE, so the batch fragment would look like this:
FOR %%F IN (C:\SOURCE\*.*) DO MOVE %%F C:\TARGET\
If file in incomplete (other process uses it), then MOVE on that file will fail. Next time the script is run, it will attempt to move that file again.
I'm a graphic designer and I use my external hd frequently to move psd/jpeg/png/tiff files from my laptop at home to my Work PC and vice versa. But as a precaution before modifying or adding a file I copy it to a folder. It's possible to make it automatically? And it's easy to make on Batch?
Thanks for help!
Robocopy is your friend; google it and have a look. It's a very powerful program by sysinternals (owned by Microsoft). Alternatively, if you want a nice GUI to go with it use SyncToy (https://www.microsoft.com/en-gb/download/details.aspx?id=15155) which can be run with a batch file (including arguments) or made into a scheduled task.
Is there a way I can queue WinRAR packing, so it does one job at the time, not all at once slowing down my CPU?
I have a lot of big files that must be splitted up into a SFX and they should not be bigger that 400 MB. I can do the job doing it file by file, but that takes a lot of time, so if I just could run a batch in the evening my PC could do all the job while I sleep.
I don't need a batch. It can be done with GUI of WinRAR.
I just select my files.
Right click and choose Add files to archive.
Go to tab Files and check Put each file to separate archive.
Thats it. WinRAR packs now one file after the other and creates for each file a separate archive as documented on help page opened on pressing button Help on tab Files.
I'm using psftp.exe to download files from the server. Is there an easy way to delete these files once I have downloaded them but leave the new ones that might have appeared on the server when I was downloading to be downloaded next time?
Here's my command line:
psftp.exe domain.com -i keys\private.ppk
get *.xml
Edit: I want to download the files from a Linux box to a Windows PC.
There's no easy way to do this with psftp. You would have to parse its output to find files that were successfully downloaded.
Though you can do this easily with WinSCP. Just use get -delete *.xml command.
Full WinSCP script would be:
open sftp://domain.com/ -privatekey=keys\private.ppk -hostkey=...
get -delete *.xml
exit
See an introduction to WinSCP scripting.
See also a guide for converting PSFTP script to WinSCP.
You can also have WinSCP GUI generate script like this for you.
(I'm the author of WinSCP)
Martin's answer is good. The below is more industrial.
Moving them to a staging area before the download may be prudent.
Generally you would move/rename the files on the server as a starting point. They are going to be deleted anyway so nothing should miss them? nor would you want to fall back over a recent file.
(so restart after this point in the event of a subsequent failure)
Then perform the download.
Then perform the delete.
I would approach the issue differently. Instead of deleting the file from the server, add the downloaded filename to a local table of "Already downloaded files". Then when you scan the FTP again for new files, ignore any that are in that table.
That way the next time you run your download script you only get the new files, but the old files remain on the server.
You could have another script that runs periodically and deletes all files over a certain age.
WINSCP is alright, and Martin (the author) drops in to practically every PuTTy thread to recommend it, but it's a fully GUI-based app and not for me. If you really need everything to be done on the commandline then WINSCP is often not an option.
I need to write a batch job file to keep two folder in sync.
Inputs : Source and Destination folder names.
senario :
If the destination folder/subfolder is not found – create and proceed
If the files are already present in both the folders Compare and overwrite destination file if timestamp/size differs.
If the file not present in source and present in destination, delete the destination file.
kindly help me to write the batch job.
Thanks Vikram
A batch file written by a self-professed inexperienced user - even run periodically - is not a good replacement for a sync tool like rsync or dropbox or even the Briefcase built into Windows. Robocopy looks a good bet.