My requirement is to take backup of only those files which have been modified or created recently.
At this point of time my code is as below. It is picking up the overall xml files in the destination folder under datetime folder.
#echo off
for /F "tokens=1-4 delims=/ " %%A in ('date/t') do (
set DateDay=%%A
set DateMonth=%%B
set DateYear=%%C
)
for /F "tokens=1-4 delims=/ " %%D in ('time/t') do (
set DateTime=%%D
)
set CurrentDate=%DateDay%-%DateMonth%-%DateYear%-%time:~0,2%.%time:~3,2%
xcopy "D:\Splunk\etc\apps\search\default\data\ui\views\*.xml" "\\cidcswpems2533\d$\backup\%CurrentDate%\"
Help of command xcopy output by entering xcopy /? in a command prompt window lists a parameter which you have to append here on last line and you are done.
/m
Copies source files that have their archive file attributes set. Unlike /a, /m turns off archive file attributes in the files that are specified in the source.
The archive file attribute is designed for backup processes. Every time a file is created new or modified, the archive attribute is automatically set. Therefore copying just files with archive attribute set and remove it after copy helps you here to copy just the files modified or created since last backup process.
Related
Our backup system creates .bak files everyday which we can use to restore files if we ever run into issues. If left alone these would fill up our storage so I found a batch file that I can run to delete the old batch files everyday after new ones are created.
forfiles -p "c:\xxx\yyy" -s -m *.bak /D -2 /C "cmd /c del #path"
This works fine but I want to create a safety net so that if for whatever reason our backup system fails and the new .bak files aren't created the old .bak files will stay there instead of being deleted, otherwise we would be left with no backup files in the event of an incident. So ideally I want something that will check for .bak files younger than one day and if those files are not present it won't run the above line but if those younger files are present it will run the above line and delete the older files. Not sure if this is possible with batch files or not. Thanks in advance for your help on this.
EDIT: Some more info on what I need. Everyday at around 10pm around 50 backup .bak files are created and put into folder c:\xxx\yyy
These files are quite large so I have set up a batch file to run automatically every day that removes all .bak files that are older than 1 day. This is fine for everyday use but the scenario I have in my head is what if the backup system doesnt create the .bak files for whatever reason. I want the batch file to check to make sure the new .bak files have been created before it deletes the old ones. Basically using a batch file is there a way to check if there is a certain file type in a folder that is newer than 1 day old and can we change what the batch file does depending on the outcome.
These are examples of the files created for the 18th and 19th.
2004 Apr_backup_2017_12_18_210001_2986007.bak
2004 Apr_backup_2017_12_19_210001_3168635.bak
Subscribers_backup_2017_12_19_210003_3012893.bak
model_backup_2017_12_19_210003_2544131.bak
They all seem to follow the below format:
[DESC]_backup_[YEAR]_[MONTH]_[DAY]_21000[1/2/3]_[7 DIGIT NO.].bak
I think, an unknown list of [DESC] strings in all the backup file names is most difficult to handle in batch file. The code could be very simple on knowing this list as it can be seen below, or at least on knowing if those strings do not contain characters being critical on batch file processing like !%=.
But the coding challenge for unknown list of [DESCR] strings with special characters in file names was interesting for me and so I developed first following commented batch file:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "BackupFolder=C:\xxx\yyy"
rem Search for files matching the wildcard pattern *_backup_*.bak in backup
rem folder, assign each file name without file extension to environment
rem variable FileName and call the subroutine GetUniqueDescs to get the
rem file description at beginning of each file name into a list in memory.
for /F "delims=" %%I in ('dir "%BackupFolder%\*_backup_*.bak" /A-D /B /ON 2^>nul') do (
set "FileName=%%~nI"
call :GetUniqueDescs
)
rem Run command SET with FileDesc: to output all environment variables
rem starting with that string in name and sorted by name and process
rem this list whereby each line ends with =1 as value 1 is assigned
rem to each of these environment variables.
rem For each unique file description in output list assign the file
rem description with =1 appended to environment variable FileDesc
rem and run subroutine DeleteFiles.
for /F "tokens=2 delims=:" %%I in ('set FileDesc: 2^>nul') do (
set "FileDesc=%%I"
call :DeleteFiles
)
rem Restore initial environment on starting this batch file and exit it.
endlocal
goto :EOF
rem The subroutine GetUniqueDescs first runs a string substitution which
rem gets the backup pattern part from file name, i.e. everything in file
rem name from _backup_ to end of file name.
rem Then another string substitution is used to remove this string from
rem current file name to get just the description and define an environment
rem variable of which name starts with FileDesc: and ends with the file
rem description. The value assigned to this environment variable is 1.
:GetUniqueDescs
set "BackupPart=%FileName:*_backup_=_backup_%"
call set "FileDesc:%%FileName:%BackupPart%=%%=1"
goto :EOF
rem The subroutine DeleteFiles removes first from passed file description
rem the last two characters being always =1 from list of environment
rem variables starting with FileDesc: and appends the backup wildcard
rem pattern.
rem Command DIR is used to find all files in backup folder starting
rem with current file description and _backup_ and output the found
rem files sorted by last modification date with newest modified file
rem first and oldest modified file last.
rem The command FOR processing this list skips the first file name
rem output by DIR which means the newest file. All other, older
rem files perhaps also found by DIR are deleted one after the other.
:DeleteFiles
set "FilePattern=%FileDesc:~0,-2%_backup_*.bak"
for /F "skip=1 delims=" %%J in ('dir "%BackupFolder%\%FilePattern%" /A-D /B /O-D /TW') do ECHO del "%BackupFolder%\%%J"
goto :EOF
The command ECHO in last but one line before command del results in just getting displayed which files would be deleted instead of really deleting them.
The option skip=1 in last but one line determines how many backup files are always kept.
For example using skip=5 results in keeping the newest five files according to last modification date being usually on backup files also the creation date and deleting all others.
The advantage of such a backup deletion strategy is that it does not matter:
how often a specific backup is created – daily, weekly or monthly;
if last backup creation was successful at all;
if some or even all backup files were deleted manually;
how old each individual backup file is;
how often the batch file for backup files deletion is executed.
What really matters on deletion of backups is the storage size needed for each backup and how much free storage space remains after deletion process. The file date of a backup file is not limiting the free storage size. The file sizes of all remaining backup files and the total storage size on backup media are the factors which really matter. That's why I do not understand all those "delete older than" questions. Who has to care about age of a file as long as there is enough free space for new files?
The file creation date could be also used by using /TC instead of /TW in last but one line. But the file creation date is the date on which the file was created in that directory and not on which the file itself was created. For that reason the file creation date is only useful when the file was never copied or moved to another directory since first time creation.
I tested this batch file on following files:
C:\xxx\yyy\2004 !Apr_backup_2017_12_18_210001_2986007.bak
C:\xxx\yyy\2004 !Apr_backup_2017_12_19_210001_3168635.bak
C:\xxx\yyy\model%_backup_2017_12_19_210003_2544131.bak
C:\xxx\yyy\model%_backup_2017_12_20_210003_2544131.bak
C:\xxx\yyy\Subscribers=_backup_2017_12_19_210003_3012893.bak
C:\xxx\yyy\Subscribers=_backup_2017_12_20_210003_3012893.bak
The last modification date of each file matched the date in file name.
The output of the batch file was:
del "C:\xxx\yyy\2004 !Apr_backup_2017_12_18_210001_2986007.bak"
del "C:\xxx\yyy\model%_backup_2017_12_19_210003_2544131.bak"
del "C:\xxx\yyy\Subscribers=_backup_2017_12_19_210003_3012893.bak"
That is the expected result. The older file of each file pair would be deleted.
Then I thought getting [DESC] part of file name could be done easier as the remaining part of file name without file extension has a fixed length of 33 characters.
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "BackupFolder=C:\xxx\yyy"
rem Search for files matching the long wildcard pattern
rem *_backup_????_??_??_??????_???????.bak
rem in backup folder and assign each file name without
rem file extension to environment variable.
rem The last 33 characters are removed from each file name to get the
rem file description part at beginning of each file name. Then define
rem an environment variable of which name starts with FileDesc: and
rem ends with the file description. The value assigned to this
rem environment variable is 1.
for /F "delims=" %%I in ('dir "%BackupFolder%\*_backup_????_??_??_??????_???????.bak" /A-D /B /ON 2^>nul') do (
set "FileName=%%~nI"
call set "FileDesc:%%FileName:~0,-33%%=1"
)
rem Run command SET with FileDesc: to output all environment variables
rem starting with that string in name and sorted by name and process
rem this list whereby each line ends with =1 as value 1 is assigned
rem to each of these environment variables.
rem For each unique file description in output list assign the file
rem description with =1 appended to environment variable FileDesc
rem and run subroutine DeleteFiles.
for /F "tokens=2 delims=:" %%I in ('set FileDesc: 2^>nul') do (
set "FileDesc=%%I"
call :DeleteFiles
)
rem Restore initial environment on starting this batch file and exit it.
endlocal
goto :EOF
rem The subroutine DeleteFiles removes first from passed file description
rem the last two characters being always =1 from list of environment
rem variables starting with FileDesc: and appends the backup wildcard
rem pattern.
rem Command DIR is used to find all files in backup folder starting
rem with current file description and _backup_ and output the found
rem files sorted by last modification date with newest modified file
rem first and oldest modified file last.
rem The command FOR processing this list skips the first file name
rem output by DIR which means the newest file. All other, older
rem files perhaps also found by DIR are deleted one after the other.
:DeleteFiles
set "FilePattern=%FileDesc:~0,-2%_backup_*.bak"
for /F "skip=1 delims=" %%J in ('dir "%BackupFolder%\%FilePattern%" /A-D /B /O-D /TW') do ECHO del "%BackupFolder%\%%J"
goto :EOF
That batch file containing also ECHO left to command del in last but one line produces the same result on the six files in the backup folder.
I don't know if the batch file could be even more optimized without knowing which characters could exist in [DESC] part of the file names. I did not think about a possible further optimization.
Let us assume the list of unique [DESC] strings is well known and can be hard coded in the batch file, for example 2004 !Apr, model% and Subscribers= for the six files in my test case:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "BackupFolder=C:\xxx\yyy"
for %%I in ("2004 !Apr" "model%%" "Subscribers=") do for /F "skip=1 delims=" %%J in ('dir "%BackupFolder%\%%~I*_backup_*.bak" /A-D /B /O-D /TW 2^>nul') do del "%BackupFolder%\%%J"
endlocal
This batch file really deletes files because there is no ECHO in last but one line.
Oh yes, knowing the individual backup file names makes everything much easier.
The batch file can be even optimized to a single command line:
#for %%I in ("2004 !Apr" "model%%" "Subscribers=") do #for /F "skip=1 delims=" %%J in ('dir "C:\xxx\yyy\%%~I*_backup_*.bak" /A-D /B /O-D /TW 2^>nul') do #del "C:\xxx\yyy\%%J"
Best and simple backup deletion concept
On a backup storage media is created:
a backup of an entire machine with file name ComputerName_backup_YYYY_MM.tib every three months which is huge as taking 200 GiB and where it is enough to have only last backup on the backup storage media;
a backup of a folder with files not often updated with file name Folder_backup_YYYY_MM_DD.zip every Saturday which takes about 400 MiB on storage media where it is enough to be able to restore the last four weeks;
a backup of a database file with file name Database_backup_YYYY_MM_DD.bak every day which takes at the moment 20 MiB per backup, but is growing more or less constant as typical for database files and on where it should be possible to restore data entries of the last seven days.
The required minimum storage media size is:
(1+1) × 200 GiB + (4+1) × 400 MiB + (7+1) × (20×3) MiB
A storage media size of 1 TiB is really enough for approximately the next three years depending on growing rate of database backup on which an increase by a factor of three is included already in calculation.
It would be best to delete all backup files no longer needed on creating the daily database backup to keep the backup files management simple by using a single and simple batch file.
#echo off
set "BackupFolder=C:\xxx\yyy"
call :DeleteBackups 1 "ComputerName"
call :DeleteBackups 4 "Folder"
call :DeleteBackups 7 "Database"
goto :EOF
:DeleteBackups
for /F "skip=%1 delims=" %%I in ('dir "%BackupFolder%\%~2*_backup_*" /A-D /B /O-D /TW 2^>nul') do del "%BackupFolder%\%%I"
goto :EOF
Deletion of no longer needed backups can be really so easy on thinking about right strategy.
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
call /?
del /?
dir /?
echo /?
endlocal /?
for /?
goto /?
rem /?
set /?
setlocal /?
Read also the Microsoft article about Using Command Redirection Operators for an explanation of 2>nul. The redirection operator > must be escaped with caret character ^ on FOR command lines to be interpreted as literal character when Windows command interpreter processes this command line before executing command FOR which executes the embedded dir command line with using a separate command process started in background.
Here's an untested example script which should work as long as you don't have file names where [DESC] contains _, = or other problematic characters.
#Echo Off
SetLocal DisableDelayedExpansion
For /F "Delims==" %%A In ('Set _[ 2^>Nul') Do Set "%%A="
If /I Not "%CD%"=="C:\xxx\yyy" (Set "_[:]=T"
PushD "C:\xxx\yyy" 2>Nul||Exit /B)
For /F "Tokens=1* Delims=_" %%A In ('Dir /B /O-N *_backup_*_*_*_*_*.bak'
) Do If Defined _[%%A] (Del /A /F "%%A_%%B") Else Set "_[%%A]=T"
If Defined _[:] PopD
EndLocal
Exit /B
How to copy files from a folder tree to a single folder and only the latest files using batch commands?
These copies must happen every one hour and avoid overwrite rather than copy only latest files.
Current command:
for /R "D:\Logshipping\NDWAnalyzer\" %%f in (*.*) do copy "%%f" "D:\LogShipping1\NDWAnalyzer\"
The above command copies every time whole files in the folder rather than I need only to copy the latest - the files which are not present in the destination folder.
This is an easy task using xcopy with parameter /M which results in copying only files with archive attribute set and clearing this attribute after copying. The archive attribute is automatically set for a new or modified file.
#echo off
rem Check before copying for deletion of files in destination directory
rem which were copied already before and not updated in the meantime, i.e.
rem no archive attribute set anymore on file, but file is also not existing
rem anymore in destination directory. The archive attribute is set temporarily
rem on those files before running xcopy. This helps also in case of renaming
rem a file in one of the source directories which was copied already before
rem with previous name to destination directory as the rename operation does
rem not set archive attribute.
for /R "D:\Logshipping\NDWAnalyzer" %%F in (*) do (
if not exist "D:\LogShipping1\NDWAnalyzer\%%~nxF" %SystemRoot%\System32\attrib.exe +a "%%~F"
)
rem Copy from all directories in D:\Logshipping\NDWAnalyzer all files with
rem archive attribute set (= modified or added since last run) to directory
rem D:\LogShipping1\NDWAnalyzer with clearing the archive attribute after
rem copy and with copying also all attributes including read-only attribute.
for /R "D:\Logshipping\NDWAnalyzer" %%D in (.) do (
%SystemRoot%\System32\xcopy.exe "%%~fD\*" "D:\LogShipping1\NDWAnalyzer\" /C /H /I /K /M /Q /R /Y >nul
)
It is up to you if you want to use first + second loop or just second loop.
With removing first loop it would be possible to delete files in destination directory still existing in a source directory or rename a file in a source directory which was copied already before to destination directory without being copied once again.
For more details on the used commands open a command prompt window and run
attrib /?
for /?
if /?
xcopy /?
A help is output for each command which should be read to understand the two loops above.
Since I cannot add a comment, I am asking a related question.
The original posting found here works quite well.
Is there a way to use this with a list of file names? I've seen where a file list can be passed to the ROBOCOPY command but I have been unable to get it to work.
Taking a step back, I have a series of folders and there are specific files inside of them that I want to copy out to a single folder. I have a text file which lists the names of these files.
I am looking for a batch routine that will look for each of the files in the text file in each of the folders and then copy the files out to a new folder.
Thank you!
Test this - the file.txt has a filename on each line.
It doesn't handle filename conflicts.
#echo off
cd /d "c:\base\folder"
for /f "usebackq delims=" %%a in ("file.txt") do (
for /f "delims=" %%b in ('dir "%%a" /b /s /a-d ') do copy "%%b" "d:\target\folder"
)
I recently had to tackle this problem, and many files that I wanted to move to from the hierarchy to a single folder had the same name as each other, and I wanted to still flatten the hierarchy without them to being over-written.
What I did was write a script that moves the file, but renames it with the old hierarchy path in the name
for example:
source files:
C:\files\somefiles\file.txt
C:\files\otherfiles\file.txt
destination is C:\newdir\
files are created as
C:\newdir\somefiles-file.txt
C:\newdir\otherfiles-file.txt
here is the code, batch file 1 goes thru the files, batch file 2 renames and moves them (could also copy instead, if you want to preserve the source:
#echo off
for /r %%f in (*.*pr) do #renameandmovefilespart2.bat "%%f" "%%~ff" "%%~xf"
renameandmovefilespart2.bat
#echo off
Setlocal EnableDelayedExpansion
rem set the whole file path
set origWhole=%1
set origPathOnly=%2
set extension=%3
rem here you can set where the directory to hold the flattened hierarchy is
set destDir=c:\destinationDir\
rem set the directory to do a string replace
rem make this the starting directory, that you dont want in the newly renamed files
set startingDir=C:\starting\directory\
set nothing=
set slash=\
rem here you can set what the character to represent the directory indicator \ in the new files
set reaplcementDirectoryCharacter=--
set quote="
rem cut out the starting part of the directory
call set newname=%%origWhole:!startingDir!=!nothing!%%
rem replace slashes with new character
call set newname=%%newname:!slash!=!reaplcementDirectoryCharacter!%%
rem remove quotes
call set newname=%%newname:!quote!=!nothing!%%
rem #echo shortened: %newname%
rem #echo source path: %origPathOnly% newPath: %startingDir%
rem #echo extension: %extension%
rem rename the files
ren %origWhole% %newname%
rem prepare to move the file, clean up the source path
call set origPathOnly=%%origPathOnly:!quote!=!nothing!%%
move "%origPathOnly%%newname%" "%destDir%"
I have a program that creates a backup every 30 minutes, but does not delete older files. I am looking to make a script that will run on task scheduler that will look in the folder that is created by the program. It creates folder by date as follows /year/month/day/*name of file.zip. I have done a little research and come up with
#ECHO OFF
for /f "usebackq delims== tokens= 2" %%i in ( WMIC OS GET localdatetime /format:value ) do set localTime=%%i
set "YYYY=%localTime:~0,4%"
set "M=%localTime:~4,1%"
set "MM=%localTime:~5,1%"
set "D=%localTime:~6,1%"
set "DD=%localTime:~7,1%"
if %M%==0 set M=
if %D%==0 set D=
cd /d "~%~dp0"
FOR /F "tokens=*" %%X IN ('dir .\%YYYY%\%M%%MM%\%D%%DD%\Backup-serversave-%YYYY%-%M%%MM%-%D%%DD%--*.zip /b /o:-d') DO set FILENAME=%%~X
rem FILENAME now contains the last element
if "%FILENAME%" neq "" DEL "%FILENAME%"
pause
This script looks in the folder by todays date and than should delete the file that is oldest in the folder. But every time I run it, all I get is a "cannot find file" and it tells me the exact file that it says it can't find. So I need a little help to make it work. It would be also great if there is any way to make it only delete after five files are in the folder. If anyone can help me that would be amazing.
Update. Got a good answer but need a little more help. If anyone out there can find a way to make it delete yesterdays folder at the change of the day that would be amazing I am looking all over and so far have found nothing.
You did already a quite good job on writing the batch file, but made some small mistakes.
Using in first line ECHO ON helped to find those mistakes.
The following lines with some corrections for the batch file work.
#ECHO OFF
for /f "usebackq delims== tokens=2" %%i in ( `WMIC OS GET localdatetime /format:value` ) do set localTime=%%i
set "YYYY=%localTime:~0,4%"
set "M=%localTime:~4,1%"
set "MM=%localTime:~5,1%"
set "D=%localTime:~6,1%"
set "DD=%localTime:~7,1%"
if %M%==0 set M=
if %D%==0 set D=
cd /d "%~dp0"
for /F "usebackq tokens=* skip=5" %%X in ( `dir .\%YYYY%\%M%%MM%\%D%%DD%\Backup-serversave-%YYYY%-%M%%MM%-%D%%DD%--*.zip /b /o:-d` ) do del .\%YYYY%\%M%%MM%\%D%%DD%\%%X
pause
No file was not deleted ever because you used command dir to find the ZIP files in a subdirectory. The name of a found file was referenced with %%X, but without the path to this file. As the batch file did not change into the directory on which dir searched for the ZIP files, command DEL could not find the file to delete in the current working directory.
The first 5 lines returned by command dir containing the 5 newest files according to last modification date are ignored by using skip=5. Therefore only files with todays date in name and older than the first 5 files in directory listing are deleted and the newest 5 files are kept.
There was also a mistake in line with command cd and some other small mistakes all found by using ECHO ON in first line.
Deleting the previous day folders is difficult if the batch file should find out automatically which folders should be deleted and which folders should be kept.
Here is a solution which makes this task simple and which can be appended to the other lines in my other answer for deleting the older backup files.
if exist .\%YYYY%\%M%%MM%\%D%%DD%\Backup-serversave-%YYYY%-%M%%MM%-%D%%DD%--*.zip (
md TempBackup
copy /V .\%YYYY%\%M%%MM%\%D%%DD%\*.zip TempBackup >nul
for /F "usebackq" %%D in ( `dir /AD /B 20*` ) do rd /S /Q %%D
md %YYYY%\%M%%MM%\%D%%DD%
copy /V TempBackup\*.zip %YYYY%\%M%%MM%\%D%%DD% >nul
rd /S /Q TempBackup
)
First, a check is made if there are already ZIP files today stored to prevent deleting older backups if not at least one backup is created today.
Next a directory with name TempBackup is created into which are copied the currently existing backup files created today.
Then a dir command is executed which lists only the names of the directories starting with 20. Currently this is just the directory 2014, but on 2015-01-01 it will be additionally 2015. All those directories are recursively and silently deleted.
Next the directory tree for today is recreated and the ZIP files are copied back into this directory.
Finally the temporary backup directory for the ZIP files is deleted, too.
I'm wondering now why using a directory with current year as name containing a directory with current month as name containing a directory with current day of month as name if the ZIP files itself contain also year, month and day in name, and all ZIP files except the last 5 ZIP files saved today should be deleted?
It would be perhaps easier to manage to create all ZIP files in same directory and delete all ZIP files except the last 5 according to last modification date.
On the other hand the management with directories make it possible to keep backups from several days like one month and just delete the backup of previous month from time to time either manually or automatically for example on 15th of every month.
I have a requirement to move the files once processed to a new folder and we will need to add a time stamp as suffix to the file that is moved. In a day, we might be receiving the same file(s) multiple times. I am a novice to batch scripting.
For Ex:
Source Folder:
C:\SourceFiles
a.xml,
b.xml,
c.xml
When I copy/move the files to the destination folder, this is how they need to look as below
Destination Folder:
C: \DestinationFiles
a_08082013_10_16.xml,
b_08082013_10_16.xml,
c_08082013_10_16.xml
Any help would be highly appreciated.
This might help:
for /f "tokens=*" %%n in ('date /t') do set thisdate=%%n
:: replace '/' with '-' as /'s don't work in file names.
set thisdate=%thisdate:/=-%
for /f "tokens=*" %%n in ('time /t') do set thistime=%%n
:: replace ':' with '-'
set thistime=%thistime::=-%
set firstfolder=c:\first_folder
set secondfolder=c:\second_folder
for /r "%firstfolder%" %%n in (*) do call :movefile "%%~nn" "%%~xn" "%%~dpn"
goto :EOF
:movefile
move "%~3\%~1%~2" "%secondfolder%\%~1_%thisdate%-%thistime%-%random%%~2"
Note: The output of the date and time commands depends on windows date / time settings. You can't control that from the prompt. If you need to control the output programmatically, it would likely be best to use another language (e.g. vbscript) or a third party utility (e.g. windows version of GNU date command).
Don't put the second folder under the first one as 'for /r' walks the entire sub-tree starting at first_folder and moves any files it finds to second folder, renaming them in the process.
We add a random value to the end, to take care of duplicates that are processed at the same time.
If there are multiple subfolders, 'move' won't make new folders. In that case, you might prefer robocopy or xcopy (type robocopy /? or xcopy /? at the prompt for help with these commands). What the above does is to move everything from all subfolders of the input folder to one output folder.