long time learner, first time poster. So i was tasked to find a way to be able to delete a file based off of date. the cmd/batch file should read todays date, look at the designated directory and tell you what to delete. I have borrowed a great deal from other posts here and even added a "choice" option just to be safe. The batch file will read all files in the directory and list if it should be deleted or kept. then it will ask are you sure you want to delete. if "y" is selected then it should delete the file but whenever it tries it says the file cannot be found. I know it must be because of the spaces in the file name. When i rename the files and remove the spaces it deletes them just fine. Im sorry if im all over the place, any help would be much appreciated. hereis what i have so far
#echo off
setlocal ENABLEDELAYEDEXPANSION
set day=86400
set /a year=day*365
set /a strip=day*7
set dSource=I:\Test
call :epoch %date%
set /a slice=epoch-strip
for /f "delims=" %%f in ('dir /a-d-h-s /b /s %dSource%') do (
call :epoch %%~tf
if !epoch! LEQ %slice% (echo DELETE %%f ^(%%~tf^)) ELSE echo keep %%f ^(%%~tf^)
)
echo/
if exist "%SystemRoot%\System32\choice.exe" goto UseChoice
setlocal EnableExtensions EnableDelayedExpansion
:UseSetPrompt
set "UserChoice=N"
set /P "UserChoice=Are you sure [Y/N]? "
set "UserChoice=!UserChoice: =!"
if /I "!UserChoice!" == "N" endlocal & goto :EOF
if /I not "!UserChoice!" == "Y" goto UseSetPrompt
endlocal
goto Continue
:UseChoice
%SystemRoot%\System32\choice.exe /C YN /N /M "Are you sure [Y/N]? "
if errorlevel 2 goto :EOF
for /f "delims=" %%f in ('dir /a-d-h-s /b /s %dSource%') do (
call :epoch %%~tf
if !epoch! LEQ %slice% del /f %%f ^(%%~tf^)
)
PAUSE
exit /b 0
rem Args[1]: Year-Month-Day
:epoch
setlocal ENABLEDELAYEDEXPANSION
for /f "tokens=1-6 delims=-;+^_?" %%d in ("echo %1") do set Years=%%d& set Months=%%e& set Days=%%f
if "!Months:~0,1!"=="0" set Months=!Months:~1,1!
if "!Days:~0,1!"=="0" set Days=!Days:~1,1!
set /a Days=Days*day
set /a _months=0
set i=1&& for %%m in (31 28 31 30 31 30 31 31 30 31 30 31) do if !i! LSS !Months! (set /a _months=!_months! + %%m*day&& set /a i+=1)
set /a Months=!_months!
set /a Years=(Years-1970)*year
set /a Epoch=Years+Months+Days
endlocal& set Epoch=%Epoch%
exit /b 0
Related
#Mofi helped me a lot on this answer of mine
How to organize files in folders?
Now I tried myself to add a feature, it's working, but for the respect of this wonderful job Mofi did, I think, there is a more professional way to do it.
I need the batch to stop after it created a number of folders.
Let's say, I have 100000 .mp4 files in the main directory. The script processes now them all together.
For example, I need to use only 1200 files and group them into 120 folders. I can do that now with FoldersInFolders=120 and FilesInFolder=10. The issue is that after finishing Canal-1 it continues with Canal-2 to Canal-10, etc. until all 100000 MP4 files are moved into directories and I don't want that. I want to stop after 120 folders overall and process only 1200 from 100000 MP4 files which I have in my main folder.
I added the following two lines:
set "MaxTotalFolders=120"
if "!VideosIndex!"=="%MaxTotalFolders%" exit /b 0
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "FilesInFolder=10"
set "FoldersInFolders=120"
set "MaxTotalFolders=120"
for %%# in ("%USERPROFILE%\Videos" "D:\Videos" "F:\Temp\Videos" "\\MyNAS\Videos") do (
pushd %%# 2>nul
if not errorlevel 1 (
if not exist "*!*.mp4" (
setlocal EnableDelayedExpansion
set "FileCount=%FilesInFolder%"
set "CanalIndex=0"
set "VideosIndex=%FoldersInFolders%
for /F "eol=| delims=" %%I in ('dir *.mp4 /A-D /B /ON 2^>nul') do (
if !FileCount! == %FilesInFolder% (
set FileCount=0
if !VideosIndex! == %FoldersInFolders% (
set /A CanalIndex+=1
set VideosIndex=1
) else set /A VideosIndex+=1
set "TargetFolder=Canal-!CanalIndex!\Videos-!VideosIndex!"
md "!TargetFolder!" 2>nul
if "!VideosIndex!"=="%MaxTotalFolders%" exit /b 0
)
move /Y "%%I" "!TargetFolder!\" >nul
set /A FileCount+=1
)
endlocal
) else (
echo/
echo ERROR: Moving video files not possible because of file names with ! in name.
echo/
echo Please rename first the following files(s^) in: %%#
echo/
dir *!*.mp4 /A-D /B
echo/
pause
)
popd
) else (
echo/
echo ERROR: Failed to change to directory: %%#
echo/
pause
)
)
endlocal
Another feature that would be very helpful for me is to choose the MP4 files randomly every time I run the batch file.
The first batch file is without random selection of the MP4 files.
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "FilesInFolder=10"
set "FoldersInFolders=50"
set "MaxTotalFolders=120"
for %%# in ("%USERPROFILE%\Videos" "D:\Videos" "F:\Temp\Videos" "\\MyNAS\Videos") do call :MoveVideos %%#
goto EndBatch
:MoveVideos
pushd %1 2>nul
if errorlevel 1 (
echo/
echo ERROR: Failed to change to directory: %1
echo/
pause
goto :EOF
)
if exist "*!*.mp4" (
echo/
echo ERROR: Moving video files not possible because of file names with ! in name.
echo/
echo Please rename first the following files(s^) in: %1
echo/
dir *!*.mp4 /A-D /B
echo/
popd
pause
goto :EOF
)
setlocal EnableDelayedExpansion
set "FileCount=%FilesInFolder%"
set "CanalIndex=0"
set "FolderCount=-1"
set "VideosIndex=%FoldersInFolders%
for /F "eol=| delims=" %%I in ('dir *.mp4 /A-D /B /ON 2^>nul') do (
if !FileCount! == %FilesInFolder% (
set FileCount=0
if !VideosIndex! == %FoldersInFolders% (
set /A CanalIndex+=1
set VideosIndex=1
) else set /A VideosIndex+=1
set /A FolderCount+=1
if !FolderCount! == %MaxTotalFolders% goto EndMove
set "TargetFolder=Canal-!CanalIndex!\Videos-!VideosIndex!"
md "!TargetFolder!" 2>nul
)
move /Y "%%I" "!TargetFolder!\" >nul
set /A FileCount+=1
)
:EndMove
endlocal
popd
goto :EOF
:EndBatch
endlocal
There is used a subroutine with name MoveVideos to be able to use goto :EOF to stop moving files in the current directory into subdirectories on reaching the maximum number of total folders as defined with MaxTotalFolders and counted with FolderCount inside the second FOR loop.
This batch file creates with the three values defined at top for a directory with the MP4 files Video001.mp4 to Video1203.mp4 a directory structure with following files:
Canal-1
Videos-1
Video0001.mp4
:
Video0010.mp4
:
Videos-50
Video0491.mp4
:
Video0500.mp4
Canal-2
Videos-1
Video0501.mp4
:
Video0510.mp4
:
Videos-50
Video0991.mp4
:
Video1000.mp4
Canal-3
Videos-1
Video1001.mp4
:
Video1010.mp4
:
Videos-20
Video1191.mp4
:
Video1200.mp4
Video1201.mp4
Video1202.mp4
Video1203.mp4
This first batch file took 1.58 seconds for the file moving task on my PC.
The second batch file is with random selection of the MP4 files.
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "FilesInFolder=10"
set "FoldersInFolders=50"
set "MaxTotalFolders=120"
for %%# in ("%USERPROFILE%\Videos" "D:\Videos" "F:\Temp\Videos" "\\MyNAS\Videos") do call :MoveVideos %%#
goto EndBatch
:MoveVideos
pushd %1 2>nul
if errorlevel 1 (
echo/
echo ERROR: Failed to change to directory: %1
echo/
pause
goto :EOF
)
if exist "*!*.mp4" (
echo/
echo ERROR: Moving video files not possible because of file names with ! in name.
echo/
echo Please rename first the following files(s^) in: %1
echo/
dir *!*.mp4 /A-D /B
echo/
popd
pause
goto :EOF
)
setlocal EnableDelayedExpansion
set "FileCount=%FilesInFolder%"
set "CanalIndex=0"
set "FolderCount=-1"
set "VideosIndex=%FoldersInFolders%
set "TotalFileCount=0"
for /F "eol=| delims=" %%I in ('dir *.mp4 /A-D /B /ON 2^>nul') do set /A TotalFileCount+=1
:NextFile
if %TotalFileCount% == 0 goto EndMove
if %TotalFileCount% LEQ 32768 set /A "FileIndex=%RANDOM% %% TotalFileCount" & goto MoveFile
set /A FileGroups=TotalFileCount / 32768
set /A LastFileCount=TotalFileCount %% 32768
set "LastGroupIndex=%FileGroups%"
if not %LastFileCount% == 0 set /A FileGroups+=1
set /A GroupMultiplier=%RANDOM% %% FileGroups
if not %GroupMultiplier% == %LastGroupIndex% (set /A "FileIndex=GroupMultiplier * 32768 + %RANDOM%") else set /A "FileIndex=(FileGroups - 1) * 32768 + (%RANDOM% %% LastFileCount)"
:MoveFile
if not %FileIndex% == 0 (set "SkipValue=skip=%FileIndex% ") else set "SkipValue="
for /F "%SkipValue%eol=| delims=" %%I in ('dir *.mp4 /A-D /B 2^>nul') do (
if !FileCount! == %FilesInFolder% (
set FileCount=0
if !VideosIndex! == %FoldersInFolders% (
set /A CanalIndex+=1
set VideosIndex=1
) else set /A VideosIndex+=1
set /A FolderCount+=1
if !FolderCount! == %MaxTotalFolders% goto EndMove
set "TargetFolder=Canal-!CanalIndex!\Videos-!VideosIndex!"
md "!TargetFolder!" 2>nul
)
move /Y "%%I" "!TargetFolder!\" >nul
set /A FileCount+=1
set /A TotalFileCount-=1
goto NextFile
)
:EndMove
endlocal
popd
goto :EOF
:EndBatch
endlocal
This variant is much slower because of the randomization. It took 18.93 seconds to do the job on my PC.
It produces the same directory tree as the first batch file on current directory containing the MP4 files Video001.mp4 to Video1203.mp4, but which file is moved into which directory is really random.
Here is a third variant also with random selection of the next video file to move.
#echo off
echo %TIME%
setlocal EnableExtensions DisableDelayedExpansion
set "FilesInFolder=10"
set "FoldersInFolders=50"
set "MaxTotalFolders=120"
for /F "delims==" %%I in ('set # 2^>nul') do set "%%I="
for %%# in ("%USERPROFILE%\Videos" "D:\Videos" "F:\Temp\Videos" "\\MyNAS\Videos") do call :MoveVideos %%#
goto EndBatch
:MoveVideos
pushd %1 2>nul
if errorlevel 1 (
echo/
echo ERROR: Failed to change to directory: %1
echo/
pause
goto :EOF
)
if exist "*!*.mp4" (
echo/
echo ERROR: Moving video files not possible because of file names with ! in name.
echo/
echo Please rename first the following files(s^) in: %1
echo/
dir *!*.mp4 /A-D /B
echo/
popd
pause
goto :EOF
)
setlocal EnableDelayedExpansion
set "FileCount=%FilesInFolder%"
set "CanalIndex=0"
set "FolderCount=-1"
set "VideosIndex=%FoldersInFolders%
set "TotalFileCount=0"
for /F "eol=| delims=" %%I in ('dir *.mp4 /A-D /B /ON 2^>nul') do set "#!TotalFileCount!=%%I" & set /A TotalFileCount+=1
:NextFile
if %TotalFileCount% == 0 goto EndMove
if %TotalFileCount% LEQ 32768 set /A "FileIndex=%RANDOM% %% TotalFileCount" & goto MoveFile
set /A FileGroups=TotalFileCount / 32768
set /A LastFileCount=TotalFileCount %% 32768
set "LastGroupIndex=%FileGroups%"
if not %LastFileCount% == 0 set /A FileGroups+=1
set /A GroupMultiplier=%RANDOM% %% FileGroups
if not %GroupMultiplier% == %LastGroupIndex% (set /A "FileIndex=GroupMultiplier * 32768 + %RANDOM%") else set /A "FileIndex=(FileGroups - 1) * 32768 + (%RANDOM% %% LastFileCount)"
:MoveFile
if not %FileIndex% == 0 (set "SkipValue=skip=%FileIndex% ") else set "SkipValue="
for /F "%SkipValue%tokens=1* delims==" %%I in ('set #') do (
if !FileCount! == %FilesInFolder% (
set FileCount=0
if !VideosIndex! == %FoldersInFolders% (
set /A CanalIndex+=1
set VideosIndex=1
) else set /A VideosIndex+=1
set /A FolderCount+=1
if !FolderCount! == %MaxTotalFolders% goto EndMove
set "TargetFolder=Canal-!CanalIndex!\Videos-!VideosIndex!"
md "!TargetFolder!" 2>nul
)
set "%%I="
move /Y "%%J" "!TargetFolder!\" >nul
set /A FileCount+=1
set /A TotalFileCount-=1
goto NextFile
)
:EndMove
endlocal
popd
goto :EOF
:EndBatch
endlocal
It defines for each MP4 video file name an environment variable with a name beginning with # and a number incremented on each file and assigns the current file name to the environment variable. It processes this list of environment variables reduced by one environment variable after each file move.
It causes less file system accesses as the second batch file. But it took 20.54 seconds in my test on my PC to complete the video file movement task.
The main cause of the much longer time with random file selection is the fact that for each file to move one more cmd.exe must be started in background to output the new list of video file names in current directory, capture that file names list, skip all file names up to current random file index number and move the randomly selected file.
It would be much better if the list of environment variables with the video file names in current directory could be updated with removal of the environment variable of a just moved file and check if an environment variable with current randomly determined file index number still exists in the list to move that file and otherwise determine a new random file number index as often as needed. But such an approach can easily result in determining again and again randomly a file index number of video files which were moved already before. The result could be a nearly endless running loop on number of remaining files in environment variables list becomes small. The Windows Command Processor is not designed for such a task.
The randomization is tricky in case of the number of files in a directory is greater than 32768 because of %RANDOM% expands to a random decimal number between 0 and 32767. A cascaded randomized file index number determination is added for that reason to the code of the batch file.
The randomization of the selection of next video file to move causes also a lot of batch file open, seeking to next line to process, processing that line and perhaps read more lines and process them to, remember current position in batch file, batch file close. The first batch file is for that reason much more efficient. A more modern and powerful script interpreter like PowerShell would be much better for that task if randomized movement of video files is an important requirement.
To understand the commands used and how they work, open a command prompt window, execute there the following commands, and read the displayed help pages for each command, entirely and carefully.
call /?
dir /?
echo /?
endlocal /?
for /?
goto /?
if /?
md /?
move /?
pause /?
popd /?
pushd /?
set /?
setlocal /?
Im Marvin and I'd like to write a batch file (that works with windows command prompt) which would delete all folders which are older than 7 days in a specific folder, but based on the date written in the file name and not the modification date of the files.
Here are how the files are named in the folder, they all follow this specific template : CYYYYMMDDHHMMSS
Thanks in advance for your help.
Sorry for the question. I dont have much knowledge on creating batch file. I try to copy some of the solution on other thread related on my concern but their issue is file deletion with extension name. On my concern is folder deletion with a file name template which I already mentioned above.
This is my sample code based on Folder Name Format and Path/Location of the folders:
Folder Name Format: CYYYYMMDDHHMMSS (e.g. C20200527180716, C20200528123944)
Folder Path: F:/v9Backup
#echo off
setlocal
set "folder=F:\v9backup"
REM set the number of days to substract
SET DAYS=7
REM Call function to check if the date is valid.
CALL :validdate "%days%" subdate
echo Older than: %subdate%
pushd "%folder%"
REM Get a list of the files
REM file pattern is: CYYYYMMDDHHMMSS
set "search=[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]"
FOR /F "tokens=1,2* delims=_" %%G IN ('dir /a-d /b C* 2^>nul ^|findstr /I /R /C:"^C%search%"') DO (
setlocal enabledelayedexpansion
set "fdate=%%H"
set "fdate=!fdate:~0,8!"
IF !fdate! lss %subdate% del "%%G_%%H_%%I"
endlocal
)
popd
pause
endlocal
GOTO :EOF
:validdate
setlocal
set "day=%~1"
set rand=%random%
md "%temp%\dummy%rand%\empty%rand%"
REM Get todays date
for /f "tokens=2 delims==" %%a in ('wmic OS Get localdatetime /value') do
set "dt=%%a"
REM set year month and day into its own variables.
set /a y=%dt:~0,4%
set /a m=1%dt:~4,2%
set /a d=1%dt:~6,2%
:loop
if "%day%"=="0" (
rd /s /q "%temp%\dummy%rand%"
endlocal &set "%~2=%y%%m:~-2%%d:~-2%"
GOTO :EOF
)
set /a d-=1
if %d% lss 101 (
set d=131
set /a m-=1
if %m% lss 101 (
set m=112
set /a y-=1
)
)
xcopy /d:%m:~-2%-%d:~-2%-%y% /t "%temp%\dummy%rand%\empty%rand%"
"%temp%\dummy%rand%" >nul 2>&1 && (set /a day-=1 & goto loop) || goto loop
GOTO :EOF
However, it is not running properly. Could you please help me on these.
Marvin - give this a try. I'm a novice as well, but believe this modified script will work for your needs. I've modified it to your specifications and have removed the pause which requires a keypress for the script to complete.
#echo off
setlocal
set "folder=F:\v9backup"
REM set the number of days to substract
SET DAYS=7
REM Call function to check if the date is valid.
CALL :validdate "%days%" subdate
echo Older than: %subdate%
pushd "%folder%"
REM Get a list of the files
REM file pattern is: CYYYYMMDDHHMMSS
set "search=[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]"
FOR /F "tokens=1,2* delims=C" %%G IN ('dir /a:d /b * 2^>nul ^|findstr /I /R /C:"^C%search%"') DO (
setlocal enabledelayedexpansion
set "fdate=%%G"
set "fdate=!fdate:~0,8!"
IF !fdate! lss %subdate% del /f /s /q "C%%G" 1>nul
IF !fdate! lss %subdate% rmdir /s /q "C%%G"
endlocal
)
popd
endlocal
GOTO :EOF
:validdate
setlocal
set "day=%~1"
set rand=%random%
md "%temp%\dummy%rand%\empty%rand%"
REM Get todays date
for /f "tokens=2 delims==" %%a in ('wmic OS Get localdatetime /value') do set "dt=%%a"
REM set year month and day into its own variables.
set /a y=%dt:~0,4%
set /a m=1%dt:~4,2%
set /a d=1%dt:~6,2%
:loop
if "%day%"=="0" (
rd /s /q "%temp%\dummy%rand%"
endlocal &set "%~2=%y%%m:~-2%%d:~-2%"
GOTO :EOF
)
set /a d-=1
if %d% lss 101 (
set d=131
set /a m-=1
if %m% lss 101 (
set m=112
set /a y-=1
)
)
xcopy /d:%m:~-2%-%d:~-2%-%y% /t "%temp%\dummy%rand%\empty%rand%" "%temp%\dummy%rand%" >nul 2>&1 && (set /a day-=1 & goto loop) || goto loop
GOTO :EOF
I have some CMD code that Rojo and Magoo helped me write that runs against some XML files in a directory. The code grabs a date and time from the files in the file name and creates a year and month folder from it and then moves the files into them. The problem that I'm having is the folder itself contains 914,000 xml files and the script just can't handle it. I need something faster or a way to multithread the script. Another option I was considering is to move a few thousand files at a time and just run it on those from a temp directory and at the very end of the script move those folders into the production location. Here is the code and another script to create the XML files to test. The date isn't validated but for this exercise, they don't need to be. This will be running on a Microsoft Server 2012 R2 VM.
running Processor Intel(R) Xeon(R) CPU E5-2650 0 # 2.00GHz, 2000 Mhz, 1 Core(s), 1 Logical Processor(s) and 4 gigs of ram. I'm also including the Powershell and VbScript tags in case someone can offer any advise for writing the code in those languages.
XML move script
#ECHO OFF
SETLOCAL
Title Reorganizing XMLs - DO NOT CLOSE THIS WINDOW!
color 0F
mode con: cols=100 lines=6
prompt $t $d$_$p$g
::Get start time
for /F "tokens=1-4 delims=:.," %%a in ("%time%") do (
set /A "start=(((%%a*60)+1%%b %% 100)*60+1%%c %% 100)*100+1%%d %% 100"
)
Echo Start time: %start%
set "sourcedir=C:\Temp\TestDummyFiles"
set "tempdir=C:\temp\xmlreorgtemp"
::call :Get1000Files %sourcedir% %tempdir% %total%
pushd %sourcedir%
SET "spinChars=\|/-"
for /f %%a in ('"prompt $H&for %%b in (1) do rem"') do set "BS=%%a"
set "spaces= "
SET /a filesMoved = 0, spinPos = 0, prev = 0
echo Moving XML Files...
setlocal enabledelayedexpansion
for /L %%I in (1,1,7) do set "BS=!BS!!BS!"
for /L %%I in (1,1,3) do set "spaces=!spaces!!spaces!"
For %%A in (%sourcedir%\*.xml) do set /a cnt+=1
echo.
Echo Total XML files: %cnt%
echo.
FOR /f "tokens=1*delims=" %%a IN ('dir /b /a-d "%sourcedir%\*.xml" ' ) DO (
set /a filesmoved += 1
call :spinner !filesmoved! "%%~nxa"
)
call :spinner %filesMoved% Done.
for /F "tokens=1-4 delims=:.," %%a in ("%time%") do (
set /A "end=(((%%a*60)+1%%b %% 100)*60+1%%c %% 100)*100+1%%d %% 100"
)
echo End time: %end%
set /A elapsed=end-start
rem Show elapsed time:
set /A hh=elapsed/(60*60*100), rest=elapsed%%(60*60*100), mm=rest/(60*100), rest%%=60*100, ss=rest/100, cc=rest%%100
if %mm% lss 10 set mm=0%mm%
if %ss% lss 10 set ss=0%ss%
if %cc% lss 10 set cc=0%cc%
echo Elapsed Time: %hh%:%mm%:%ss%
endlocal & echo;
exit /b 0
:Get1000Files
#echo off
setlocal enabledelayedexpansion
for /f %%a in ('dir "%~1" /b /a-d *.xml') do (
set /a cnt+=1 & move "%%~a" "%~2"
if !cnt! EQU 1000 exit /b
)
exit /b
:spinner <filecount> <filename>
set /a spinPos += 1, spinPos %%= 4, ten = %~1 / 10 * 10
if "%~2"=="Done." set ten=%~1
set "str=[!spinChars:~%spinPos%,1!] %ten% files moved... [%~2]"
set "str=%str:~0,79%"
call :length len "%str%"
set /a diff = 79 - len
if %diff% gtr 0 set "str=%str%!spaces:~-%diff%!"
set /P "=!BS:~-79!%str%"<NUL
if "%~2" NEQ "Done." call :process %~2
exit /b 0
:length <return_var> <string>
setlocal enabledelayedexpansion
if "%~2"=="" (set ret=0) else set ret=1
set "tmpstr=%~2"
for %%I in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) do (
if not "!tmpstr:~%%I,1!"=="" (
set /a ret += %%I
set "tmpstr=!tmpstr:~%%I!"
)
)
endlocal & set "%~1=%ret%"
exit /b 0
:process
FOR /f "tokens=2,3,6delims=_" %%m IN ("%~1") DO SET "date1=%%m"&SET "date2=%%n"&SET "whichdate=%%o"
IF DEFINED whichdate SET "date1=%date2%"
IF NOT DEFINED date2 exit /b 1
If not exist .\%date1:~0,4%\%date1:~4,2% MD .\%date1:~0,4%\%date1:~4,2%
MOVE %~1 .\%date1:~0,4%\%date1:~4,2%\ > nul
And the script to create some dummy files
#echo off
setlocal EnableDelayedExpansion
cd /d %~dp0
For /f %%a in ('copy /Z "%~dpf0" nul') Do set "CR=%%a"
set fileSize=%~Z1
set /a cnt=0
echo Creating files. Please wait.&echo.
:loop
if %cnt% GTR 5000 exit /b
set /a cnt+=1
set /p "=Creating %cnt% File(s) !CR!"<nul:
Call :random 2009 2015 yyyy
call :random 1 12 mm
call :random 1 31 dd
if %mm% LSS 10 set mm=0%mm%
if %dd% LSS 10 set dd=0%dd%
set /P "=0" > thisSize.txt < NUL
(for /L %%i in (0,1,30) do (
set /A "bit=(1<<%%i)&fileSize, fileSize&=~(1<<%%i)"
if !bit! neq 0 type thisSize.txt
if !fileSize! neq 0 type thisSize.txt >> thisSize.txt
)) > IDABCDEFG001_STUFF_%yyyy%%mm%%dd%_ABC_0_1234567890.xml
del thisSize.txt
goto :loop
exit /b
:random Min Max [RtnVar]
#echo off & setlocal
set /a rtn=%random% %% ((%~2)-(%~1)+1) + (%~1)
(endlocal
if "%~3" neq "" (set %~3=%rtn%) else echo:%rtn%
)
exit /b
The server has Powershell 4 on it.
Not powershell, but maybe this could do the work
#echo off
setlocal enableextensions disabledelayedexpansion
set "xmlFolder=C:\Temp\TestDummyFiles"
pushd "%xmlFolder%" && (
for %%x in ("*_*_*.xml") do if exist "%%x" (
for /f "tokens=2-4 delims=_" %%a in ("%%~nx") do if "%%c"=="" (set "fileDate=%%a") else (set "fileDate=%%b")
setlocal enabledelayedexpansion
for /f "tokens=1,2" %%a in ("!fileDate:~0,4! !fileDate:~4,2!") do (
endlocal
<nul set /p "=%%a\%%b : "
md ".\%%a\%%b" 2>nul
move /y "*_%%a%%b??_*.xml" ".\%%a\%%b" 2>nul | find /v ":"
)
)
popd
)
There are three reasons for your code to be slow (appart from the fact that you are handling 914000 files):
There are 914000!! files
call usage is slow. 914000 * #calls for each file = very slow
914000 status updates to console are slow
for /f
Yes, the for /f commands used in
FOR /f "tokens=1*delims=" %%a IN ('dir /b /a-d "%sourcedir%\*.xml" ' ) DO (
for /f %%a in ('dir "%~1" /b /a-d *.xml') do (
have one problem because:
The dir command has to enumerate the 914000 files
The full list needs to be loaded into memory before starting to process it
The for /f command loads data into a buffer. When the buffer is full a new bigger (4KB increase in windows 7) buffer is created and data is copied from the old buffer to the new and this process is repeated until all the data has been retrieved. Each time the buffer is resized a larger memory copy operation needs to be done so the time needed to handle all the data increases exponentially.
This means
914000 files * ( 50 chars file name + CR LF ) = 47528000 characters
47528000 characters / 4KB buffer increase = 11603 redim operations
11603 redim operations = 1103170928640 bytes moved in memory copy operations
To handle all this, the proposed code will
Use a simple for to enumerate the files. The process starts on the first file being found and more search operations are done as the files are being iterated.
Instead of processing each file, all the files matching a date are moved in only one move operation.
If you have a lot of files, then you may reorder its processing by the smallest number of groups. In your example code you create 5000 dummy files, but just in 6 years. The code below process files by year, then month:
#ECHO OFF
SETLOCAL EnableDelayedExpansion
set "sourcedir=C:\Temp\TestDummyFiles"
pushd %sourcedir%
:nextYear
for %%a in (*.xml) do set "fileName=%%a" & goto break
:break
if not defined fileName goto :EOF
FOR /f "tokens=2,3,6 delims=_" %%m IN ("%fileName%") DO SET "date1=%%m" & SET "date2=%%n" & SET "whichdate=%%o"
IF DEFINED whichdate SET "date1=%date2%"
IF NOT DEFINED date2 exit /b 1
set "YYYY=%date1:~0,4%"
set "MM=100"
for /L %%m in (1,1,12) do (
set /A MM+=1
MD "%YYYY%\!MM:~1!" 2> NUL
MOVE "*_%YYYY%!MM:~1!??_*.xml" "%YYYY%\!MM:~1!"
)
goto nextYear
In my opinion, you should started this topic with a description of the problem, like "I have 914,000 files with this format IDABCDEFG001_STUFF_yyyymmdd_ABC_0_1234567890.xml and I want to move they to folders with yyyy\mm structure". I really don't like to try to know the details of a problem reading code. I don't understand your code to get the date from the file name, so I just copied it...
I have the following code for a spinner that I found somewhere long ago. I'm trying to figure out how to modify it so it displays an update for every 1000 files moved. So, it would look like this:
Moving XML Files...| 1,000 Files moved
Moving XML Files.../ 2,000 Files moved
Moving XML Files...- 3,000 Files moved
Moving XML Files...\ 4,000 Files moved
Where the spinner chars continue to move. I'll be running this on close to a million files, so I really need to have an indication of what the status is. Any help or suggestions of a better way is greatly appreciated.
CODE
#echo off
setlocal
Call :SpinnerEx
exit /b
:SpinnerEx
setlocal EnableDelayedExpansion
for /f %%a in ('copy /Z "%~dpf0" nul') do set "CR=%%a"
FOR /L %%n in (1,1,50) DO (
call :spinner
ping localhost -n 1 > nul
)
exit /b
:spinner
set /a "spinner=(spinner + 1) %% 4"
set "spinChars=\|/-"
<nul set /p ".=Moving XML Files...!spinChars:~%spinner%,1!!CR!"
exit /b
And HERE is the code to actually do the moving provided by Magoo
Building on Magoo's script, I'd replace the
) DO SET "filename=%%a"&CALL :process
with
) DO (
SET "filename=%%a"&CALL :process
rem increment file counter
rem if total divided by 1000 has no remainder, advance the spinner
)
Something like this:
#ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir\t w o"
SET "spinChars=\|/-"
for /f %%a in ('"prompt $H&for %%b in (1) do rem"') do set "BS=%%a"
SET "filesmoved=0"
PUSHD "%sourcedir%"
set /P "=Moving XML Files...%spinChars:~0,1% 0 Files moved"<NUL
FOR /f "tokens=1*delims=" %%a IN (
'dir /b /a-d "%sourcedir%\*_*_*.xml" '
) DO (
SET "filename=%%a"&CALL :process
set /a filesmoved += 1, thousand = filesmoved %% 1000
setlocal enabledelayedexpansion
if !thousand! equ 0 call :spinner
endlocal
)
POPD
GOTO :EOF
:process
FOR /f "tokens=2,3,6delims=_" %%m IN ("%filename%") DO SET "date1=%%m"&SET "date2=%%n"&SET "whichdate=%%o"
IF DEFINED whichdate SET "date1=%date2%"
IF NOT DEFINED date2 GOTO :eof
ECHO(MD .\%date1:~0,4%\%date1:~4,2%
ECHO(MOVE "%filename%" .\%date1:~0,4%\%date1:~4,2%\
GOTO :EOF
:spinner
set "moved=%filesmoved%"
:spinner2
if %filesmoved% geq 4000 set /a filesmoved -= 4000 & goto :spinner2
set /a spinpos = filesmoved / 1000
for /L %%I in (1,1,50) do set /P "=%BS%"<NUL
set /P "=Moving XML Files...!spinChars:~%spinPos%,1! %moved% Files moved"<NUL
goto :EOF
The for /f... ("prompt $H...") line captures a backspace character to a variable (to %BS%). The for /L %%I in (1,1,50) line backspaces 50 times. Hopefully the rest is fairly self-explanatory.
If you'd like to test the logic without actually moving any files, here's the same script with the file iteration loop replaced with a simple for /L loop:
#ECHO OFF
SETLOCAL
SET "spinChars=\|/-"
for /f %%a in ('"prompt $H&for %%b in (1) do rem"') do set "BS=%%a"
SET "filesmoved=0"
set /P "=Moving XML Files...%spinChars:~0,1% 0 Files moved"<NUL
for /L %%I in (1,1,50000) do (
set /a filesmoved += 1, thousand = filesmoved %% 1000
setlocal enabledelayedexpansion
if !thousand! equ 0 call :spinner
endlocal
)
goto :EOF
:spinner
set "moved=%filesmoved%"
:spinner2
if %filesmoved% geq 4000 set /a filesmoved -= 4000 & goto :spinner2
set /a spinpos = filesmoved / 1000
for /L %%I in (1,1,50) do set /P "=%BS%"<NUL
set /P "=Moving XML Files...!spinChars:~%spinPos%,1! %moved% Files moved"<NUL
goto :EOF
I'm trying to use a batch file to move files in blocks of 30 if there are less than 20 files in %DataLoc%. I modified code from a prior question. The problem is in the FMove section of the file. No matter what I put in the for line, it gives me an error.
I want this to have the %HoldLoc% value, but have been hard coding it because of errors I get.
The environment is Windows 2008 R2 server.
Variations I have tried, as well as with and without quotes in the parentheses:
FOR %F IN (%HoldLoc%)
FOR %F IN (%%HoldLock%)
FOR %F IN (c:\Play\hold\*.tmp)
My Code:
SETLOCAL ENABLEEXTENSIONS
SETLOCAL ENABLEDELAYEDEXPANSION
echo on
set DataMax=20
set DataLoc=C:\Play\Data
Set HoldLoc=C:\Play\Hold
set count=0
FOR /F %%a in ('DIR /B %DataLoc%\*.tmp') do set /A count=count+1
if %count% GEQ %DataMax% (Goto Exit) else (GOTO FMove)
:FMove
Echo Gather Top 30 files
set SrcCount=0
set SrcMax=30
echo %HoldLoc%
FOR %F IN (c:\Play\hold\*.tmp) DO IF !SrcCount! LSS %SrcMax% (
SET /A SrcCount += 1
move /y %F "%DataLoc%"
)
Problem is that I get this in the output window, why won't the C be seen?
C:>set /A count=count+1
C:>if 19 GEQ 20 (Goto Exit ) else (GOTO FMove )
C:>Echo Gather Top 30 files
Gather Top 30 files
C:>set SrcCount=0
C:>set SrcMax=30
C:>echo C:\Play\Hold
C:\Play\Hold
\Play\hold\*.tmp) was unexpected at this time.
C:>FOR \Play\hold\*.tmp) DO IF !SrcCount! LSS 30 (
C:>
#echo off
set Source=C:\perl\Drift_Bat\IN
set Target=C:\perl\Drift_Bat\OUT
set MaxLimit=20
for /f "tokens=1* delims=[]" %%G in ('dir /A-D /B "%Source%\*.*" ^| find /v /n ""') do (
move /y "%Source%\%%~nxH" "%Target%"
if %%G==%MaxLimit% exit /b 0
)
Your script uses incorrect syntax for the loop variable in one of the two loops:
FOR %F IN (c:\Play\hold\*.tmp) DO …
Just try changing %F to %%F. Single percent sign plus letter is the syntax for loop variables when running loops directly from the command prompt. In batch scripts you should always use double-percent references for loop variables, just like in your FOR /F %%a loop.
I updated the code and got it to work by changing some things. Thanks Andy for the tips but I could not get it to work with the suggestions - I wouldn't be surprised if I did not follow them and that is on MY side, not yours.
FOR /F %%G IN ('DIR /B "%HoldLoc%"\*.tmp') DO IF !SrcCount! LSS %SrcMax% (
SET /A SrcCount += 1
Echo "%HoldLoc%"
Echo "%%G%"
Echo "%SrcCount%
move /y "%HoldLoc%"\"%%G" "%DataLoc%"
)
Here is what I ended up with - longer but still functional:
SETLOCAL ENABLEEXTENSIONS
SETLOCAL ENABLEDELAYEDEXPANSION
echo on
set DataMax=50
set DataLoc=C:\Test Data (x86)
Set HoldLoc=C:\Test Hold
set count=0
FOR /F %%a in ('DIR /B "%DataLoc%"\*.tmp') do set /A count=count+1
if %count% GEQ %DataMax% (Goto Exit) else (GOTO FMove)
:FMove
Echo Gather Top 30 files
set SrcCount=0
set SrcMax=30
FOR /F "TOKENS=*" %%a IN ('dir /A-D /O-D /B "%HoldLoc%"\*.tmp') DO (
SET /A SrcCount += 1
if !SrcCount! LEQ %SrcMax% (
MOVE /y "%HoldLoc%\%%a" "%DataLoc%"
)
)
goto Exit
:Exit
close