speed up folder reorg code - batch-file

Question

I have some CMD code that Rojo and Magoo helped me write that runs against some XML files in a directory. The code grabs a date and time from the files in the file name and creates a year and month folder from it and then moves the files into them. The problem that I'm having is the folder itself contains 914,000 xml files and the script just can't handle it. I need something faster or a way to multithread the script. Another option I was considering is to move a few thousand files at a time and just run it on those from a temp directory and at the very end of the script move those folders into the production location. Here is the code and another script to create the XML files to test. The date isn't validated but for this exercise, they don't need to be. This will be running on a Microsoft Server 2012 R2 VM.
running Processor Intel(R) Xeon(R) CPU E5-2650 0 # 2.00GHz, 2000 Mhz, 1 Core(s), 1 Logical Processor(s) and 4 gigs of ram. I'm also including the Powershell and VbScript tags in case someone can offer any advise for writing the code in those languages.
XML move script
#ECHO OFF
SETLOCAL
Title Reorganizing XMLs - DO NOT CLOSE THIS WINDOW!
color 0F
mode con: cols=100 lines=6
prompt $t $d$_$p$g
::Get start time
for /F "tokens=1-4 delims=:.," %%a in ("%time%") do (
set /A "start=(((%%a*60)+1%%b %% 100)*60+1%%c %% 100)*100+1%%d %% 100"
)
Echo Start time: %start%
set "sourcedir=C:\Temp\TestDummyFiles"
set "tempdir=C:\temp\xmlreorgtemp"
::call :Get1000Files %sourcedir% %tempdir% %total%
pushd %sourcedir%
SET "spinChars=\|/-"
for /f %%a in ('"prompt $H&for %%b in (1) do rem"') do set "BS=%%a"
set "spaces= "
SET /a filesMoved = 0, spinPos = 0, prev = 0
echo Moving XML Files...
setlocal enabledelayedexpansion
for /L %%I in (1,1,7) do set "BS=!BS!!BS!"
for /L %%I in (1,1,3) do set "spaces=!spaces!!spaces!"
For %%A in (%sourcedir%\*.xml) do set /a cnt+=1
echo.
Echo Total XML files: %cnt%
echo.
FOR /f "tokens=1*delims=" %%a IN ('dir /b /a-d "%sourcedir%\*.xml" ' ) DO (
set /a filesmoved += 1
call :spinner !filesmoved! "%%~nxa"
)
call :spinner %filesMoved% Done.
for /F "tokens=1-4 delims=:.," %%a in ("%time%") do (
set /A "end=(((%%a*60)+1%%b %% 100)*60+1%%c %% 100)*100+1%%d %% 100"
)
echo End time: %end%
set /A elapsed=end-start
rem Show elapsed time:
set /A hh=elapsed/(60*60*100), rest=elapsed%%(60*60*100), mm=rest/(60*100), rest%%=60*100, ss=rest/100, cc=rest%%100
if %mm% lss 10 set mm=0%mm%
if %ss% lss 10 set ss=0%ss%
if %cc% lss 10 set cc=0%cc%
echo Elapsed Time: %hh%:%mm%:%ss%
endlocal & echo;
exit /b 0
:Get1000Files
#echo off
setlocal enabledelayedexpansion
for /f %%a in ('dir "%~1" /b /a-d *.xml') do (
set /a cnt+=1 & move "%%~a" "%~2"
if !cnt! EQU 1000 exit /b
)
exit /b
:spinner <filecount> <filename>
set /a spinPos += 1, spinPos %%= 4, ten = %~1 / 10 * 10
if "%~2"=="Done." set ten=%~1
set "str=[!spinChars:~%spinPos%,1!] %ten% files moved... [%~2]"
set "str=%str:~0,79%"
call :length len "%str%"
set /a diff = 79 - len
if %diff% gtr 0 set "str=%str%!spaces:~-%diff%!"
set /P "=!BS:~-79!%str%"<NUL
if "%~2" NEQ "Done." call :process %~2
exit /b 0
:length <return_var> <string>
setlocal enabledelayedexpansion
if "%~2"=="" (set ret=0) else set ret=1
set "tmpstr=%~2"
for %%I in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) do (
if not "!tmpstr:~%%I,1!"=="" (
set /a ret += %%I
set "tmpstr=!tmpstr:~%%I!"
)
)
endlocal & set "%~1=%ret%"
exit /b 0
:process
FOR /f "tokens=2,3,6delims=_" %%m IN ("%~1") DO SET "date1=%%m"&SET "date2=%%n"&SET "whichdate=%%o"
IF DEFINED whichdate SET "date1=%date2%"
IF NOT DEFINED date2 exit /b 1
If not exist .\%date1:~0,4%\%date1:~4,2% MD .\%date1:~0,4%\%date1:~4,2%
MOVE %~1 .\%date1:~0,4%\%date1:~4,2%\ > nul
And the script to create some dummy files
#echo off
setlocal EnableDelayedExpansion
cd /d %~dp0
For /f %%a in ('copy /Z "%~dpf0" nul') Do set "CR=%%a"
set fileSize=%~Z1
set /a cnt=0
echo Creating files. Please wait.&echo.
:loop
if %cnt% GTR 5000 exit /b
set /a cnt+=1
set /p "=Creating %cnt% File(s) !CR!"<nul:
Call :random 2009 2015 yyyy
call :random 1 12 mm
call :random 1 31 dd
if %mm% LSS 10 set mm=0%mm%
if %dd% LSS 10 set dd=0%dd%
set /P "=0" > thisSize.txt < NUL
(for /L %%i in (0,1,30) do (
set /A "bit=(1<<%%i)&fileSize, fileSize&=~(1<<%%i)"
if !bit! neq 0 type thisSize.txt
if !fileSize! neq 0 type thisSize.txt >> thisSize.txt
)) > IDABCDEFG001_STUFF_%yyyy%%mm%%dd%_ABC_0_1234567890.xml
del thisSize.txt
goto :loop
exit /b
:random Min Max [RtnVar]
#echo off & setlocal
set /a rtn=%random% %% ((%~2)-(%~1)+1) + (%~1)
(endlocal
if "%~3" neq "" (set %~3=%rtn%) else echo:%rtn%
)
exit /b
The server has Powershell 4 on it.

Answer 1

Not powershell, but maybe this could do the work
#echo off
setlocal enableextensions disabledelayedexpansion
set "xmlFolder=C:\Temp\TestDummyFiles"
pushd "%xmlFolder%" && (
for %%x in ("*_*_*.xml") do if exist "%%x" (
for /f "tokens=2-4 delims=_" %%a in ("%%~nx") do if "%%c"=="" (set "fileDate=%%a") else (set "fileDate=%%b")
setlocal enabledelayedexpansion
for /f "tokens=1,2" %%a in ("!fileDate:~0,4! !fileDate:~4,2!") do (
endlocal
<nul set /p "=%%a\%%b : "
md ".\%%a\%%b" 2>nul
move /y "*_%%a%%b??_*.xml" ".\%%a\%%b" 2>nul | find /v ":"
)
)
popd
)
There are three reasons for your code to be slow (appart from the fact that you are handling 914000 files):
There are 914000!! files
call usage is slow. 914000 * #calls for each file = very slow
914000 status updates to console are slow
for /f
Yes, the for /f commands used in
FOR /f "tokens=1*delims=" %%a IN ('dir /b /a-d "%sourcedir%\*.xml" ' ) DO (
for /f %%a in ('dir "%~1" /b /a-d *.xml') do (
have one problem because:
The dir command has to enumerate the 914000 files
The full list needs to be loaded into memory before starting to process it
The for /f command loads data into a buffer. When the buffer is full a new bigger (4KB increase in windows 7) buffer is created and data is copied from the old buffer to the new and this process is repeated until all the data has been retrieved. Each time the buffer is resized a larger memory copy operation needs to be done so the time needed to handle all the data increases exponentially.
This means
914000 files * ( 50 chars file name + CR LF ) = 47528000 characters
47528000 characters / 4KB buffer increase = 11603 redim operations
11603 redim operations = 1103170928640 bytes moved in memory copy operations
To handle all this, the proposed code will
Use a simple for to enumerate the files. The process starts on the first file being found and more search operations are done as the files are being iterated.
Instead of processing each file, all the files matching a date are moved in only one move operation.

Answer 2

If you have a lot of files, then you may reorder its processing by the smallest number of groups. In your example code you create 5000 dummy files, but just in 6 years. The code below process files by year, then month:
#ECHO OFF
SETLOCAL EnableDelayedExpansion
set "sourcedir=C:\Temp\TestDummyFiles"
pushd %sourcedir%
:nextYear
for %%a in (*.xml) do set "fileName=%%a" & goto break
:break
if not defined fileName goto :EOF
FOR /f "tokens=2,3,6 delims=_" %%m IN ("%fileName%") DO SET "date1=%%m" & SET "date2=%%n" & SET "whichdate=%%o"
IF DEFINED whichdate SET "date1=%date2%"
IF NOT DEFINED date2 exit /b 1
set "YYYY=%date1:~0,4%"
set "MM=100"
for /L %%m in (1,1,12) do (
set /A MM+=1
MD "%YYYY%\!MM:~1!" 2> NUL
MOVE "*_%YYYY%!MM:~1!??_*.xml" "%YYYY%\!MM:~1!"
)
goto nextYear
In my opinion, you should started this topic with a description of the problem, like "I have 914,000 files with this format IDABCDEFG001_STUFF_yyyymmdd_ABC_0_1234567890.xml and I want to move they to folders with yyyy\mm structure". I really don't like to try to know the details of a problem reading code. I don't understand your code to get the date from the file name, so I just copied it...