I'm trying to parallelise a script im using to backup some files using TSM.
Basically I run the following commands:
net use y: \\Server\share
dsmc incr y: -optfile="c:\path\folder\optionfile.opt" > c:\logs\logfile
.. and TSM backus up the entire directory tree in serial.
But I want to do this to create some seperate scripts to use more than one client to backup subsets of the directory tree (to speed things up)
FOR each subdirectory in \\Server\share
IF not processed 1/3 of the directories in \\server\share
DO
net use y: \\Server\share\**<append subdirectory here>**
Backup y
This is what I've got so far:
FOR /D %%a IN (W:) DO (
net use y: \\server\share\%%a
run command to backup y:
(y:\ should be now \server\share\first_dir_name)
net use y: /delete
timeout /t 10
)
I still need the if logic to count the number of folders in the share.
The first step was to find how many top level sub directories are in the share:
I use dir /a:d /s /b "Folder Path" | find /c ":\" but the command never completes.
Hope that makes sense!
Thanks.
You could do something like this using semaphore files.
In this example I have it set up for 8 separate tasks.
I recommend that you copy this code and test to see how it works and then change the number of scripts you want to run simultaneously or figure out the optimum number to run simultaneously and change the code to keep that number of scripts running until done. You didn't say how many separate scripts you will have.
SeparateProcess.bat
#echo off
setlocal EnableDelayedExpansion
set "Script[1]=Script1"
set "Script[2]=Script2"
set "Script[3]=Script3"
set "Script[4]=Script4"
set "Script[5]=Script5"
set "Script[6]=Script6"
set "Script[7]=Script7"
set "Script[8]=Script8"
set /a nScripts=8
rem For test only. Change values (seconds) below if desired
set /A testDelay[1]=10
set /A testDelay[2]=4
set /A testDelay[3]=8
set /A testDelay[4]=14
set /A testDelay[5]=17
set /A testDelay[6]=6
set /A testDelay[7]=9
set /A testDelay[8]=7
rem time to wait between checking status of all tasks
set /A timeout=5
set /A tasksProcessing=0
rem Set all Status to 0 and delete semaphore files.
for /L %%L in (1,1,%nScripts%) do set /A Status[%%L]=0 & del /f /q "SemaphoreFile!Script[%%L].txt!" 2>nul
echo This program will simulate starting separate processes
echo Open a Windows Explorer to %CD% and observe that semaphore files appear.
echo Observe that the files disappear when the processes complete after varying delays.
pause
echo Starting %nScripts% processes
for /L %%L in (1,1,%nScripts%) do (
echo( >"SemaphoreFile!Script[%%L]!.txt"
start /MIN cmd.exe /C !Script[%%L]!.bat "SemaphoreFile!Script[%%L]!.txt" !testDelay[%%L]!
set /A Status[%%L]=1
set /A tasksProcessing+=1
)
:KeepChecking
for /L %%A in (1,1,%nScripts%) do (
echo Waiting %timeout% seconds for !tasksProcessing! processes to complete
timeout %timeout% >nul
for /L %%L in (1,1,%nScripts%) do (
if !Status[%%L]!==1 if not exist "SemaphoreFile!Script[%%L]!.txt" (
set /A Status[%%L]=0
set /A tasksProcessing-=1
echo !Script[%%L]! finished.
)
)
if !tasksProcessing! GTR 0 goto :KeepChecking
goto :Done
)
:done
echo All processes completed.
pause
endlocal
Then have 8 scripts that are identical (for this test) where n is 1-8.
Scriptn.bat (code below)
#echo off
REM 1=Filespec of semaphor file
REM 2=Delay in seconds. This is for test only.
echo %~nx1 is doing something
timeout %~2
del "%~1"
got :eof
Execute SeparateProcess.bat to see how it works. When you are satisfied with the results replace the code in Script1-8 with whatever you need them to do. Just make sure that the last thing they do is delete the semaphore file to indicate that they are done.
I currently have a batch script that I'm using to copy one large zip file from a network folder to multiple machines on the network. I need to do these copies in parallel so I've got a for loop that runs through addresses and runs start robocopy. Here's what I've got
for /F "tokens=*" %%A in (IPlist.txt) do (
start robocopy "\\networkfolder" \\%%A
)
The problem is that I need to execute an extract on all the machines I just copied to but I need to wait until the robocopies have finished. I can't use start /wait in the for loop since that destroys the parallel copy. Is there a way I can force the script to wait until all the robocopies have finished? or maybe an alternate solution?
FYI: I cannot extract on the network folder first since the zip is a lot of little files and severely slows the transfer rate. It needs to be one big file when copying over the network.
setlocal EnableDelayedExpansion
set number=0
for /F "tokens=*" %%A in (IPlist.txt) do (
set /A number+=1
echo Flag > roboRunning.!number!
start robocopy "\\networkfolder" \\%%A ^& del roboRunning.!number!
)
:wait
if exist roboRunning.* goto wait
echo All robocopy processes have finished here
Here's some code I had lying around (syntax for inside a batch file):
SETLOCAL ENABLEDELAYEDEXPANSION
set robocopycount=0
:loop
for /F "usebackq" %%e IN (`tasklist /FI "IMAGENAME eq robocopy.exe"`) do if %%e==robocopy.exe set /A robocopycount=!robocopycount!+1
if %robocopycount% GEQ 1 goto continue
rem goto loop
:continue
This actually counts the number of executables running at once but it could easily be adapted to just check for existence. (In the case of this code it was checking to ensure that a certain number were running.)
SETLOCAL EnableDelayedExpansion
SET num=0
FOR /F "tokens=*" %%A in (IPlist.txt) do (
SET /A num+=1
ECHO Flag > RoboRunNum.!num!
START robocopy "\\networkfolder" \\%%A ^& del RoboRunNum.!num!
)
:Check
if exist RoboRunNum.* GOTO CHECK
echo Robocopy Processes fave Finished
New to batch scripting. Trying to write a script which checks for files with filename starting with LEND by polling a directory and if the file is not received by 17:30, then write a log message to log file. I have written the below batch script, the script runs fine as long as there are no files in the directory. As soon as I put files in there, it stops running. And restarts if I delete the files from that directory. Could you please advise where I am going wrong?
Thanks
#echo off
set I=0
set log=C:\logs\alerting.log
:recurse
for /f %%P in ('dir /b "C:\incoming\LEND*"') do (call :countfiles)
set Time=%time:~0,5%
echo Filecount: %I% at Time: %Time% >> %log%
if %Time%==17:30 goto OUT
pause 60
goto :recurse
:countfiles
set /a I+=1
:OUT
if %I%==0 echo LEND Files not received >> %log%
EXIT
Your code goes from countfiles to exit. Try this:
:countfiles
set /a I+=1
goto:eof
There were actually several flaws - reusing the time variable, expecting pause to be used as a delay, the time frame could have been missed in the test, and other more minor things.
This is tested as far as the file counting goes but not the 17:30 branch
I changed the style of a few commands and variable names (I is too much like l and i and 1 in many fonts)
#echo off
set log=C:\logs\alerting.log
:loop
for /f %%P in ('dir /b "C:\incoming\LEND*" 2^>nul ^| find /c /v "" ') do set c=%%P
set now=%time:~0,5%
echo Filecount: %c% at Time: %now%
echo Filecount: %c% at Time: %now% >> %log%
if %now:~0,2%%now:~3,2% GTR 1730 goto :OUT
ping -n 60 localhost >nul
goto :loop
:OUT
if %c% EQU 0 echo LEND Files not received >> %log%
EXIT
Why does this batch file never break out of the loop?
For /L %%f In (1,1,1000000) Do #If Not Exist %%f Goto :EOF
Shouldn't the Goto :EOF break out of the loop?
Edit:
I guess I should've asked more explicitly... how can I break out of the loop?
Based on Tim's second edit and this page you could do this:
#echo off
if "%1"=="loop" (
for /l %%f in (1,1,1000000) do (
echo %%f
if exist %%f exit
)
goto :eof
)
cmd /v:on /q /d /c "%0 loop"
echo done
This page suggests a way to use a goto inside a loop, it seems it does work, but it takes some time in a large loop. So internally it finishes the loop before the goto is executed.
You could simply use echo on and you will see that goto :eof or even exit /b doesn't work as expected.
The code inside of the loop isn't executed anymore, but the loop is expanded for all numbers to the end.
That's why it's so slow.
The only way to exit a FOR /L loop seems to be the variant of exit like the exsample of Wimmel, but this isn't very fast nor useful to access any results from the loop.
This shows 10 expansions, but none of them will be executed
echo on
for /l %%n in (1,1,10) do (
goto :eof
echo %%n
)
My answer
Use nested for loops to provide break points to the for /l loop.
for %%a in (0 1 2 3 4 5 6 7 8 9) do (
for %%b in (0 1 2 3 4 5 6 7 8 9) do (
for /l %%c in (1,1,10) do (
if not exist %%a%%b%%c goto :continue
)
)
)
:continue
Explanation
The code must be tweaked significantly to properly use the nested loops. For example, what is written will have leading zeros.
"Regular" for loops can be immediately broken out of with a simple goto command, where for /l loops cannot. This code's innermost for /l loop cannot be immediately broken, but an overall break point is present after every 10 iterations (as written). The innermost loop doesn't have to be 10 iterations -- you'll just have to account for the math properly if you choose to do 100 or 1000 or 2873 for that matter (if math even matters to the loop).
History
I found this question while trying to figure out why a certain script was running slowly. It turns out I used multiple loops with a traditional loop structure:
set cnt=1
:loop
if "%somecriteria%"=="finished" goto :continue
rem do some things here
set /a cnt += 1
goto :loop
:continue
echo the loop ran %cnt% times
This script file had become somewhat long and it was being run from a network drive. This type of loop file was called maybe 20 times and each time it would loop 50-100 times. The script file was taking too long to run. I had the bright idea of attempting to convert it to a for /l loop. The number of needed iterations is unknown, but less than 10000. My first attempt was this:
setlocal enabledelayedexpansion
set cnt=1
for /l %%a in (1,1,10000) do (
if "!somecriteria!"=="finished" goto :continue
rem do some things here
set /a cnt += 1
)
:continue
echo the loop ran %cnt% times
With echo on, I quickly found out that the for /l loop still did ... something ... without actually doing anything. It ran much faster, but still slower than I thought it could/should. Therefore I found this question and ended up with the nested loop idea presented above.
Side note
It turns out that the for /l loop can be sped up quite a bit by simply making sure it doesn't have any output. I was able to do this for a noticeable speed increase:
setlocal enabledelayedexpansion
set cnt=1
#for /l %%a in (1,1,10000) do #(
if "!somecriteria!"=="finished" goto :continue
rem do some things here
set /a cnt += 1
) > nul
:continue
echo the loop ran %cnt% times
you do not need a seperate batch file to exit a loop using exit /b if you are using call instead of goto like
call :loop
echo loop finished
goto :eof
:loop
FOR /L %%I IN (1,1,10) DO (
echo %%I
IF %%I==5 exit /b
)
in this case, the "exit /b" will exit the 'call' and continue from the line after 'call'
So the output is this:
1
2
3
4
5
loop finished
So I realize this is kind of old, but after much Googling, I couldn't find an answer I was happy with, so I came up with my own solution for breaking a FOR loop that immediately stops iteration, and thought I'd share it.
It requires the loop to be in a separate file, and exploits a bug in CMD error handling to immediately crash the batch processing of the loop file when redirecting the STDOUT of DIR to STDIN.
MainFile.cmd
ECHO Simple test demonstrating loop breaking.
ECHO.
CMD /C %~dp0\LOOP.cmd
ECHO.
ECHO After LOOP
PAUSE
LOOP.cmd
FOR /L %%A IN (1,1,10) DO (
ECHO %%A
IF %%A EQU 3 DIR >&0 2>NUL )
)
When run, this produces the following output. You'll notice that both iteration and execution of the loop stops when %A = 3.
:>MainFile.cmd
:>ECHO Simple test demonstrating loop breaking.
Simple test demonstrating loop breaking.
:>ECHO.
:>CMD /C Z:\LOOP.cmd
:>FOR /L %A IN (1 1 10) DO (
ECHO %A
IF %A EQU 3 DIR 1>&0 2>NUL
)
:>(
ECHO 1
IF 1 EQU 3 DIR 1>&0 2>NUL
)
1
:>(
ECHO 2
IF 2 EQU 3 DIR 1>&0 2>NUL
)
2
:>(
ECHO 3
IF 3 EQU 3 DIR 1>&0 2>NUL
)
3
:>ECHO.
:>ECHO After LOOP
After LOOP
:>PAUSE
Press any key to continue . . .
If you need to preserve a single variable from the loop, have the loop ECHO the result of the variable, and use a FOR /F loop in the MainFile.cmd to parse the output of the LOOP.cmd file.
Example (using the same LOOP.cmd file as above):
MainFile.cmd
#ECHO OFF
ECHO.
ECHO Simple test demonstrating loop breaking.
ECHO.
FOR /F "delims=" %%L IN ('CMD /C %~dp0\LOOP.cmd') DO SET VARIABLE=%%L
ECHO After LOOP
ECHO.
ECHO %VARIABLE%
ECHO.
PAUSE
Output:
:>MainFile.cmd
Simple test demonstrating loop breaking.
After LOOP
3
Press any key to continue . . .
If you need to preserve multiple variables, you'll need to redirect them to temporary files as shown below.
MainFile.cmd
#ECHO OFF
ECHO.
ECHO Simple test demonstrating loop breaking.
ECHO.
CMD /C %~dp0\LOOP.cmd
ECHO After LOOP
ECHO.
SET /P VARIABLE1=<%TEMP%\1
SET /P VARIABLE2=<%TEMP%\2
ECHO %VARIABLE1%
ECHO %VARIABLE2%
ECHO.
PAUSE
LOOP.cmd
#ECHO OFF
FOR /L %%A IN (1,1,10) DO (
IF %%A EQU 1 ECHO ONE >%TEMP%\1
IF %%A EQU 2 ECHO TWO >%TEMP%\2
IF %%A EQU 3 DIR >&0 2>NUL
)
Output:
:>MainFile.cmd
Simple test demonstrating loop breaking.
After LOOP
ONE
TWO
Press any key to continue . . .
I hope others find this useful for breaking loops that would otherwise take too long to exit due to continued iteration.
As jeb noted, the rest of the loop is skipped but evaluated, which makes the FOR solution too slow for this purpose. An alternative:
set F=1
:nextpart
if not exist "%F%" goto :EOF
echo %F%
set /a F=%F%+1
goto nextpart
You might need to use delayed expansion and call subroutines when using this in loops.
It is impossible to get out of a FOR /L before it completes all iterations.
I have debugged the execution of a FOR /L by the cmd.exe process.
Microsoft could document it better and save us all this effort.
Facts:
The loop is a simple while (TRUE) and the break only happens when the iteration limit is reached.
When an EXIT /b or a GOTO is encountered, no more commands are executed until the end of the iterations.
When an EXIT is encountered, the cmd.exe process is terminated.
Tests:
12 seconds
FOR /L %%G in (1,1,5000000) do (ECHO Only once & GOTO :EOF)
7 seconds
FOR /L %%G in (1,1,5000000) do (ECHO Only once & EXIT /b)
0 seconds, but this terminates the cmd.exe process
FOR /L %%G in (1,1,5000000) do (ECHO Only once & EXIT)
Assuming that the OP is invoking a batch file with cmd.exe, to properly break out of a for loop just goto a label;
Change this:
For /L %%f In (1,1,1000000) Do If Not Exist %%f Goto :EOF
To this:
For /L %%f In (1,1,1000000) Do If Not Exist %%f Goto:fileError
.. do something
.. then exit or do somethign else
:fileError
GOTO:EOF
Better still, add some error reporting:
set filename=
For /L %%f In (1,1,1000000) Do(
set filename=%%f
If Not Exist %%f set tempGoto:fileError
)
.. do something
.. then exit or do somethign else
:fileError
echo file does not exist '%filename%'
GOTO:EOF
I find this to be a helpful site about lesser known cmd.exe/DOS batch file functions and tricks: https://www.dostips.com/
Did a little research on this, it appears that you are looping from 1 to 2147483647, in increments of 1.
(1, 1, 2147483647): The firs number is the starting number, the next number is the step, and the last number is the end number.
Edited To Add
It appears that the loop runs to completion regardless of any test conditions. I tested
FOR /L %%F IN (1, 1, 5) DO SET %%F=6
And it ran very quickly.
Second Edit
Since this is the only line in the batch file, you might try the EXIT command:
FOR /L %%F IN (1, 1, 2147483647) DO #IF NOT EXIST %%F EXIT
However, this will also close the windows cmd prompt window.
I have 1 task where there is 1 file start generating through DB and took almost 1 hour to generate, there is 1 batch file which check this file whenever it is available it calls a new batch file and trigger "file sent", the issue is that this file which is generating continuously did not completely generated at the file picked the same and call the batch file
What i want to do to build a logic where i can compare the size of file within a loop with 2 variable sizeA and sizeB, and call another batch file when sizeA==sizeB now the only issue is i am not sure how to built this logic in a code.\
Here is what I have tried:
#echo off
setlocal EnableDelayedExpansion
set file = "C:\Users\rb54761\Desktop\New folder\File.txt"
set "size=0"
pause
:loop for /f "tokens=*" %%x in ('dir /s /a /b "%file%"2^>nul') do set /a size=%%~zx
echo !size!
PAUSE
if !size! == !size! goto call
goto loop
:call echo Success
EDIT I only saw your comment now for being a 2gb file, then the below code will not work.
If the file were to be smaller..here is an example:
#echo off
set "myfile=C:\Users\rb54761\Desktop\New folder\File.txt"
:start
for /f %%I in ("%myfile%") do set size=%%~zI
if %size% LSS 100 (echo file not ready %size% & timeout /t 10 & goto :start)
echo copy file.
as per above, It will check the file for a size of 100kb, if not that size yet, it will timeout for 10 seconds and goto the beginning and test again until the file reaches 100kb, where it will no longer meet the if statement and pass that line and simply echo copy file.
Please note there are no spaces in my set commands. I would suggest you run from cmd the help for /? for more on this command.
We can use below approach in order to compare the size of same file within a span of time
#echo off
setlocal EnableDelayedExpansion
set file="d:\File_Mask1\New_File\File.txt"
set "size=0"
If exist %file% GOTO loop
GOTO nofile
:loop
for /f "tokens=*" %%x in ('dir /s /a /b %file%') do set /a size=%%~zx
echo !size!
waitfor SomethingThatIsNeverHappening /T 120 >nul 2>&1
for /f "tokens=*" %%y in ('dir /s /a /b %file%') do set /a size1=%%~zy
echo !size1!
if !size! EQU !size1! goto call
goto loop
:call
echo Success
endlocal
This takes advantage of the archive attribute. Windows sets this attribute each time it writes to the file.
So if we can unset it, wait for a time and check it (I use dir /aa which will not find the file, when the attribute is not set)
#echo off
set sec=10
set "file="C:\Users\rb54761\Desktop\New folder\File.txt""
set /a secs=sec+1
:loop
attrib -a "%file%"
timeout %sec% >nul
dir /b /aa "%file%" >nul 2>&1 || goto :loop
echo %file% didn't change since %secs% seconds
(Note: your set file = ... line is wrong. The spaces around the = will become part of the variable name respective the value)
Advantage: no file size limit (files bigger 2GB will be handled fine too)