I have the following layout for my test suite:
TestSuite1.cmd:
Run my program
Check its return result
If the return result is not 0, convert the error to textual output and abort the script. If it succeeds, write out success.
In my single .cmd file, I call my program about 10 times with different input.
The problem is that the program that I run 10 times takes several hours to run each time.
Is there a way for me to parallelize all of these 10 runnings of my program while still somehow checking the return result and providing a proper output file and while still using a single .cmd file and to a single output file?
Assuming they won't interfere with each other by writing to the same files,etc:
test1.cmd
:: intercept sub-calls.
if "%1"=="test2" then goto :test2
:: start sub-calls.
start test1.cmd test2 1
start test1.cmd test2 2
start test1.cmd test2 3
:: wait for sub-calls to complete.
:loop1
if not exist test2_1.flg goto :loop1
:loop2
if not exist test2_2.flg goto :loop2
:loop3
if not exist test2_3.flg goto :loop3
:: output results sequentially
type test2_1.out >test1.out
del /s test2_1.out
del /s test2_1.flg
type test2_2.out >test1.out
del /s test2_2.out
del /s test2_2.flg
type test2_3.out >test1.out
del /s test2_3.out
del /s test2_3.flg
goto :eof
:test2
:: Generate one output file
echo %1 >test2_%1.out
ping -n 31 127.0.0.1 >nul: 2>nul:
:: generate flag file to indicate finished
echo x >test2_%1.flg
This will start three concurrent processes each which echoes it's sequence number then wait 30 seconds.
All with one cmd file and (eventually) one output file.
Running things in parallel in batch files can be done via 'start' executable/command.
Windows:
you create a Batch File that essentially calls:
start TestSuite1.cmd [TestParams1]
start TestSuite1.cmd [TestParams2]
and so on, which is essentially forking new command lines,
which would work, if the application can handle concurrent users (even if its the same User), and your TestSuite1.cmd is able to handle parameters.
You will need to start the script with different parameters on different machines because whatever makes the program take so long for a task (IO, CPU time) will be in even shorter supply when multiple instances of your program run at once.
Only exception: the run time is cause by the program putting itself to sleep.
try the command start, it spawns a new command prompt and you can send along any commands you want it to run.
I'd use this to spawn batch files that run the tests and then appends to a output.txt using >> as such:
testthingie.cmd >> output.txt
Related
I'm trying to start a fixed number of concurrent batch processes that have similar filenames, all in the same directory:
TestMe1.bat
TestMe2.bat
TestMe3.bat
All batches should start at the same time, and all should complete before the batch continues, e.g. a master.bat:
echo Starting batches
(
start "task1" cmd /C "TestMe1.bat"
start "task2" cmd /C "TestMe2.bat"
start "task3" cmd /C "TestMe3.bat"
) | pause
echo All batches have stopped and we can safely continue
I'm trying to find a way to start all batches in the directory that match TestMe*.bat, so that I don't have to craft a new master.bat file each time. Something like this, but, you know, working:
echo Starting batches
(
for /f "delims=" %%x in ('dir /b /a-d TestMe*.bat') do start "task" cmd /C "%%x"
) | pause
echo All batches have stopped and we can safely continue
Thanks to this and this for getting me this far.
Advice and ideas gratefully received!
First you have to understand how this special method of using pipe with the pause command works and why it can be used for waiting for multiple parallel processes.
Taking your first working code sample as the starting point
(
start "task1" cmd /C "TestMe1.bat"
start "task2" cmd /C "TestMe2.bat"
start "task3" cmd /C "TestMe3.bat"
) | pause
It works because each new instance of CMD which is invoked by the start command will be started in a new console with its stdout and stderr redirected to to that new console, so the outputs of the child CMDs will not be redirected to the pipe and so will not be consumed by the pause command which is waiting for input from the pipe.
BUT,
Still each of the processes have inherited the pipe handle, so the handle to the pipe will remain open as long as the child processes are alive.
As a consequence the pause command in the right side of the pipe will remain active, waiting for input from the left side of the pipe until it receives input from the pipe(which will never happen) or all child processes have terminated and closed the handle to the pipe.
So the main CMD instance which is executing your master batch file, is actually waiting for the pause command (right side of the pipe) to terminate which in turn is waiting for child processes (potential pipe writers) to terminate.
Now it becomes clear why your second attempted code involving the FOR loop is not working.
(
for /f "delims=" %%x in ('dir /b /a-d TestMe*.bat') do start "task" cmd /C "%%x"
) | pause
The FOR command inside the pipe is executed by the child CMD in command line mode, In this mode the command echoing in on by default, so every command inside the FOR body will be echoed to standard output(the pipe) before execution, which in turn feeds the pause command and terminate its process before even the first child process is created, Therefor the batch file execution continues without ever waiting for the child processes to finish.
This can be easily resolved by putting the # after do will turn the command echoing off inside the FOR body.
(
for /f "delims=" %%x in ('dir /b /a-d TestMe*.bat') do #start "task" cmd /C "%%x"
) | pause>nul
You may also want to hide the output of pause command by redirecting it to nul
EDIT: It seems I missed the point of this question. Despite that, I think I'll leave this here anyway for other people to find. Feel free to downvote though.
Here are other suggestions on waiting for subordinate processes to complete.
Your main process should periodically check if every subordinate processes is done. There is a multitude of ways of doing that. I'll name a few:
Before that, please insert a delay between checks to not consume all CPU in a tight loop like this:
timeout /t 1 /nobreak
Marker files
Make subordinate processes create or delete a file when they are about to finish
You can create files like this:
echo ANYTHING>FILENAME
Main script should periodically check if those files exist like this:
if exist FILENAME goto IT_EXISTS
When all/none of the files exist your task is complete.
To prevent clutter, create files in a %random% folder inside %temp% directory and pass its name to subordinates via arguments %1, %2 ...
Check process existance by window title
Run tasklist and parse its output to determine if your subordinate programs still run.
Probably the easiest way is to use window name to filter out "your" processes.
Start them like this:
start "WINDOW_TITLE" "BATCH_FILE" ARGUMENTS
then search for them like this:
TASKLIST /fi "Windowtitle eq WINDOW_TITLE" | find ".exe"
if "%errorlevel%" == "0" goto PROCESS_EXISTS
if none are found your task is finished.
Further information can be found at: A, B, C
Check process existance by PID
Instead of window title you can use processes' PID.
To obtain it run your process with WMIC as described here
External programs
You can download or write an external program to facilitate inter-process communication. Examples include:
TCP server and clients with netcat
Use mkfifo from GnuWin32 coreutils to create a named pipe and use it
Windows semaphores and events via custom C/C#/AutoIT program
Utilities such as NirCmd and PsExec may simplify PID checking procedure
....and more
If none of solutions work for you please edit the question to narrow down your query.
I'm attempting to use Batch for the first time, and I'm running into some trouble with the timeout command. I'm making a simple backup program to backup certain files to my flash drive, and this is the beginning.I'm trying to make it so that the prompt does not show how much of the countdown is left. This is what I have:
ECHO Deleting current backup location...
RD /s /q F:\CurrentBackup
#TIMEOUT /t 10
ECHO Setting up new backup...
MKDIR F:\CurrentBackup
MKDIR F:\CurrentBackup\Documents
MKDIR F:\CurrentBackup\Pictures
MKDIR F:\CurrentBackup\Desktop
MKDIR F:\CurrentBackup\Music
rem xcopy C:\Eric D:\
Can anyone help me with this seemingly simple problem?
you can tell a command, where to write it's output. If you don't, it writes it to screen
TIMEOUT /t 10 >nul
will write the output to a "Null-Device" (also known as "Nirwana")
by the way: # does not suppress the output of a command, but suppress the repetition of the commandline. It's a kind of "one-line-echo off"
Normally, you put
#echo off
as the first line of a script.
echo off will turn command repetition off, and the # does the same thing for this very line (as the echo off is not yet active for this line)
I have a bunch of batch files that each start a bunch of executables to run concurrently. Each batch file starts 30 executables. When those 30 are done, I want the next batch of executables to run, again 30 at a time. The .exe's are called using the "start" command in the batch files and they work just fine - I can run the individual batch files for each group of 30 exe's and they run concurrently like they should.
I have created a "master" batch file that calls each sub-batch file but I can't figure out how to get it to run the sub-batch files in sequence, waiting for one to finish before starting the next.
If the master batch file is like this:
Batch1.bat
Batch2.bat
Batch3.bat
then only the first batch file is called - the others are never called.
If the master batch file is like this:
call Batch1.bat
call Batch2.bat
call Batch3.bat
then all of the sub-batch files start running at the same time and I get hundreds of executables trying to start up at the same time.
How do I make the master batch file call the first batch file, wait for it to finish, then call the next, wait for it to finish, then call the next, etc?
Thanks in advance,
rgames
When starting another batch CALL will start it in the same window and the called batch has access to the same variable context. So it can also change variables which affects the caller.
Using wait in your batch file to call the executable will wait for them to exit before.
START /WAIT batch1.bat
START /WAIT batch2.bat
Hope this helps
Excuse me. I think there is a misunderstanding here. If your master Batch file is this:
call Batch1.bat
call Batch2.bat
call Batch3.bat
then the Batch2.bat is called after Batch1.bat ends, and so on. You may do a small test to confirm this. On the other side, is possible that each BatchN.bat program uses the same variables? If so, then the last values left from Batch1.bat may interfere with Batch2.bat, and so on. In this case, you must add a Setlocal command at beginning of each Batch file.
I had to run a data export program for several files. My solution:
MasterBatch.bat:
#echo off
start /w batch1.bat
start /w batch2.bat
Batch1.bat
#echo off
cmd /c "c: & cd Program Files (x86)/PATH & targetProgram.exe -parametersToExportVideo1"
EXIT
Batch2.bat
#echo off
cmd /c "c: & cd Program Files (x86)/PATH & targetProgram.exe -parametersToExportVideo2"
EXIT
It may be adapted to run programs other problems.
You will have to create a signaling mechanism to aware EXE completation.
I would create a third level batch to run each EXE, creating a temp file before executing EXE and deleting it after.
In sub-batch I would wait until there were no more temp files.
So, initial batch:
call Batch1.bat
call Batch2.bat
call Batch3.bat
Sub-batch:
Set Index=0
Call :Exec exefile1 args ...
Call :Exec exefile2 args ...
...
:WaitAll
If Exist %Temp%\RUNNING_EXE.*.TMP GoTo :WaitAll
GoTo :EOF
:Exec
Set /A Index+=1
Echo %Index% > %Temp%\RUNNING_EXE.%Index%.TMP
Start Batch_3rd.BAT %*
GoTo :EOF
Finally, 3rd level batch, Batch_3rd.BAT:
%*
Del %Temp%\RUNNING_EXE.%Index%.TMP
%* are arguments passed from sub-batch (exe+arguments), %Index% is correct as start copies environment from sub-batch upon creation, and sub-batch don't change this copy.
One final note: you can probably merge all batches into a single batch file, calling it recursively.
My solution:
1)
I have four batch files:
Parent.bat and Batch1.bat, Batch2.bat, Batch3.bat
2)
Parent.bat contains the below lines (take a close note):
call Batch1.bat > result1.log
call Batch2.bat > result2.log
call Batch3.bat > result3.log
3)
Make sure "into the end of Each Child batch file", you have an echo statement.
After this echo statement there shouldn't be any code...
Say content of Batch1.bat file is:
echo begin
robocopy "C:\Users\DD\Documents\A" "C:\Users\DD\Documents\B"
echo end
echo this_is_the_last_line
Just for testing, i will use simple sub-batches like (all of them are the same)
#echo off
for /l %%a in (1 1 5) do start "" notepad.exe
And a master batch file
#echo off
setlocal enableextensions disabledelayedexpansion
set "flagFile=%temp%\%random%%random%%random%.tmp"
for %%a in ( "batch1.cmd" "batch2.cmd" "batch3.cmd" ) do (
call :startBatch %%a "%flagFile%"
)
:retryClean
echo %time% waiting for last batch to end
2>nul ( 9>"%flagFile%" break ) || ( >nul timeout 5 & goto :retryClean )
del /q "%flagFile%"
echo DONE
pause
goto :eof
:startBatch batchFile flagFile
echo %time% waiting to start "%~1"
2>nul ( 9>"%~2" call "%~1" ) || ( >nul timeout 5 & goto :startBatch )
echo %time% [ "%~1" ] STARTED
goto :eof
This code starts each of the sub-batches with an active redirection (user available stream 9 is used) to a temporary flag file. This will lock the flag file until all the processes started from sub-batches have ended as the redirection is inherited during the process creation.
All we have to do is to keep trying start the next batch file with the same redirection:
If the file is still locked (processes are running), the batch file can not be started, wait 5 seconds and retry again
If the file is not locked, the redirection can be created and the next batch file is started.
I'm executing programs in a batch file. I want to break if the batch script runs more than some time(say 10 sec). Because it might become an infinite loop.
cls
::set timeout here
program.exe
::after timeout continue from here
Is there any suggestion to do this in a windows batch file? Plus, is it possible to define the line of endless loop in a batch script?
When you run a program from a batch file, you either run it and wait for it to terminate before executing the next command in the batch file, or you launch the program and executing immediately continues in the batch file. There is no way to measure how long a program runs. However, you can use certain techniques to see if a program is running and terminate it. You can also insert delays. So you can do what you are asking (more or less):
cls
start program.exe
ping -n 10 -w 1000 127.0.0.1 > nul 2>&1
tasklist | find /i "program.exe"
if %errorlevel%==0 taskkill /im program.exe
This starts the program and immediately starts to execute the next line of code, the ping. The ping causes a 10 second delay. The tasklist command pipes the list to find, which looks for the program. If it is found in the list, then it is still running and errorlevel is set to 0, which means the next line terminates the program.
This isn't fool proof however, because if program.exe is a common program name, then tasklist and taskkill will terminate all processes with the same program name (not just the instance you started).
How about that?
start calc.exe
ping -n 10 localhost >nul
taskkill /F /IM calc.exe
How to run commands in a batch file which is inside another batch file......
I am trying to run commands in different console other than command prompt in a batch file but not able to do so.I am able to start the other console in batch file but not able to pass commands on to it.
My first interpretation of the question led me to believe that Sampath wanted one batch script that has two sets of commands. Calling it would run the 1st set of commands in the parent window, and a second window would open that would run the same script with thd 2nd set of commands.
"%~f0" will give the full path to the currently executing batch script. A simple command line argument serves as a switch to determine which code to run.
#echo off
if "%~1"==":PART2" goto %~1
::use this line if 2nd window is to remain open upon completion
::start "%~f0" :PART2
::use this line if 2nd window is to close upon completion
start cmd /c "%~f0" :PART2
echo Test parent output
pause
exit /b
:PART2
echo Test child output
pause
exit /b
Andriy M suggests Sampath wants to be able to dynamically send commands to the 2nd window. This can be done with 2 scripts that I will call master.bat and slave.bat.
The slave.bat simply reads commands from stdin and executes them. The master.bat launches the slave with input redirected to a command file and then appends commands to the command file.
Here is an example of master.bat that demonstrates dymamically sending commands to the slave. Note that the master prompts for a command, but the slave window will have the focus. Make sure you click on the master so you can enter the command of your choice.
#echo off
:: create an empty command file
type nul >cmds.txt
:: start the slave with input redirected to the command file
start slave.bat ^<cmds.txt
:: issue some commands by appending them to the command file
>>cmds.txt echo echo command 1
>>cmds.txt echo echo command 2
>>cmds.txt echo echo(
>>cmds.txt echo rem /?
:: ask for a command to send to the slave
set /p "cmd=Enter a command to be sent to the slave: "
:: send the command
>>cmds.txt echo %cmd%
::pause so we can see the results in the slave window
for /l %%n in (1 1 1000000) do rem
::tell the slave to exit
>>cmds.txt echo exit
And here is the slave.bat
#echo off
:top
set "cmd="
set /p "cmd="
%cmd%
goto :top
You could try a call statement:
call batchname.bat
this will run the specified batch file in the current open prompt
It almost sounds like what you want is a file that holds commands that you want to run, and to use a batch script to call on those commands when you want?
I've implemented this by creating a batch file that holds all the commands (code snippets) that I find useful, and then using my other batch scripts to call on that "master" file for my snippets.
For example, in my MASTER_BAT.BAT file, an example of a snippet to create dates in different format for usage look like this:
GOTO:%~1
:GET_CURRENT_DATE
:: Created: 1/19/2012
:: Creates variables for the date format in different forms.
:: No additional arguments required
SET DISABLED=0
IF [%DISABLED%] == [1] GOTO:EOF
:: Created: 11/30/11
:: Creates date formats.
Set mdy=%date:~4,2%-%date:~7,2%-%date:~12,4%
Set mdY=%date:~4,2%-%date:~7,2%-%date:~10,4%
Set Dmdy=%date:~0,4%%date:~4,2%-%date:~7,2%-%date:~12,4%
Set DmdY=%date:~0,4%%date:~4,2%-%date:~7,2%-%date:~10,4%
Set ymd=%date:~12,4%-%date:~4,2%-%date:~7,2%
Set ymd=%date:~10,4%-%date:~4,2%-%date:~7,2%
GOTO:EOF
And in my CHILD_BAT.BAT, I want to use that snippet to create the date formats... lets say I want to make it so that I can call the date by the current date in mm/dd/yy format:
CALL MASTER_BAT.BAT "GET_CURRENT_DATE"
ECHO %mdy%
PAUSE
Your output for CHILD_BAT.BAT would be:
1-23-12
Press any key to continue...
Also, any variables created in your CHILD_BAT.BAT prior to the CALL command will be passed to the MASTER_BAT.BAT script as well. However, for loop interation that includes a CALL will not pass the for loop temporary variable.
Hope this is helpful.
EDIT: Note that my snippet is usable for the U.S. date format.