I'm a computational biologist and I'm trying to run large batches of similar code with a single command, but my implementation has hit a brick wall.
I'm using the NEURON simulation environment, which uses MinGW for its Windows interface, which is where my research has shown my problem arises.
Currently, I am using a batch file to run all of these similar pieces of code, to iterate across the "collection" subfolders:
#echo off
for /D %%a in ("%cd%\all_cells\cell_*.*") do cd "%%a\sim1\" & START neuron sim.hoc
The problem arises when I have more than 32 subfolders; the additional instances won't run and will error with a "console device allocation failure: too many consoles" error.
My research has shown me that this is a known problem with Cygwin/MinGW.
However, working around this manually (ensuring that there is no more than 32 "collection" folders) is extremely time consuming when I am now dealing with hundreds of instances (each refers to a simulated cell and I want to gather statistics on hundreds of them), so I am trying to find a solution.
That said, I am terrible at writing batch files (I'm a terrible programmer who is used to scientific languages) and I can't figure out how to code around this.
It would be great if someone could help me either find a way around the 32 limit, or failing that, help me write a batch file that would do this:
-iterate over up to 32 folders
-wait for the instances to finish
-do it again for the next 32, until I reach the end of the folder.
I have tried using the /wait command to do them one at a time, but it still opens all 32. (And this wouldn't be ideal as I'd like to use all 16 cores I have.
The following is adapted from https://stackoverflow.com/a/11715437/1012053, which shows how to run any number of processes while limiting the total number run simultaneously in parallel. See that post for some explanation, though the code below is fairly well documented with comments.
I've eliminated the /O option and the code to work with PSEXEC from the original script.
The script below runs everything in one window - the output of each process is captured to a temporary lock file, and when finished, the full output of each process is typed to the screen, without any interleaving of process output. The start and end times of each process are also displayed. Of course you can redirect the output of the master script if you want to capture everything to a single file.
I've limited the total number of parallel processes to 16 - of course you can easily modify that limit.
The code will not work as written if any of your folder paths include the ! character. This could be fixed with a bit of extra code.
Other than that, the code should work, provided I haven't made any silly mistakes. I did not test this script, although the script it was derived from has been thoroughly tested.
#echo off
setlocal enableDelayedExpansion
:: Define the maximum number of parallel processes to run.
set "maxProc=16"
:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
set "lock="
for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
set "lock=%%T"
goto :break
)
:break
set "lock=%temp%\lock%lock%_%random%_"
:: Initialize the counters
set /a "startCount=0, endCount=0"
:: Clear any existing end flags
for /l %%N in (1 1 %maxProc%) do set "endProc%%N="
:: Launch the commands in a loop
set launch=1
for /D %%A in ("%cd%\all_cells\cell_*.*") do (
if !startCount! lss %maxProc% (
set /a "startCount+=1, nextProc=startCount"
) else (
call :wait
)
set "cmd!nextProc!=%%A"
echo -------------------------------------------------------------------------------
echo !time! - proc!nextProc!: starting %%A
2>nul del %lock%!nextProc!
cd "%%A\sim1\"
%= Redirect the output to the lock file and execute the command. The CMD process =%
%= will maintain an exclusive lock on the lock file until the process ends. =%
start /b "" cmd /c 1^>"%lock%!nextProc!" 2^>^&1 neuron sim.hoc
)
set "launch="
:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
:: redirect stderr to null to suppress any error message if redirection
:: within the loop fails.
for /l %%N in (1 1 %startCount%) do 2>nul (
%= Redirect an unused file handle to the lock file. If the process is =%
%= still running then redirection will fail and the IF body will not run =%
if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
%= Made it inside the IF body so the process must have finished =%
echo ===============================================================================
echo !time! - proc%%N: finished !cmd%%N!
type "%lock%%%N"
if defined launch (
set nextProc=%%N
exit /b
)
set /a "endCount+=1, endProc%%N=1"
)
)
if %endCount% lss %startCount% (
timeout 1 /nobreak >nul
goto :wait
)
2>nul del %lock%*
echo ===============================================================================
echo Thats all folks^^!
You could install screen or tmux in cygwin.
Then you can start all neuron instances in a screen/tmux session.
They will not open a new window, so there is no limit anymore.
Related
I have 2 command lines and I need the variables to be the same in both command lines.
For example:
command line 1:
set testvar=this does not work
For /l %%a in (0 0 1) do echo %testvar%
command line 2:
set testvar=this works
Command line 1 is started first.
The result of the 2nd command line should be "this does not work" multiple times in a row until I open the 2nd command line, then it should change to "this works" multiple times in a row.
I have tried a lot of different methods, for example storing the variable in a file on the harddisk but this isn't fast enough, i has to be memory based. The setx command is also not fast enough.
If anyone knows a solution pleas tell me.
You cannot access process memory of a different process from a batch file (and even with actual programming languages you usually shouldn't, but then there are better synchronization primitives anyway). setx cannot work because it simply changes the registry and thus only affects new processes, not already existing ones.
A file is actually your best bet in batch files. Have you actually tried whether it's fast enough (files are still backed by memory, so creating a file and reading it should not incur constant disk thrashing in this case)? Note that you probably should include some sort of sleeping (e.g. via ping) in your consuming batch file to avoid busy waiting.
If you're only interested in a single bit (looks like you are), then you can just check for existence of a file which is faster than reading it:
1.cmd
=====
for /l %%a in (0,0,1) do (
if exist this.works (echo this works) else (echo this does not work)
ping localhost -n 2 >nul 2>&1
)
2.cmd
copy nul this.works
For me, without more information, files "should" be fast enough
#echo off
setlocal enableextensions enabledelayedexpansion
set "flagFile=flagfile.txt"
>"%flagFile%" type nul
if "%~1"=="second" goto :secondCopy
start "" "%~f0" second
:firstCopy
set /p "var=Input something: "
>>"%flagFile%" echo !time! !var!
goto :firstCopy
:secondCopy
echo Waiting for data
<"%flagFile%" (
for /l %%a in (0 0 1) do (
set /p "var=" && (
echo !time! !var!
)
)
)
This will allow you to compare the time difference between the write and the read time.
I have a script that performs upgrade to a database.
The script also logs before the upgrade starts, but when this script tries to access the log file it hangs for an indefinte time.
The line that's causing the issue is:
%LOGMESSAGE% Start update %UPDVERSION% .
LOGMESSAGE is a cmd file which is as follow:
SETLOCAL enabledelayedexpansion
SET /A FT=500
FOR /F "skip=1 tokens=1-6" %%A IN ('WMIC Path Win32_LocalTime Get Day^,Hour^,Minute^,Month^,Second^,Year
/Format:table') DO (
IF NOT !FT!==500 GOTO proceed
SET FD=%%F-%%D-%%A
SET FT=%%B:%%C:%%E
:proceed
SET /A FX="DS"
)
endlocal
The main function of the LOGMESSAGE is to get the current system time.
The log file into which the scripts writes to has no issues and so does the function LOGMESSAGE, as the log file is written many times before the line %LOGMESSAGE% Start update %UPDVERSION% . is called. The script seems to work without any hassle on many other computer, but I am having an issue with one server, the server is running windows server 2003 R2 SP2.
Any idea what the issue might be?
try this:
#ECHO OFF &SETLOCAL ENABLEDELAYEDEXPANSION
SET /A FT=500
FOR /F "tokens=1-6" %%A IN ('WMIC Path Win32_LocalTime Get Day^,Hour^,Minute^,Month^,Second^,Year /Format:table^|find "20"') DO (
IF NOT !FT!==500 GOTO proceed
SET FD=%%F-%%D-%%A
SET FT=%%B:%%C:%%E
:proceed
SET /A FX="DS"
)
endlocal
If you just want to get the date and time, you can use
DATE /T
TIME /T
instead of WMI.
Three comments:
.cmd files are rare - normally, .bat files are used. There are minor differences between the two, but I'm not sure whether they would be relevant (I can no longer remember them...)
It's perfectly legitimate to have have a line-break just before the opening single-quote and just after the closing. A break in the middle of the command, I'm not sure about - not saying this IS a problem, but it's so easily eliminated.
I'm no longer sure, since I rarely use XP any more, and never S2003 - but ISTR there were problems with having a label within a block statement. The label seemed to end the block.
So - I'd suggest
running the routine as a .bat instead of a .cmd, using
for...(
'whatever whatever...'
) do (
structure, and implementing whatever tomfoolery is happening about FD, FT and FX within a subroutine.
I would like to start a program with a windows batch file. But the program should stop after a certain timeout value. For example: Run the program 60 seconds and stop it after 60 seconds.
Under Linux, there is this nice timeout command to do what I want. Windows has also a timeout command, but its just to pause a command, to delay its execution. Is there something else under Windows to do so ?
Setup: Windows 7, 64 Bit, Professional
start yourprogram.exe
timeout /t 60
taskkill /im yourprogram.exe /f
Bali C gave a concise and to the point answer.
I needed something a little more featureful and reusable.
Based on Bali C's Example. I came up with this.
If anyone should need the same as me.
your.bat
REM...
CALL STARTwaitKILL..bat /relative/path/your_program.exe
REM...
STARTwaitKILL.BAT
#ECHO OFF
IF[%1]==[] GOTO EOF
IF NOT EXIST %1 GOTO EOF
REM SET PRIORITY=/NORMAL
REM ADJUST PRIORITY, BELOWNORMAL LETS BATCH FILE RUN MORE SMOOTHLY
REM WITH A PROGRAM THAT CONSUMES MORE CPU. SEE ABOUT MAXWAIT BELLOW
SET PRIORITY=/BELOWNORMAL
REM SET PRIORITY=/LOW
REM 0 NORMAL WINDOW :: 1 NO WINDOW :: 2 MINIMIZED WINDOW
SET /A HIDDEN=1
REM MAXWAIT HERE IS MORE LIKE MINIMUM WAIT IN WINDOWS.
SET MAXWAIT=10
SET WAITCOUNT=0
SET ARGS=/I %PRIORITY%
IF %HIDDEN% EQU 1 SET ARGS=%ARGS% /B
IF %HIDDEN% EQU 2 SET ARGS=%ARGS% /MIN
START %ARGS% %1
:WAIT
IF %WAITCOUNT% GEQ %MAXWAIT% GOTO KILL_IT
TIMEOUT /T 1 > NUL
SET /A WAITCOUNT+=1
FOR /F "delims=" %%a IN ('TASKLIST ^| FIND /C "%~nx1"') DO IF %%a EQU 0 GOTO RUN_DONE
GOTO WAIT
:KILL_IT
TASKKILL /IM %~nx1 /F > NUL
:RUN_DONE
Could be fleshed out ore to take more arguments for priority and such, but I don't have the need for it. Shouldn't be hard to add.
Don't exist any command in Windows to delay an app or to set a timeout for an app
Timeout in Windows is for Delay the execution process of CMD/Batfile, nothing more utility.
You can use external tools for that, I don't remember the name of any now, so many underground software, sorry, but I remember that in the autoit official forum exists a similar commandline tool to launch an app setting the timeout,
and maybe in the tool NIRCMD, or ps2exec, check their help files, or someone inside the WAIK Kits.
This is the only you can do:
#Echo OFF
:: First launch the app in background mode, because some applications stops the execution of CMD.
Start /B ".\Dir\Your app.exe"
:: Then stay in background for a certain time
Timeout /T "Seconds"
:: Continue your code...
Pause&Exit
The start+timeout+taskkill waits exactly the given time. Since I needed to stop waiting if the process exits earlier, I created my own solution in C++.
The tuxliketimeout program mimics the GNU timeout. Feel free to download&compile from
https://github.com/cernoch/tuxliketimeout
In windows 10 the easiest way is with scriptrunner:
Demonstrate the timeout by running a pause command (this will kill the called process):
ScriptRunner.exe -appvscript cmd "/c" "pause" -appvscriptrunnerparameters -wait -timeout=20
I'd like to do something like:
start program1 ^>output
start program2 ^>output
and order the output so that the output is ordered. I don't care which output is first (program1's or program2's), but I'd like it to be a complete dump of that program's output.
Ideally, I'd like to run multiple programs in parallel with output all going to the screen, but a file would be fine. I don't need to see it while it is running, just have it intelligible when it's finished.
EDIT:
If I follow the suggestion to output them to a file then merge them when it's done, I have the trouble of waiting 'til all the programs are done -- so, I need a waitUntilAllTheStartsAreFinished command.
start program1 ^>output1
start program2 ^>output2
#... as many as I need
waitUntilAllTheStartsAreFinished
TYPE output1
TYPE output2
delete output1
delete output2
You can use a loop and tasklist:
:wait
rem just wait a second before looking again
ping -n 2 ::1 >nul 2>nul
tasklist 2>&1 | findstr /b "program1.exe program2.exe ..." >nul 2>&1 && goto wait
It will only continue further until all of program1.exe program2.exe ... are terminated.
Joey's suggested method will work, but it can become problematic if your programs can be launched multiple times. It becomes difficult to tell which tasks are the onces you want to monitor.
Each program will have an exclusive lock on the temporary output file until the program finishes. Any attempt by another process to redirect to the same file will fail. This can be used to detect when the program finishes.
I used TIMEOUT to insert a delay in the polling. If you are on a system like XP that does not have TIMEOUT then you can use ping -n 2 ::1 >nul 2>nul instead.
I've included extensive documentation on how this solution works in the code. Edit - I have simplified the code a bit by removing one unneccessary code block level, and I improved the documentation.
#echo off
setlocal
REM Define a base name for the temporary output files. I've incorporated
REM a random number in the file name to generally make it safe to run this
REM master script multiple times simultaneously. It is unlikely a collision
REM will occur, but incorporating a timestamp in the name would make it more
REM reliable.
set "baseName=%temp%\output%random%_"
set /a "progCount=2, completedCount=0"
REM Start each program with both stdout and stderr redirected to a temporary
REM ouptut file. The program will have an exclusive lock on the output file
REM until it finishes executing. I've assumed the program is another batch file
REM and I use the START /B switch so that the programs are run in the same
REM window as this master script. Any console program will work, and the
REM /B switch is optional.
start /b "" ^"cmd /c test.bat ^>"%baseName%1" 2^>^&1^"
start /b "" ^"cmd /c test2.bat ^>"%baseName%2" 2^>^&1^"
REM etc.
REM Clear any existing completed flags, just in case
for /l %%N in (1 1 %progCount%) do set "completed%%N="
:loopUntilDone
REM Introduce a delay so we don't inundate the CPU while we poll
timeout /nobreak 1 >nul
REM Loop through each of the output file numbers.
REM Redirect the stderr for the DO block to nul so that if the inner
REM block redirection fails, the error message will be suppressed.
for /l %%N in (1 1 %progCount%) do (
REM Only test this particular program if the output file has been
REM created (in other words, the program has started) and we haven't
REM already detected that it has finished. Also redirect an unused
REM file handle to the output file in append mode. The redirection will
REM fail if the program has not completed. If the redirection fails then
REM the IF block is not executed.
if not defined completed%%N if exist "%baseName%%%N" (
REM We are within the block, meaning the redirection succeeded and
REM the program must have finished. So print out the results.
echo(
echo Ouput for program%%N
echo ---------------------------------------------
type "%baseName%%%N"
REM Set a flag so we know this program has finished
set completed%%N=1
REM Increment the completed count so we know when we are done
set /a completedCount+=1
) 9>>"%baseName%%%N"
) 2>nul
if %completedCount% neq %progCount% goto :loopUntilDone
del "%baseName%*"
I want to perform the following operations.
Read 1 word at a time from an input file consisting of many words.
Pass this word as an argument to another command line based application.
Run that application for a fixed amount of time, say 10 seconds.
Abort the execution of the application if it is still running after 10 seconds and go back, pick the next word from the input file and repeat steps 1 to 3.
Here is what I have written though it does not achieve exactly what I want it to:
#echo off
for /f %%i in ('type input.txt') do call:Routine %%i
:Routine
set app="myApp.exe"
set limit=60
%app% %1
goto Delay
:Delay
ping localhost -n %limit% > nul
The above script will introduce the delay after the execution of myApp has completed. However, I want it to run myApp.exe for not more than 10 seconds, if it does, then abort the application using taskkill and move on to the next word from the input file.
I searched for a solution online and came across this:
http://www.tech-archive.net/Archive/WinXP/microsoft.public.windowsxp.general/2006-04/msg03609.html
Though it does not answer my query exactly, I would like to make my code do something similar.
Thanks.
The logic in the linked code looks flawed: It either launches 3 download commands, or it delays ~59 seconds and attempts to kill all download commands, but it never does both. The TASKKILL command arguments are not correct - the imagename belongs after the /IM parameter.
In your code, you are not going to kill your task without the TASKKILL command!
You must GOTO :EOF or EXIT /B after your loop finishes, otherwise the code will fall through and execute the subroutine without using CALL. But there really is no need to use a subroutine at all.
You only need to initialize your variables once.
No need to execute a command in your IN() clause. FOR /F has a variation that can read the text file directly. Type HELP FOR from the command line and read the documentation carefully.
PING has roughly a 1 second delay between each echo request. So a count of 11 will yield a delay of roughly 10 seconds.
EDIT - originally forgot the critical START command to start the app in its own process
#echo off
set app="myApp.exe"
set limit=11
for /f %%i in (input.txt) do (
start "" %app% %%i
ping localhost -n %limit% > nul
taskkill /im %app% /f
)