parallel processing in bat file for loop - batch-file

Question

I want a batch file to do parallel processing. I have a storedprocedure which returns 1000+ records(has unique rowid column along with other info). Iterating though each row works fine. However, it takes long time to complete 1000 loops. Is there a way to run two loops parallel without overlapping or having to maintain separate batch files. Can this be accomplished by having one .bat file.
WORKING CODE:
#echo off
setlocal EnableExtensions
set ListFile=%TEMP%\StudentList.tmp
Set varServerPath=http://xyz/ReportServer
sqlcmd -Q "exec dbo.Storedproc_StudentList" -S ServerName -d DatabaseName >"%ListFile%" 2>nul
if exist "%ListFile%" (
for /F "usebackq tokens=1,2,3,4 skip=2 delims=', " %%A in ("%ListFile%") do (
echo Processing StudentID %%A and SubjectID %%B ...
if not exist "%%D" mkdir "%%D"
rs -i C:\ReportRender\Student.rss -s%varServerPath% -e Exec2005 -v StudentID="%%A" -v SubjectID="%%B" -v vOutputFilePath="%%C" -v vReportPath="/Student Reports/ReportName.rdl" -l 900
)
del "%ListFile%"
)
exit
I tried doing something like having two for loops one from 1 to 200 and other from 201 to 400 and so on....but seems like i'm on the wrong track. It doesn't work, Please suggest.
#echo off
setlocal EnableExtensions
set ListFile=%TEMP%\StudentList.tmp
Set varServerPath=http://xyz/ReportServer
sqlcmd -Q "exec dbo.Storedproc_StudentList" -S ServerName -d DatabaseName >"%ListFile%" 2>nul
if exist "%ListFile%" (
for /F "usebackq tokens=1,2,3,4 skip=2 delims=', " %%A in ("%ListFile%") do (
for /L %%A in(1,1,200) do (
echo Processing StudentID %%A and SubjectID %%B ...
if not exist "%%D" mkdir "%%D"
rs -i C:\ReportRender\Student.rss -s%varServerPath% -e Exec2005 -v StudentID="%%A" -v SubjectID="%%B" -v vOutputFilePath="%%C" -v vReportPath="/Student Reports/ReportName.rdl" -l 900
)
for /L %%A in(201,1,400) do (
echo Processing StudentID %%A and SubjectID %%B ...
if not exist "%%D" mkdir "%%D"
rs -i C:\ReportRender\Student.rss -s%varServerPath% -e Exec2005 -v StudentID="%%A" -v SubjectID="%%B" -v vOutputFilePath="%%C" -v vReportPath="/Student Reports/ReportName.rdl" -l 900
)
)
del "%ListFile%"
)
exit
Thanks,

Answer 1

Your approach is wrong. You are executing a for /L %%A in (1,1,200) ... and a for /L %%A in (201,1,400) ... for each record in the %ListFile%. You need to distribute the records in the %ListFile% into the two parallel processes. Although this can be done in groups of 200 records, it is much simpler to do that one-by-one. Also, the only way to have parallel processes in a Batch file is via start command or using a | pipe. In this case you want to distribute several input records that will be read (and processed) by two "output processes", so the pipe approach is simpler.
#echo off
setlocal EnableDelayedExpansion
if "%1" neq "" goto %1
set ListFile=%TEMP%\StudentList.tmp
Set varServerPath=http://xyz/ReportServer
sqlcmd -Q "exec dbo.Storedproc_StudentList" -S ServerName -d DatabaseName >"%ListFile%" 2>nul
if not exist "%ListFile%" exit
set numProcs=2
( "%~F0" Input | "%~F0" Output ) 2>&1 | "%~F0" Output
del "%ListFile%"
exit
:Input
set i=0
for /F "usebackq tokens=1,2,3,4 skip=2 delims=', " %%A in ("%ListFile%") do (
set /A i+=1, proc=i%%numProcs
if !proc! equ 1 (
echo %%A %%B %%C %%D
) else (
>&2 echo %%A %%B %%C %%D
)
)
exit /B
:Output
for /F "tokens=1-4" %%A in ('findstr "^"') do (
echo Processing StudentID %%A and SubjectID %%B ...
if not exist "%%D" mkdir "%%D"
rs -i C:\ReportRender\Student.rss -s%varServerPath% -e Exec2005 -v StudentID="%%A" -v SubjectID="%%B" -v vOutputFilePath="%%C" -v vReportPath="/Student Reports/ReportName.rdl" -l 900
)
exit /B
The :Input part just distribute the "%ListFile%" records to Stdout (channel 1) and Stderr (channel 2), one by one.
The :Output part just take the %%A %%B %%C %%D values sent by :Input part and process they in the usual way; the input data is read from Stdin via findstr command.
The :Input and :Output parts could be in separate Batch files, but they are included in the same file and selected via a parameter and the if "%1" neq "" goto %1 command placed at beginning.
The most interesting code is the pipeline that run the 3 processes in parallel. The :Input part run and its Stdout output is feed into the first :Output process. The Stderr output (channel 2) of :Input part is redirected into Stdin via the 2>&1, so this output is feed into the second :Output process.
This method may also be used for more than two output parallel processes; you just need to add more similar parts changing the number 2 for 3, etc. For example, with three output processes the pipeline should be this one:
( ( "%~F0" Input | "%~F0" Output ) 2>&1 | "%~F0" Output ) 3>&1 | "%~F0" Output
However, it is very important that you note that this method does NOT necessarily imply that the whole process will run faster! This point depends on several factors, like the number of CPU cores and the speed/buffers of the disk drive. Just a test can answer this question...
Post the result, please.