Slow processing a for loop that utilizes findstr - batch-file

Question

I've got a somewhat weird case, where a for-loop is incredibly slow when I use findstr as the string for DO.
Its worth mentioning that the file (old-file.xml) that I'm processing contains about 200 000 lines.
This part is blazing fast, but can be rendered slower if I remove | find /c ":"
rem find total number of lines in xml-file
findstr /n ^^ old-file.xml | find /c ":" > "temp-count.txt"
set /p lines=< "temp-count.txt"
The code which is slow looks like this and I can't use the pipe trick above. It seems like the slow part is the for itself, as i'm not seeing any progress in the title bar until after 10 min.
setlocal DisableDelayedExpansion
rem start replacing wrong dates with correct date
for /f "usebackq Tokens=1* Delims=:" %%i in (`"findstr /n ^^ old-file.xml"`) do (
rem cache the value of each line in a variable
set read-line=%%j
set line=%%i
rem restore delayed expansion
setlocal EnableDelayedExpansion
rem write progress in title bar
title Processing line: !line!/%lines%
rem remove trailing line number
rem set read-line=!read-line:*:=!
for /f "usebackq" %%i in ("%tmpfile%") do (
rem replace all wrong dates with correct dates
set read-line=!read-line:%%i=%correctdate%!
)
rem write results to new file
echo(!read-line!>>"Updated-file.xml"
rem end local
endlocal
)
EDIT:
Further investigation showed me that using this single line that should display the current line number being looped takes about 10 minutes on my 8MB file of 200 000 lines. That's just for getting it to start displaying the lines.
for /f "usebackq Tokens=1* Delims=:" %%i in (`"findstr /n ^^ old-file.xml"`) do echo %%i
So it seems like findstr is writing screen output hidden for the user, but visible for the for-loop. How can I prevent that from happening while still getting the same results?
EDIT 2: Solution
The solution as proposed by Aacini and finally revised by me.
This is a snippet from a much bigger script. Wrong dates are retrieved in another loop. And total number of lines are also retrieved from another loop.
setlocal enabledelayedexpansion
rem this part is for snippet only, dates are generated from another loop in final script
echo 2069-04-29 > dates-tmp.txt
echo 2069-04-30 >> dates-tmp.txt
findstr /n ^^ Super-Large-File.xml > out.tmp
set tmpfile=dates-tmp.txt
set correctdate=2011-11-25
set wrong-dates=
rem hardcoded total number of lines
set lines=186442
for /F %%i in (%tmpfile%) do (
set wrong-dates=!wrong-dates! %%i
)
rem process each line in out.tmp and loop them through :ProcessLines
call :ProcessLines < out.tmp
rem when finished with above call for each line in out.tmp, goto exit
goto ProcessLinesEnd
:ProcessLines
for /L %%l in (1,1,%lines%) do (
set /P read-line=
rem write progress in title bar
title Processing line: %%l/%lines%
for %%i in (%wrong-dates%) do (
rem replace all wrong dates with correct dates
set read-line=!read-line:%%i=%correctdate%!
)
rem write results to new file
echo(!read-line:*:=!>>"out2.tmp"
)
rem end here and continue below
goto :eof
:ProcessLinesEnd
echo this should not be printed until call has ended
:exit
exit /b

Answer 1

Two points here:
1- The setlocal EnableDelayedExpansion command is executed with every line of the file. This means that about 200000 times the complete environment must be copied to a new local memory area. This may cause several problems.
2- I suggest you to start with the most basic part. How much time takes the findstr to execute? Run findstr /n ^^ old-file.xml alone and check this before trying to fix any other part. If this process is fast, then add a single step to it and test again until you discover the cause of the slow down. I suggest you not use pipes nor for /f over the execution of findstr, but over the file generated by a previous redirection.
EDIT A faster solution
There is another way to do this. You may pipe findstr output into a Batch subroutine, so the lines can be read with SET /P command. This method allows to process the lines entirely via delayed expansions and not via the command-line susbtitution of FOR /F, so the pair of setlocal EnableDelayedExpansion and endlocal commands are no longer necessary. However, if you still want to display the line number it is necessary to calculate it again.
Also, it is faster to load the wrong dates in a variable instead of process the %tmpfile% with every line of the big file.
setlocal EnableDelayedExpansion
rem load wrong dates from tmpfile
set wrong-dates=
for /F %%i in (%tmpfile%) do (
set wrong-dates=!wrong-dates! %%i
)
echo creating findstr output, please wait...
findstr /n ^^ old-file.xml > findstr.txt
echo :EOF>> findstr.txt
rem start replacing wrong dates with correct date
call :ProcessLines < findstr.txt
goto :eof
.
:ProcessLines
set line=0
:read-next-line
set /P read-line=
rem check if the input file ends
if !read-line! == :EOF goto :eof
rem write progress in title bar
set /A line+=1
title Processing line: %line%/%lines%
for %%i in (%wrong-dates%) do (
rem replace all wrong dates with correct dates
set read-line=!read-line:%%i=%correctdate%!
)
rem write results to new file
echo(!read-line:*:=!>>"Updated-file.xml"
rem go back for next line
goto read-next-line
SECOND EDIT An even faster modification
Previous method may be slighlty speeded up if the loop is achieved via for /L command instead of via a goto.
:ProcessLines
for /L %%l in (1,1,%lines%) do (
set /P read-line=
rem write progress in title bar
title Processing line: %%l/%lines%
for %%i in (%wrong-dates%) do (
rem replace all wrong dates with correct dates
set read-line=!read-line:%%i=%correctdate%!
)
rem write results to new file
echo(!read-line:*:=!>>"Updated-file.xml"
)
This modification also omit the :EOF comparison and the calculation of line number, so the time gain may be significative after repeated it 200000 times. If you use this method, don't forget to delete the echo :EOF>> findstr.txt line in first part.

Answer 2

A FOR /F expression will always executed/read/evaluated complete before the inner loop starts.
You can try it with
(
echo line1
echo line2
) > myFile.txt
FOR /F "delims=" %%a in (myFile.txt) DO (
echo %%a
del myFile.txt 2> nul >nul
)
It will display
line1
line2
In your case the complete ('"findstr /n ^^ old-file.xml"') will executed and cached before the loop can start
EDIT: Added Solution
I measured with a file ~20MB with 370.000 lines
type testFile.txt > nul
findstr /n ^^ testFile.txt > nul
for /F "delims=" %%a in (testFile.txt) do (
rem Nothing
)
for /f "usebackq delims=" %%a in (`"findstr /n ^^ testFile.txt"`) do ...
findstr /n ^^ testFile.txt > out.tmp
type_nul ~10000ms
findstr_nul ~30000ms
for_file ~ 1600ms
for_findstr cancled after 10 minutes
findstr_tmp ~ 500ms !!!
I would recommend to use a temporary file, it's extremly fast.
findstr /n ^^ myFile.txt > out.tmp
set lineNr=0
(
for /f "usebackq delims=" %%a in ("out.tmp") do (
set /a lineNr+=1
set "num_line=%%a"
setlocal EnableDelayedExpansion
set "line=!num_line:*:=!"
echo(!line!
endlocal
)
) > out2.tmp
Btw. Your for/F splitting can fail, if the original line begins with a colon
for /f "usebackq Tokens=1* Delims=:"
Sample: :ThisIsALabel
:ThisIsALabel
Findstr /n prepends a line number
17::ThisIsALabel
the delims=: will split the first token and handles all colons as only one seperator
ThisIsALabel