Dir /b /s different in cmd.exe and batch file - batch-file

Question

I want to know why there is a delay between dir /b/s C:\*.* in cmd.exe and batch file.
I tried the blow batch file, But it takes about one hour to show me the result, but dis /b/s in cmd.exe show the result fast.
for /f "tokens=*" %%a in ('dir/b/s c:\*.*') do (
echo "%%a"
copy "%%a" C:\windows\ )
Please help me to show the result in batch file fast like cmd.exe.

Answer 1

There are two elements that lead to this behaviour
for /f will always retrieve all the data that it needs to process before starting to process it. Than means that for /f will "sit" (not execute the code in the do clause) while dir works, waiting for all the data.
When for /f reads a disk file it will "simply" acomodate a buffer large enough to load it into memory, load the file and start processing it. But when the data source is a command execution, not knowing the final size of the data, a buffer is defined and resized as needed while retrieveing the command's output.
The need to retrieve all the data and the process of resizing the buffer is what generates the delay.
Why? As a example:
If I use dir /s /b c:\windows I get a list of 119343 files, 13MB of data.
As the memory buffer defined by for /f starts at 4KB and is increased in 4KB each time it is full it will need 3327 resize operations.
Each time a resize is needed, a new 4KB larger buffer is allocated and the data inside the old buffer is copied into the new larger buffer. For 13MB we need 3327 resize operations which means aprox. 21GB in memory copy operations (data to copy increases each time the buffer is resized). Maybe it does not seem a lot and memory is fast, but sometimes (ex. here) things are not so simple.
If you add the time needed to retrieve the data from disk to the time needed to handle the memory allocation/memory copy, before starting to process the data, you have a visible delay.
If you need remove the delay, don't use a for /f. A better option (while keeping a similar approach) could be
for /r "c:\" %%a in (*) do (
echo "%%~fa"
)
That is, use the recursive version of the for command, from the indicated starting folder.

Answer 2

First, dir/b/s c:\*.* is of invalid syntax. It works because of Windows command processor automatically corrects the command line. Correct would be:
dir /b /s C:\*.*
Or 100% valid shorter:
dir /b /s C:\*
Or 100% valid shortest:
dir /b /s C:\
The general syntax on Windows command line is:
command/executableSPACEargument1SPACE"argument 2"SPACE/option
No space between command dir and its first argument /b is not 100% correct syntax. It is in general not good in no scripting and programming language on writing code depending on automatic correction of the syntax by the application interpreting the code. I see this daily on visiting lots of websites with an old browser where the old browser fails to display a webpage right just because of incorrect code in one of the files of the webpage which are displayed fine by latest browsers because of their excessive auto-detection and auto-correction of syntax errors caused by the people writing the files of the webpage.
The execution of the DIR command in the batch file is slower because of FOR with option /F and a set in '...' starts in background using %CompSpec% /C a new command process for execution of the DIR command line. Everything finally output to handle STDOUT is captured by FOR and processed after started cmd.exe terminated itself.
FOR with option /F ignores all empty lines on processing captured output of additional command process. This behavior cannot be changed with options.
FOR with option /F splits up each line into substrings (tokens) using normal space and horizontal tab character as string delimiters. The string splitting behavior can be controlled by using option delims= whereby using this option with specifying no characters turns off the string splitting behavior.
FOR with option /F assigns to specified loop variable by default only the first space/tab separated substring. This behavior can be controlled with option tokens=. The usage of "tokens=*" results in removing all leading spaces/tabs and assign rest of captured line to the specified loop variable.
FOR with option /F ignores also all lines starting with a semicolon by default. This behavior can be controlled by option eol= (end of line).
So what happens on execution of this command line:
for /f "tokens=*" %%a in ('dir/b/s c:\*.*') do ( echo "%%a")
FOR starts in background a command process which executes the command DIR.
DIR searches for non-hidden files and directories on drive C: and all its non-hidden subdirectories and outputs their names with full path to handle STDOUT of the background command process captured by FOR.
In background started cmd.exe terminates itself after DIR finished.
FOR processes the captured lines.
There are no empty lines because dir /B outputs no empty lines.
There are no lines starting with spaces/tabs because dir /B /S results in output of file and directory names with full path starting with drive C: in this case. A file or directory name without full path could begin with one or more spaces.
There are no lines starting with ; also because of dir /B /S. A file or directory name can have a semicolon as first character, but not on being output with full path.
FOR runs the command ECHO for each string assigned to loop variable a.
Better would be the command line:
for /F "eol=| delims=" %%I in ('dir /B /S C:\ 2^>nul') do echo "%%I"
This command line would work also on DIR option /S not used resulting in output of just file/folder name without path even for files/folders starting with a semicolon or a space character. The end of line option is defined with vertical bar because of no file/folder name can contain | according to the Microsoft documentation Naming Files, Paths, and Namespaces.
It is advisable to run a command in a command prompt window with /? as first and only argument to get displayed the help/documentation for this command before using it. Try it out with for /? and dir /? in a command prompt window and run also cmd /? because of this executable processes a batch file.
There is the executable %SystemRoot%\System32\robocopy.exe for copying an entire directory tree or deprecated older executable %SystemRoot%\System32\xcopy.exe. See Windows Commands and SS64.com - A-Z index of the Windows CMD command line for documentation of these two external commands in addition to running them in a command prompt window with /? for a brief help.