Perfectly forwarding arguments in batch - batch-file

Question

I have a small python script:
# args.py
import sys; print(sys.argv)
How can I write a .bat wrapper file that forwards all of the arguments to this script?
To eliminate my shell from the tests, I'm going to invoke it as:
import subprocess
import sys
def test_bat(*args):
return subprocess.check_output(['args.bat'] + list(args), encoding='ascii')
The obvious choice of batch file
#echo off
python args.py %*
Works for simple cases:
>>> test_bat('a', 'b', 'c')
"['args.py', 'a', 'b', 'c']\n"
>>> test_bat('a', 'b c')
"['args.py', 'a', 'b c']\n"
But rapidly falls apart when tried on arbitrary strings:
>>> test_bat('a b', 'c\n d')
"['args.py', 'a b', 'c']\n" # missing d
>>> test_bat('a', 'b^^^^^c')
"['args.py', 'a', 'b^c']\n" # missing ^^^^
Is it even possible to make a bat file pass on its arguments unmodified?
To prove it's not subprocess causing the issue - try running the above with
def test_py(*args):
return subprocess.check_output([sys.executable, 'args.py'] + list(args), encoding='ascii')
All of the tests behave as expected
Similar questions:
Get list of passed arguments in Windows batch script (.bat) - does not address lossless forwarding
Redirecting passed arguments to a windows batch file - addresses the same ideas as my question, but incorrectly closed as a duplicate of the above, and with less clear test-cases
Forwarding all batch file parameters to inner command - question does not consider corner-cases, accepted answer does not work for them

Answer 1

In short: There is no robust way to pass arguments through as-is via a batch file, because of how cmd.exe interprets arguments; note that cmd.exe is invariably involved, given that it is the interpreter needed to execute batch files, even if you invoke the batch file using an API that requests no shell involvement.
The problem in a nutshell:
On Windows, invoking an external program requires use of a command line as a single string for technical reasons. Therefore, even using an array-based, shell-free way of invoking an external program requires automated composition of a command line in which the individual arguments are embedded.
E.g., Python's subprocess.check_output() accepts the target executable and its arguments individually, as the elements of an array, as demonstrated in the question.
The target executable is invoked directly, using a command line that was automatically composed behind the scenes, without using the platform's shell as an intermediary (the way that Python's os.system() call does, for instance) - unless it so happens the target executable itself requires that shell as the executing interpreter, as is the case with cmd.exe for batch files.
Composing the command line requires selective double-quoting and escaping of embedded " chars. when embedding the individual arguments; typically that involves:
Using enclosing double-quoting ("..."), but only around arguments that contain whitespace (spaces).
Escaping embedded double quotes as \"
Notably, no other characters trigger double-quoting or individual escaping, even though those characters may have special meaning to a given shell.
While this approach works well with most external programs, it does NOT work reliably with batch files:
Unfortunately, cmd.exe doesn't treat the arguments as literals, but interprets them as if you had submitted the batch-file call in an interactive console (Command Prompt).
Combined with how the command line is composed (as described above), this results in many ways that the arguments can be misinterpreted and break the invocation altogether.
The primary problem is that arguments that end up unquoted in the command line that cmd.exe sees may break the invocation, namely if they contain characters such as & , |, > or <.
Even if the invocation doesn't break, characters such as ^ may get misinterpreted.
See below for specific examples of problematic arguments.
Trying to work around the problem on the calling side with embedded quoting - e.g., using '"^^^^^" as an argument in Python - does not work, because most languages, including Python, use \" to escape " characters behind the scenes, which cmd.exe does not recognize (it only recognizes "").
Hypothetically, you could painstakingly ^-escape individual characters in whitespace-free arguments, but not only is that quite cumbersome, it still wouldn't address all issues - see below.
Jeb's answer commendably addresses some of these issues inside the batch file, but it is quite complex and it too cannot address all issues - see next point.
There is no way to work around the following fundamental restrictions:
cmd.exe fundamentally cannot handle arguments with embedded newlines (line breaks):
Parsing the argument list simply stops at the first newline encountered.
CR (0xD) chars. in isolation are quietly removed.
The interpretation of % as part of an environment-variable reference (e.g, %OS%) cannot be suppressed:
%% does NOT help, because, curiously and unfortunately, the parsing rules of an interactive cmd.exe session apply(!), where the only way to suppress expansion is to employ the "variable-name disrupter trick", e.g., %^OS%, which only works in unquoted arguments - in double-quoted arguments, you fundamentally cannot prevent expansion.
You're lucky if the env. variable happens not to exist; the token is then left alone (e.g., %NoSuchVar% or %No Such Var% (note that cmd.exe does support variable names with spaces).
Examples of whitespace-free arguments that either break batch-file invocation or result in unwanted alteration of the value:
^^^^^
^ in unquoted strings is cmd.exe's escape character that escapes the next character, i.e., treats it as a literal; ^^ therefore represents a literal, single ^, so the above yields ^^, with the last ^ getting discarded
a|b
| separates commands in a pipeline, so cmd.exe will attempt to pipe the part of the command line before | to a command named b and the invocation will most likely break or, perhaps worse, will not work as intended and execute something it shouldn't.
To make this work, you'd need to define the argument as 'a^^^|b' (sic) on the Python side.
Note that a & b would not be affected, because the embedded whitespace would trigger double-quoting on the Python side, and use of & inside "..." is safe.
Other characters that pose similar problems are & < >

Answer 2

Interessting question, but it's tricky.
The main problem is, that %* can't be used here, as it modifies the content or completely fails dependent of the content.
To get the unmodified argv, you should use a technic like Get list of passed arguments in Windows batch script (.bat).
#echo off
SETLOCAL DisableDelayedExpansion
SETLOCAL
for %%a in (1) do (
set "prompt=$_"
echo on
for %%b in (1) do rem * #%*#
#echo off
) > argv.txt
ENDLOCAL
for /F "delims=" %%L in (argv.txt) do (
set "argv=%%L"
)
SETLOCAL EnableDelayedExpansion
set "argv=!argv:*#=!"
set "argv=!argv:~0,-2!"
REM argv now contains the unmodified content of %* .
c:\dev\Python35-32\python.exe args.py !argv!
This can be used to build a wrapper with limitations.
Carriage returns can't be fetched at all.
Line feeds currently can not be fetched in a safe way