How to receive even the strangest command line parameters? - batch-file

as discussed in an other thread How to avoid cmd.exe interpreting shell special characters like < > ^
it is not easy to get all parameters from the command line.
A simple
set var=%1
set "var=%~1"
are not enough, if you have a request like
myBatch.bat abc"&"^&def
I have one solution, but it needs a temporary file, and it is also not bullet proof.
#echo off
setlocal DisableDelayedExpansion
set "prompt=X"
(
#echo on
for %%a in (4) do (
rem #%1#
)
) > XY.txt
#echo off
for /F "delims=" %%a in (xy.txt) DO (
set "param=%%a"
)
setlocal EnableDelayedExpansion
set param=!param:~7,-4!
echo param='!param!'
It fails with something like myBatch.bat %a, it display 4 not the %a
in this situation a simple echo %1 would work.
It's obviously the for-loop but I don't know how to change this.
Perhaps there exists another simple solution.
I don't need this to solve an actual problem, but I like solutions that are bullet proof in each situation, not only in the most cases.

I don't think anyone found any holes in this, except for the inability to read newlines in the parameters:
#echo off
setlocal enableDelayedExpansion
set argCnt=1
:getArgs
>"%temp%\getArg.txt" <"%temp%\getArg.txt" (
setlocal disableExtensions
set prompt=#
echo on
for %%a in (%%a) do rem . %1.
echo off
endlocal
set /p "arg%argCnt%="
set /p "arg%argCnt%="
set "arg%argCnt%=!arg%argCnt%:~7,-2!"
if defined arg%argCnt% (
set /a argCnt+=1
shift /1
goto :getArgs
) else set /a argCnt-=1
)
del "%temp%\getArg.txt"
set arg
The above comes from a lively DosTips discussion - http://www.dostips.com/forum/viewtopic.php?p=13002#p13002. DosTips user Liviu came up with the critical SETLOCAL DisableExtensions piece.

The code below is based on the rambling Foolproof Counting of Arguments topic on DosTips and this answer by jeb:
#echo off & setLocal enableExtensions disableDelayedExpansion
(call;) %= sets errorLevel to 0 =%
:: initialise variables
set "paramC=0" & set "pFile=%tmp%\param.tmp"
:loop - the main loop
:: inc param counter and reset var storing nth param
set /a paramC+=1 & set "pN="
:: ECHO is turned on, %1 is expanded inside REM, GOTO jumps over REM,
:: and the output is redirected to param file
for %%A in (%%A) do (
setLocal disableExtensions
set prompt=#
echo on
for %%B in (%%B) do (
#goto skip
rem # %1 #
) %= for B =%
:skip - do not re-use this label
#echo off
endLocal
) >"%pFile%" %= for A =%
:: count lines in param file
for /f %%A in ('
find /c /v "" ^<"%pFile%"
') do if %%A neq 5 (
>&2 echo(multiline parameter values not supported & goto die
) %= if =%
:: extract and trim param value
for /f "useBack skip=3 delims=" %%A in ("%pFile%") do (
if not defined pN set "pN=%%A"
) %= for /f =%
set "pN=%pN:~7,-3%"
:: die if param value is " or "", else trim leading/trailing quotes
if defined pN (
setLocal enableDelayedExpansion
(call) %= OR emulation =%
if !pN!==^" (call;)
if !pN!=="" (call;)
if errorLevel 1 (
for /f delims^=^ eol^= %%A in ("!pN!") do (
endLocal & set "pN=%%~A"
) %= for /f =%
) else (
>&2 echo(empty parameter values (""^) not supported & goto die
) %= if errorLevel =%
) else (
:: no more params on cmd line
set /a paramC-=1 & goto last
) %= if defined =%
:: die if param value contains "
if not "%pN:"=""%"=="%pN:"=%" (
>&2 echo(quotes (^"^) in parameter values not supported & goto die
) %= if =%
:: assign nth param, shift params, and return to start of loop
set "param%paramC%=%pN%" & shift /1 & goto loop
:last - reached end of params
:: no param values on cmd line
if %paramC% equ 0 (
>&2 echo(no parameter values found & goto die
) %= if =%
:: list params
set param
goto end
:die
(call) %= sets errorLevel to 1 =%
:end
:: exit with appropriate errorLevel
endLocal & goto :EOF
The following conditions will terminate the program immediately:
no parameters found
multiline parameter
empty parameter (""", or " is permitted for the last parameter)
one or more quotes (") in a parameter value
To ease these restrictions, simply comment out the relevant lines. Read the inline comments for more information. Do not attempt to turn off the multiline parameter trap!

I invented the syntax-error-technic to solve the problem (partially).
With this solution it's even possible to receive multiline parameters and also carriage return characters.
There is no known parameter which fails!
BUT the drawback of this solution, the main process exits and only a child process continues.
That is a consequence of the capture trick, a syntax error is created by using an invalid parenthesis block ( Prepare ) PARAMS....
But the syntax error itself outputs the complete block, including the expanded value of %*.
The output is redirected to a file by the permanent redirect technic.
And the child process can retrieve the complete parameter from the file.
This solution can be useful, when the batch file only handles the parameter and always exit afterwards.
#echo off
REM *** Thread redirector
for /F "tokens=3 delims=:" %%F in ("%~0") do goto %%F
REM *** Clear params.tmp
break > params.tmp
start "" /b cmd /k "%~d0\:StayAlive:\..\%~pnx0 params.tmp"
(set LF=^
%=empty=%
)
REM *** Change prompt for better recognition
prompt #PROMPT#
REM *** Change streams permanently
REM *** stream1 redirects to params.tmp
REM *** stream2 redirects to nul
echo on >nul 2>nul 0>nul 3>params.tmp 4>nul 5>&3
#REM *** This is the magic part, it forces a syntax error, the error message itself shows the expanded %asterix without ANY modification
( Prepare ) PARAMS:%LF%%*%LF%
echo Works
exit /b
REM *** Second thread to fetch and show the parameters
:StayAlive
:__WaitForParams
if %~z1 EQU 0 (
goto :__WaitForParams
)
REM *** Show the result
findstr /n "^" %1

It's up to the user who types the command to escape any special characters. Your program cannot do anything about what the shell does before your program even runs. There is no other "bullet proof" solution to this.

Related

Replace a string in text file using batch file [duplicate]

i have to search a string from a txt like Pippo.K=5 and replace it with Pippo.K=1. I need to search the entire string. What i did is:
set "search=Pippo.K=5"
set "replace=Pippo.K=1"
set "textFile=%SettingFile%.txt"
for /f "delims=" %%i in ('type "%textFile%" ^& break ^> "%textFile%" ') do (
set "line=%%i"
setlocal enabledelayedexpansion
set "line=!line:%search%=%replace%!"
>>"%textFile%" echo(!line!
endlocal
)
but what i returned is
5=Pippo.K=1=5
How can i fix this error?
The following script constitutes a pure batch-file solution. Supposing it is stored as repl-str.bat, you need to call it like this for your application:
repl-str.bat "%SettingFile%.txt" "Pippo.K=5" "Pippo.K=1" "%SettingFile%.txt"
This specifies the input file %SettingFile%.txt, the literal and case-sensitive search string Pippo.K=5, the replacement string Pippo.K=1 and the output file %SettingFile%.txt that is the same as the input file (the related technique has been taken from this answer: Batch script to find and replace a string in text file without creating an extra output file for storing the modified file). If no output file is given, the result is output to the console (useful for testing). If a fifth command line argument is given (arbitrary value), the search is done in a case-sensitive manner.
Here is the code of the script repl-str.bat:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "FILE_I=%~1"
set "SEARCH=%~2"
set "REPLAC=%~3"
set "FILE_O=%~4"
set "CASE=%~5"
set "FLAG=%~6"
if not defined FILE_I exit /B 1
if not defined SEARCH exit /B 1
if not defined FILE_O set "FILE_O=con"
if defined CASE set "CASE=#"
if defined FLAG set "FLAG=#"
for /F "delims=" %%L in ('
findstr /N /R "^" "%FILE_I%" ^& break ^> "%FILE_O%"
') do (
set "STRING=%%L"
setlocal EnableDelayedExpansion
set "STRING=!STRING:*:=!"
call :REPL RETURN STRING SEARCH REPLAC "%CASE%" "%FLAG%"
>> "%FILE_O%" echo(!RETURN!
endlocal
)
endlocal
exit /B
:REPL rtn_string ref_string ref_search ref_replac case flag
setlocal EnableDelayedExpansion
set "STR=!%~2!"
set "SCH=!%~3!"
set "RPL=!%~4!"
if "%~5"=="" (set "OPT=/I") else (set "OPT=")
if not defined SCH endlocal & set "%~1=" & exit /B 1
set "SCH_CHR=!SCH:~,1!"
if not "%~6"=="" set "SCH_CHR="
if "!SCH_CHR!"=="=" set "SCH_CHR=" & rem = terminates search string
if "!SCH_CHR!"==""^" set "SCH_CHR=" & rem " could derange syntax
if "!SCH_CHR!"=="%%" set "SCH_CHR=" & rem % ends variable expansion
if "!SCH_CHR!"=="^!" set "SCH_CHR=" & rem ! ends variable expansion
call :LEN SCH_LEN SCH
call :LEN RPL_LEN RPL
set /A RED_LEN=SCH_LEN-1
set "RES="
:LOOP
call :LEN STR_LEN STR
if not defined STR goto :END
if defined SCH_CHR (
set "WRK=!STR:*%SCH_CHR%=!"
if %OPT% "!WRK!"=="!STR!" (
set "RES=!RES!!STR!"
set "STR="
) else (
call :LEN WRK_LEN WRK
set /A DFF_LEN=STR_LEN-WRK_LEN-1,INC_LEN=DFF_LEN+1,MOR_LEN=DFF_LEN+SCH_LEN
for /F "tokens=1,2,3 delims=," %%M in ("!DFF_LEN!,!INC_LEN!,!MOR_LEN!") do (
rem set "RES=!RES!!STR:~,%%M!"
if defined WRK set "WRK=!WRK:~,%RED_LEN%!"
if %OPT% "!STR:~%%M,1!!WRK!"=="!SCH!" (
set "RES=!RES!!STR:~,%%M!!RPL!"
set "STR=!STR:~%%O!"
) else (
set "RES=!RES!!STR:~,%%N!"
set "STR=!STR:~%%N!"
)
)
)
) else (
if %OPT% "!STR:~,%SCH_LEN%!"=="!SCH!" (
set "RES=!RES!!RPL!"
set "STR=!STR:~%SCH_LEN%!"
) else (
set "RES=!RES!!STR:~,1!"
set "STR=!STR:~1!"
)
)
goto :LOOP
:END
if defined RES (
for /F delims^=^ eol^= %%S in ("!RES!") do (
endlocal
set "%~1=%%S"
)
) else endlocal & set "%~1="
exit /B
:LEN rtn_length ref_string
setlocal EnableDelayedExpansion
set "STR=!%~2!"
if not defined STR (set /A LEN=0) else (set /A LEN=1)
for %%L in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) do (
if defined STR (
set "INT=!STR:~%%L!"
if not "!INT!"=="" set /A LEN+=%%L & set "STR=!INT!"
)
)
endlocal & set "%~1=%LEN%"
exit /B
Basically, this approach takes the first character of the search string and looks it up in the input text. At each match, it is checked whether the whole search string occurs. If so, it is replaced by the replacement string by removing as many characters as the search string consists of, hence avoiding sub-string replacement syntax which fails in case the search string contains =, or the search or the replacement string contains % or !.
However, if the first character of the search string is =, ", % or !, the approach is different, the script checks every single character position for occurrence of the search string then, with the disadvantage of reduced overall performance. If a sixth command line argument is given (arbitrary value), this (slow) mode is forced.
Batch variable substring substitution does have limitations. Dealing with literal equal signs is one of them.
powershell "(gc \"%textFile%\") -replace '%search%','%replace%'"
would work. That PowerShell one-liner is a simple alternative to your for /f loop without that limitation.
If you prefer a for /F loop, if your text file is an ini-style file, try this:
#echo off & setlocal
set "searchItem=Pippo.K"
set "searchVal=5"
set "newVal=1"
set "textFile=test.txt"
>"outfile.txt" (
for /f "eol=; usebackq tokens=1* delims==" %%I in ("%textFile%") do (
if /I "%%~I"=="%searchItem%" (
if "%%~J"=="%searchVal%" (
echo %%I=%newVal%
) else echo %%I=%%J
) else (
if not "%%~J"=="" (echo %%I=%%J) else echo %%I
)
)
)
move /y "outfile.txt" "%textFile%"
Be advised that if any of the items in your file has a blank value (e.g. valuename=), the equal sign will be stripped unless you add some additional logic.
You might also consider using ini.bat from this answer.

use batch file to read and write version number into a file

I'm trying to write a batch file to increment version number every time i run it, but I'm confuse about "for /f" and the behaviour of the batch file when I test it by using command prompt. Please help me with this.
here's my batch file
for /f "tokens=2,3 " %%i in (version.h) do (
set /a number=%%j+1
echo %%i
echo %%j
echo %number%
if %%i==REVISION (
echo line1
echo #define %%i "%number%" >> temp.file
) else (
echo line2
echo #define %%i %%j >> temp.file
)
)
del /q version.h
ren temp.file version.h
and here's my version.h
#define MAJOR "1"
#define MINOR "0"
#define REVISION "242"
The batch file can only produce correct result at the first run(#define REVISION "243"), and has a weird result at the second run(#define REVISION "0"). The third run's result is correct("#define REVISION "244"), but the forth run it goes weird again(#define REVISION "1"), and so on.
It seems that I didn't parse the correct string so I cannot have correct result every time.
I typed "for /?" in the command prompt and read the help message, but still cannot understand it, please help me with this. Any reply would be appreciate!
The following script does what you want and preserves empty lines and special characters present in the target file. There is no temporary file involved, the file modification is accomplished in-place.
So here is the code -- reference the explanatory remarks for how it works:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem // Define constants here:
set "FILE=%~1" & rem // (provide the target file as command line argument)
set "DIRECTIVE=#define" & rem // (name of the directive to search)
set "DEFINITION=REVISION" & rem // (name of the definition to search)
set "CASESENS=" & rem // (set to non-empty for case-sensitive searches)
set "QUOTED="^" & rem // (set to non-empty for quoting returned number)
rem // Resolve arguments and options:
if not defined FILE ((>&2 echo ERROR: no file specified!) & exit /B 1)
if defined CASESENS (set "CASESENS=") else (set "CASESENS=/I")
if defined QUOTED (set "QUOTED="^") else (set "QUOTED=")
rem // Loop through all lines in the target file:
setlocal EnableDelayedExpansion
for /F delims^=^ eol^= %%L in ('
rem/ /* Prefix lines with line numbers to not lose empty ones; ^& ^
rem/ after having read file, deplete its entire content: */ ^& ^
findstr /N /R "^^" "!FILE!" ^& ^> "!FILE!" break
') do (
endlocal
set "FLAG="
set "LINE=%%L"
rem // Split line into three tokens:
for /F "tokens=1-3 eol= " %%I in ("%%L") do (
set "FIELD1=%%I"
set "FIELD2=%%J"
set "NUMBER=%%~K"
setlocal EnableDelayedExpansion
rem // Check first token for matching directive name:
if %CASESENS% "!FIELD1:*:=!"=="!DIRECTIVE!" (
rem // Check second token for matching definition name:
if %CASESENS% "!FIELD2!"=="!DEFINITION!" (
endlocal
rem // Increment number of third token:
set /A "NUMBER+=1"
set "FLAG=#"
setlocal EnableDelayedExpansion
)
)
endlocal
)
setlocal EnableDelayedExpansion
rem // Write output line into target file:
>> "!FILE!" (
rem // Check whether dirctive and definition matched:
if defined FLAG (
rem // Match found, so write new line with incremented number:
echo(!DIRECTIVE! !DEFINITION! %QUOTED%!NUMBER!%QUOTED%
) else (
rem // No match found, so write original line:
echo(!LINE:*:=!
)
)
)
endlocal
endlocal
exit /B

Batch: How to remove all empty columns from a csv file

I have a CSV file like this:
P,PC,,PL,B,15feb16,P,Bay,RP,15-FEB-16,22-FEB-16,7,,,,,,11,14,138,14,16,993.42,-12,-84,-12,,,,,,,,,17,2,-10,0,0,1,1,16:05:53,15FEB16
P,PC,,PL,I,1FEB-16,P,In,RP,15-FEB-16,22-FEB-16,7,,,,,,25,5,32,5,5,-29.7,-24,-168,-24,,,,,,,,,520,14,-10,0,0,1,1,10-MAY-201606:05:53,15-FEB-16
P,PC,,PC,S,15FEB16,P,Su,RP,15-FEB-16,22-FEB-16,7,,,,,,6,5,32,56,5,4.65,0,0,0,,,,,,,,,546,0,0,0,0,1,1,10-MAY-201606:05:53,15-FEB-16
The code I have written is:
#echo off
setlocal EnableDelayedExpansion
for /F "delims=" %%a in (C:\Pca.csv) do (
set line=%%a
set line=!line:,,=, ,!
set line=!line:,,=, ,!
for /F "tokens=1,2,3* delims=," %%i in (^"!line!^") do (
echo %%i,%%l>>C:\P.csv
)
)
But it only deletes 2nd and 3rd column, no matter whether it is empty or contains data.
The sample output file should be like:
P,PC,PL,B,15feb16,P,Bay,RP,15-FEB-16,22-FEB-16,7,11,14,138,14,16,993.42,-12,-84,-12,17,2,-10,0,0,1,1,16:05:53,15FEB16
P,PC,PL,I,1FEB-16,P,In,RP,15-FEB-16,22-FEB-16,7,25,5,32,5,5,-29.7,-24,-168,-24,520,14,-10,0,0,1,1,10-MAY-201606:05:53,15-FEB-16
P,PC,PC,S,15FEB16,P,Su,RP,15-FEB-16,22-FEB-16,7,6,5,32,56,5,4.65,0,0,0,546,0,0,0,0,1,1,10-MAY-201606:05:53,15-FEB-16
Here is a quite comprehensive and adaptive script that removes empty columns from CSV-formatted data.
Before the code is shown, let us take a look at the help message that appears when called with /?:
"del-empty-cols-from-csv.bat"
This script removes any empty columns from CSV-formatted data. A column is con-
sidered as empty if the related fields in all rows are empty, unless the switch
/H is given, in which case the first line (so the header) is evaluated only.
Notice that fields containing white-spaces only are not considered as empty.
USAGE:
del-empty-cols-from-csv.bat [/?] [/H] csv_in [csv_out]
/? displays this help message;
/H specifies to regard the header only, that is the very first row,
to determine which columns are considered as empty; if NOT given,
the whole data, hence all rows, are taken into account instead;
csv_in CSV data file to process, that is, to remove empty columns of;
these data must be correctly formatted CSV data, using the comma as
separator and the quotation mark as text delimiter; regard that
literal quotation marks must be doubled; there are some additional
restrictions: the data must not contain any line-breaks; neither
must they contain any asterisks nor question marks;
csv_out CSV data file to write the return data to; this must not be equal
to csv_in; note that an already existing file will be overwritten
without prompt; if not given, the data is displayed on the console;
As you can read, there are two operation modes: standard (no switch) and header mode (switch /H).
Given that the following CSV data is fed into the script...:
A, ,C, ,E,F
1, , ,4,5,
1, , , ,5,
1, ,3,4, ,
...the returned CSV data in standard mode will look like...:
A,C, ,E,F
1, ,4,5,
1, , ,5,
1,3,4, ,
...and the returned CSV data in header mode (/H) will look like:
A,C,E,F
1, ,5,
1, ,5,
1,3, ,
Remind that the spaces in the above sample data must actually not be present in the files; they have just been inserted here for better illustration of the said operation modes.
Now, this is the complete code:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "OPT_HEAD=%~1"
if "%OPT_HEAD%"=="/?" (
goto :MSG_HELP
) else if /I "%OPT_HEAD%"=="/H" (
shift
) else if "%OPT_HEAD:~,1%"=="/" (
set "OPT_HEAD="
shift
) else set "OPT_HEAD="
set "CSV_IN=%~1"
if not defined CSV_IN (
>&2 echo ERROR: no input file specified!
exit /B 1
)
set "CSV_OUT=%~2"
if not defined CSV_OUT set "CSV_OUT=con"
for /F "delims==" %%V in ('2^> nul set CELL[') do set "%%V="
setlocal EnableDelayedExpansion
if not defined OPT_HEAD (
for /F %%C in ('^< "!CSV_IN!" find /C /V ""') do set "NUM=%%C"
) else set /A NUM=1
set /A LIMIT=0
< "!CSV_IN!" (
for /L %%L in (1,1,%NUM%) do (
set /P "LINE="
call :PROCESS LINE LINE || exit /B !ErrorLevel!
set /A COUNT=0
for %%C in (!LINE!) do (
set /A COUNT+=1
if not defined CELL[!COUNT!] set "CELL[!COUNT!]=%%~C"
if !LIMIT! LSS !COUNT! set /A LIMIT=COUNT
)
)
)
set "PAD=" & for /L %%I in (2,1,!LIMIT!) do set "PAD=!PAD!,"
> "!CSV_OUT!" (
for /F usebackq^ delims^=^ eol^= %%L in ("!CSV_IN!") do (
setlocal DisableDelayedExpansion
set "LINE=%%L%PAD%"
set "ROW="
set /A COUNT=0
setlocal EnableDelayedExpansion
call :PROCESS LINE LINE || exit /B !ErrorLevel!
for %%C in (!LINE!) do (
endlocal
set "CELL=%%C"
set /A COUNT+=1
setlocal EnableDelayedExpansion
if !COUNT! LEQ !LIMIT! (
if defined CELL[!COUNT!] (
for /F delims^=^ eol^= %%R in ("!ROW!,!CELL!") do (
endlocal
set "ROW=%%R"
)
) else (
endlocal
)
) else (
endlocal
)
setlocal EnableDelayedExpansion
)
if defined ROW set "ROW=!ROW:~1!"
call :RESTORE ROW ROW || exit /B !ErrorLevel!
echo(!ROW!
endlocal
endlocal
)
)
endlocal
endlocal
exit /B
:PROCESS var_return var_string
set "STRING=!%~2!"
if defined STRING (
set "STRING="!STRING:,=","!""
if not "!STRING!"=="!STRING:**=!" goto :ERR_CHAR
if not "!STRING!"=="!STRING:*?=!" goto :ERR_CHAR
)
set "%~1=!STRING!"
exit /B
:RESTORE var_return var_string
set "STRING=!%~2!"
if "!STRING:~,1!"==^""" set "STRING=!STRING:~1!"
if "!STRING:~-1!"==""^" set "STRING=!STRING:~,-1!"
if defined STRING (
set "STRING=!STRING:","=,!"
)
set "%~1=!STRING!"
exit /B
:ERR_CHAR
endlocal
>&2 echo ERROR: `*` and `?` are not allowed!
exit /B 1
:MSG_HELP
echo(
echo("%~nx0"
echo(
echo(This script removes any empty columns from CSV-formatted data. A column is con-
echo(sidered as empty if the related fields in all rows are empty, unless the switch
echo(/H is given, in which case the first line ^(so the header^) is evaluated only.
echo(Notice that fields containing white-spaces only are not considered as empty.
echo(
echo(
echo(USAGE:
echo(
echo( %~nx0 [/?] [/H] csv_in [csv_out]
echo(
echo( /? displays this help message;
echo( /H specifies to regard the header only, that is the very first row,
echo( to determine which columns are considered as empty; if NOT given,
echo( the whole data, hence all rows, are taken into account instead;
echo( csv_in CSV data file to process, that is, to remove empty columns of;
echo( these data must be correctly formatted CSV data, using the comma as
echo( separator and the quotation mark as text delimiter; regard that
echo( literal quotation marks must be doubled; there are some additional
echo( restrictions: the data must not contain any line-breaks; neither
echo( must they contain any asterisks nor question marks;
echo( csv_out CSV data file to write the return data to; this must not be equal
echo( to csv_in; note that an already existing file will be overwritten
echo( without prompt; if not given, the data is displayed on the console;
echo(
exit /B
assuming, your original csv looks like this:
id_users,,,quantity,,date
1,,,1,,2013
1,,,1,,2013
2,,,1,,2013
then this single line should solve your request:
(for /f "tokens=1-3 delims=," %%a in (c:\pca.csv) do echo %%a,%%b,%%c)>c:\p.csv
resulting in:
id_users,quantity,date
1,1,2013
1,1,2013
2,1,2013
The trick is: consecutive delimiters are treated as one.
Edit: another approach, as it turned out, there are much more colums, than the original question showed.
#echo off
break>out.txt
for /F "delims=" %%a in (c:\pca.csv) do call :shorten "%%a"
goto :eof
:shorten
set "line=%~1"
:remove
set "line=%line:,,=,%"
echo %line%|find ",,">nul && goto :remove
echo %line%>>c:\p.csv
break>c:\p.csv: create outputfile (overwrite if exist)
replace two consecutive commas with one;
repeat, if there are any more consecutive commas.
Write the resulting line to the outfile.

Removing non alphanumeric characters in a batch variable

In batch, how would I remove all non alphanumeric (a-z,A-Z,0-9,_) characters from a variable?
I'm pretty sure I need to use findstr and a regex.
The solutionof MC ND works, but it's really slow (Needs ~1second for the small test sample).
This is caused by the echo "!_buf!"|findstr ... construct, as for each character the pipe creates two instances of cmd.exe and starts findstr.
But this can be solved also with pure batch.
Each character is tested if it is in the map variable
:test
set "_input=Th""i\s&& is not good _maybe_???"
set "_output="
set "map=abcdefghijklmnopqrstuvwxyz 1234567890"
:loop
if not defined _input goto endLoop
for /F "delims=*~ eol=*" %%C in ("!_input:~0,1!") do (
if "!map:%%C=!" NEQ "!map!" set "_output=!_output!%%C"
)
set "_input=!_input:~1!"
goto loop
:endLoop
echo(!_output!
And it could be speed up when the goto loop is removed.
Then you need to calculate the stringLength first and iterate then with a FOR/L loop over each character.
This solution is ~6 times faster than the above method and ~40 times faster than the solution of MC ND
set "_input=Th""i\s&& is not good _maybe_!~*???"
set "_output="
set "map=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 1234567890"
%$strLen% len _input
for /L %%n in (0 1 %len%) DO (
for /F "delims=*~ eol=*" %%C in ("!_input:~%%n,1!") do (
if "!map:%%C=!" NEQ "!map!" set "_output=!_output!%%C"
)
)
exit /b
The macro $strlen can be defined with
set LF=^
::Above 2 blank lines are required - do not remove
#set ^"\n=^^^%LF%%LF%^%LF%%LF%^^":::: StrLen pResult pString
set $strLen=for /L %%n in (1 1 2) do if %%n==2 (%\n%
for /F "tokens=1,2 delims=, " %%1 in ("!argv!") do (%\n%
set "str=A!%%~2!"%\n%
set "len=0"%\n%
for /l %%A in (12,-1,0) do (%\n%
set /a "len|=1<<%%A"%\n%
for %%B in (!len!) do if "!str:~%%B,1!"=="" set /a "len&=~1<<%%A"%\n%
)%\n%
for %%v in (!len!) do endlocal^&if "%%~b" neq "" (set "%%~1=%%v") else echo %%v%\n%
) %\n%
) ELSE setlocal enableDelayedExpansion ^& set argv=,
EDITED - #jeb is right. This works but is really, really slow.
#echo off
setlocal enableextensions enabledelayedexpansion
set "_input=Th""i\s&& is not good _maybe_???"
set "_output="
:loop
if not defined _input goto endLoop
set "_buf=!_input:~0,1!"
set "_input=!_input:~1!"
echo "!_buf!"|findstr /i /r /c:"[a-z 0-9_]" > nul && set "_output=!_output!!_buf!"
goto loop
:endLoop
echo !_output!
endlocal
So, back to the drawing board. How to make it faster? lets try to do as less operations as we can and use as much long substring as we can. So, do it in two steps
1.- Remove all bad characters that can generate problems. To do it we will use the hability of for command to identify these chars as delimiters , and then join the rest of the sections of god characters of string
2.- Remove the rest of the bad characters, locating them in string using the valids charactes as delimiters to find substrings of bad characters, replacing then in string
So, we end with (sintax adapted to what has been answered here)
#echo off
setlocal enableextensions enabledelayedexpansion
rem Test empty string
call :doClean "" output
echo "%output%"
rem Test mixed strings
call :doClean "~~asd123#()%%%^"^!^"~~~^"""":^!!!!=asd^>^<bm_1" output
echo %output%
call :doClean "Thi\s&& is ;;;;not ^^good _maybe_!~*???" output
echo %output%
rem Test clean string
call :doClean "This is already clean" output
echo %output%
rem Test all bad string
call :doClean "*******//////\\\\\\\()()()()" output
echo "%output%"
rem Test long string
set "zz=Thi\s&& is not ^^good _maybe_!~*??? "
set "zz=TEST: %zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%%zz%"
call :doClean "%zz% TEST" output
echo %output%
rem Time long string
echo %time%
for /l %%# in (1 1 100) do call :doClean "%zz%" output
echo %time%
exit /b
rem ---------------------------------------------------------------------------
:doClean input output
setlocal enableextensions enabledelayedexpansion
set "map=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890 "
set "input=%~1"
set "output="
rem Step 1 - Remove critical delimiters
(
:purgeCritical
for /L %%z in (1 1 10) do (
for /f tokens^=1^-9^,^*^ delims^=^=^"^"^~^;^,^&^*^%%^:^!^(^)^<^>^^ %%a in ("!input!") do (
set "output=!output!%%a%%b%%c%%d%%e%%f%%g%%h%%i"
set "input=%%j"
)
if not defined input goto outPurgeCritical
)
goto purgeCritical
)
:outPurgeCritical
rem Step 2 - remove any remaining special character
(
:purgeNormal
for /L %%z in (1 1 10) do (
set "pending="
for /f "tokens=1,* delims=%map%" %%a in ("!output!") do (
set "output=!output:%%a=!"
set "pending=%%b"
)
if not defined pending goto outPurgeNormal
)
goto purgeNormal
)
:outPurgeNormal
endlocal & set "%~2=%output%"
goto :EOF
Maybe not the fastest, but at least a "decent" solution
#echo eof
call :purge "~~asd123#()%%%^"^!^"~~~^:^=asd^>^<bm_1" var
echo (%var%)
goto :eof
:purge StrVar [RtnVar]
setlocal disableDelayedExpansion
set "str1=%~1"
setlocal enableDelayedExpansion
for %%a in ( - ! # # $ % ^^ ^& + \ / ^< ^> . ' [ ] { } ` ^| ^" ) do (
set "str1=!str1:%%a=!"
)
rem dealing with some delimiters
set "str1=!str1:(=!"
set "str1=!str1:)=!"
set "str1=!str1:;=!"
set "str1=!str1:,=!"
set "str1=!str1:^^=!"
set "str1=!str1:^~=!"
set "temp_str="
for %%e in (%str1%) do (
set "temp_str=!temp_str!%%e"
)
endlocal & set "str1=%temp_str%"
setlocal disableDelayedExpansion
set "str1=%str1:!=%"
set "str1=%str1::=%"
set "str1=%str1:^^~=%"
for /f "tokens=* delims=~" %%w in ("%str1%") do set "str1=%%w"
endlocal & set "str1=%str1%"
endlocal & if "%~2" neq "" (set %~2=%str1%) else echo %str1%
goto :eof
Still cannot deal with ~ and = but working on it
EDIT: = now will be cleared
EDIT: ~ now will be cleared

How can a batch script do the equivalent of "cat << eof"?

In Linux (Bash) there is a very useful functionality for dumping literal text out to another file like this:
cat > see.txt << EOF
contents going into
my file
EOF
What I need is the equivalent for a Windows batch script. I haven't found this kind of functionality built-in, but I was thinking I could write a subroutine to do it (I don't want to rely on anything that isn't natively in Windows since XP), but I'm having trouble. Here's what I have so far with help from various sources:
call:catMyChunk myCustomText c:\see.txt
exit /b
goto:myCustomText
This is my test file
Hope you like it.
<got these>
% and these %
! these too
yeah
:myCustomText
:catMyChunk
::Should call this function with 2 args, MYDELIM and outFile.txt
::where is to be catted to outFile.txt
::and text starts with <beginning of line>goto:MYDELIM
::and ends with <beginning of line>:MYDELIM
set searchStart=goto:%~1
set searchStop=:%~1
set outFile=%~2
set startLine=0
set endLine=0
for /f "delims=:" %%a in ('findstr -b -n !searchStart! %~dpnx0') do set "startLine=%%a"
for /f "delims=:" %%a in ('findstr -b -n !searchStop! %~dpnx0') do set "endLine=%%a"
set /a linesLeftToRead=%endLine% - %startLine%
del %outFile%
if "%linesLeftToRead%" LEQ "0" (
echo Error finding start and end delmieters for %searchStop% in catMyChunk routine
exit /B 1
)
setlocal DisableDelayedExpansion
for /f "usebackq skip=%startLine% delims=" %%a in (`"findstr /n ^^ %~dpnx0"`) do (
set "oneLine=%%a"
setlocal EnableDelayedExpansion
set "oneLine=!oneLine:*:=!"
set /a linesLeftToRead-=1
if !linesLeftToRead! LEQ 0 exit /B
echo(!oneLine!>>%outFile%
)
goto: EOF
So my requirement is that the chunk of text is output literally without any changes whatsoever (i.e. I don't want to have to escape blank lines, %, !, <, etc.). This code is doing almost everything I need, except I haven't found a way to get the exclamation points output properly. Here is the output I get, which isn't quite right:
This is my test file
Hope you like it.
<got these>
% and these %
these too
yeah
Edit:
For anyone wanting the modified version of the subroutine that now works, here it is:
:catMyChunk
::Should call this function with 2 args, MYDELIM and outFile.txt
::where is to be catted to outFile.txt
::and text starts with <beginning of line>goto:MYDELIM
::and ends with <beginning of line>:MYDELIM
set searchStart=goto:%~1
set searchStop=:%~1
set outFile=%~2
set startLine=0
set endLine=0
for /f "delims=:" %%a in ('findstr -b -n !searchStart! %~dpnx0') do set "startLine=%%a"
for /f "delims=:" %%a in ('findstr -b -n !searchStop! %~dpnx0') do set "endLine=%%a"
set /a linesLeftToRead=%endLine% - %startLine%
del %outFile%
if "%linesLeftToRead%" LEQ "0" (
echo Error finding start and end delmieters for %searchStop% in catMyChunk routine
exit /B 1
)
setlocal DisableDelayedExpansion
for /f "usebackq skip=%startLine% delims=" %%a in (`"findstr /n ^^ %~dpnx0"`) do (
set "oneLine=%%a"
set /a linesLeftToRead-=1
setlocal EnableDelayedExpansion
set "oneLine=!oneLine:*:=!"
if !linesLeftToRead! LEQ 0 exit /B
echo(!oneLine!>>%outFile%
endlocal
)
endlocal
goto: EOF
Your code is missing a single endlocal in your FOR-loop.
You will then create for each loop a new local-context through the setlocal EnableDelayedExpansion, this will explode with some more lines in your text file.
Also, to preserve the exclamation marks (and also the carets) you need the toggling technique:
DOS batch files: How to read a file?
setlocal DisableDelayedExpansion
for /f "usebackq skip=%startLine% delims=" %%a in (`"findstr /n ^^ %~dpnx0"`) do (
set "oneLine=%%a"
setlocal EnableDelayedExpansion
set "oneLine=!oneLine:*:=!"
set /a linesLeftToRead-=1
if !linesLeftToRead! LEQ 0 exit /B
echo(!oneLine!>>%outFile%
endlocal
)
+1, Interesting application! I modified your code for a simpler and faster version:
#echo off
call :catMyChunk myCustomText see.txt
exit /b
goto:myCustomText
This is my test file
Hope you like it.
<got these>
% and these %
! these too
yeah
:myCustomText
:catMyChunk
::Should call this function with 2 args, MYDELIM and outFile.txt
::where is to be catted to outFile.txt
::and text starts with <beginning of line>goto:MYDELIM
::and ends with <beginning of line>:MYDELIM
setlocal DisableDelayedExpansion
set searchStart=goto:%~1
set searchStop=:%~1
set outFile=%~2
if exist %outFile% del %outFile%
set copyFlag=No
echo No> copyFlag
for /f "delims=" %%a in (%~f0) do (
set "oneLine=%%a"
setlocal EnableDelayedExpansion
if !copyFlag! == No (
if !oneLine! == %searchStart% echo Yes> copyFlag
) else (
if !oneLine! == %searchStop% (
echo No> copyFlag
) else (
echo/!oneLine!>> %outFile%
)
)
endlocal
set /p copyFlag=< copyFlag
)
endlocal
goto :eof
I also created another version that looks more like the Linux version, that is, the lines to copy are placed directly after invoking the routine, and the execution continue after the last copied line. Of course, to make this possible the routine is not invoked via call, but entered with a goto, and when the routine ends it execute a goto %MYDELIM% instead of a "return" (exit /b or goto :eof). Also, because a goto can not have parameters, the "parameters" are defined in variables before the invocation.
#echo off
set searchStop=EndOfMyText
set outFile=see.txt
goto :catMyChunk EndOfMyText
This is my test file
Hope you like it.
<got these>
% and these %
! these too
yeah
:EndOfMyText
exit /b
:catMyChunk
::Before JUMP to this "function" define 2 vars: searchStop and outFile
::where is to be catted to %outFile%
::and text starts with goto :catMyChunk %searchStop%
::and ends with :%searchStop%
setlocal DisableDelayedExpansion
set searchStart=goto :catMyChunk %searchStop%
if exist %outFile% del %outFile%
set copyFlag=No
echo No> copyFlag
for /f "delims=" %%a in (%~f0) do (
set "oneLine=%%a"
setlocal EnableDelayedExpansion
if !copyFlag! == No (
if /I !oneLine! == !searchStart! echo Yes> copyFlag
) else (
if !oneLine! == :%searchStop% (
echo No> copyFlag
) else (
echo/!oneLine!>> %outFile%
)
)
endlocal
set /p copyFlag=< copyFlag
)
endlocal
goto %searchStop%
EDIT
This new version is even faster and now works with all special cases, including empty lines:
:catMyChunk
::Should call this function with 2 args, MYDELIM and outFile.txt
::where is to be catted to outFile.txt
::and text starts with <beginning of line>goto:MYDELIM
::and ends with <beginning of line>:MYDELIM
setlocal EnableDelayedExpansion
set searchStart=goto:%~1
set searchStop=:%~1
set outFile=%~2
if exist %outFile% del %outFile%
findstr /n ^^ "%~f0" > pipeline.txt
call :seekMyChunk < pipeline.txt
del pipeline.txt
exit /B
:seekMyChunk
set oneLine=:EOF
set /P oneLine=
if !oneLine! == :EOF goto startNotFound
set oneLine=!oneLine:*:=!
if not !oneLine! == %searchStart% goto seekMyChunk
:catNextLine
set oneLine=:EOF
set /P oneLine=
if !oneLine! == :EOF goto stopNotFound
set oneLine=!oneLine:*:=!
if !oneLine! == %searchStop% goto :eof
echo/!oneLine!>> %outFile%
goto catNextLine
:startNotFound
echo Error finding start delimiter for %searchStart% in catMyChunk
goto :eof
:stopNotFound
echo Error finding stop delimiter for %searchStop% in catMyChunk
goto :eof

Resources