So I wrote a batch that has some code to check how many times it has been run by reading a textfile and then writing back into that textfile the new, increased number.
#ECHO OFF
for /f "delims=" %%x in (TimesRun.txt) do set Build=%%x
set Build=%Build%+1
#echo Build value : %Build%
echo %Build%>>TimesRun.txt
Pause
That does append the textfile allright, but it adds "1+1" to it. Silly me! I forgot to use the /a switch to enable arithmetic operations! But when I change the code accordingly...
#ECHO OFF
for /f "delims=" %%x in (TimesRun.txt) do set Build=%%x
set /a Build=%Build%+1
#echo Build value : %Build%
echo %Build%>>TimesRun.txt
Pause
... something funny happens: Instead of appending my file, ECHO is off. gets written on the console. Now, I know that this usually happens when ECHO is used without text or with an empty variable. I have added the first #echo Build value : %Build% specifically to see whether the variable Build is empty or not; it is not, and the calculation was carried out correctly.
I already figured out that
>>TimesRun.txt (echo %Build%)
does bring the desired result. I still do not understand why
echo %Build%>>TimesRun.txt
does not, however. What am I missing?
You are unintentionally specifying a redirection handle.
Redirection allows you to specify a certain handle that defines what is to be redirected:
0 = STDIN (keyboard input)
1 = STDOUT (text output)
2 = STDERR (error text output)
3 ~ 9 = undefined
For the input redirection operator <, handle 0 is used by default; for the output redirection operators > and >>, the default handle is 1.
You can explicitly specify a handle by putting a single numeric figure in front of the redirection operator; for instance 2> defines to redirect the error text output.
In your echo command line you are doing exactly this unintentionally, when %Build% is a single numberic digit, like 1 for example:
echo 1>>TimesRun.txt
To avoid that, you have the following options:
To reverse the statement so that the redirection definition comes first:
>>TimesRun.txt echo %Build%
This is the most general and secure way of doing redirections.
To enclose the redirected command in parentheses:
(echo %Build%)>>TimesRun.txt
This also works safely.
To put a SPACE in front of the redirection operator:
echo %Build% >>TimesRun.txt
This works too, but the additional SPACE is included in the output of echo.
See also this great post: cmd.exe redirection operators order and position.
Batch file redirection can be customized to specify where you're outputting to.
command 1>file.txt redirects the output of STDOUT to file.txt
command 2>file.txt redirects the output of STDERR to file.txt
Your build value was 1, so you inadvertently told CMD to send the output of echo to TimesRun.txt - when you run echo by itself, it prints it's status (ON or OFF).
You also could have said echo %Build% >>TimesRun.txt and the space would prevent the value of Build from being treated as a redirection command.
The Microsoft article Using command redirection operators explains the 3 standard handles and how to redirect them to another handle, command, device, file or console application.
Redirection of output written to handle 1 - STDOUT - to a file should be done with just
using > ... create file if not already existing or overwrite existing file, or
using >> ... create file if not already existing or append to existing file.
The redirection operators are usually appended at end of a command line. But this is problematic in case of using command ECHO and the string output to STDOUT ends with 1 to 9.
One of several solutions is to specify in this case the redirection at beginning of the command line:
#for /F "delims=" %%x in (TimesRun.txt) do #set Build=%%x
#set /A Build+=1
#echo Build value : %Build%
>>TimesRun.txt echo %Build%
Executing this small batch file without #echo off at top from within a command prompt window shows what Windows command processor executes after preprocessing each line with text file TimesRun.txt containing currently the value 0 or does not exist at all.
echo 1 1>>TimesRun.txt
It can be seen that Windows command interpreter moved the redirection to end of line with inserting a space and 1 left to >>.
With above batch code the line with >> really executed after preprocessing is:
echo 2 1>>TimesRun.txt
Specifying the redirection at end with 1>>, i.e. use in the batch file
echo %Build%1>>TimesRun.txt
is also no good idea as this would result on first run in executing the line:
echo 11 1>>TimesRun.txt
So 11 is written into the file instead of 1. This wrong output could be avoided by inserting a space before >> or 1>>, i.e. use one of those two:
echo %Build% >>TimesRun.txt
echo %Build% 1>>TimesRun.txt
But then the space after %Build% is also written into the file as really executed is:
echo 1 1>>TimesRun.txt
The trailing space would be no problem here, but should be nevertheless avoided.
Note: On using arithmetic operations, i.e. set /A ... any string not being a number or operator is automatically interpreted as variable name and the current value of this variable is used on evaluating the arithmetic expression. Therefore after set /A with environment variable names consisting only of word characters and starting with an alphabetic character as usually used for environment variables no %...% or !...! must be used inside the arithmetic expression. This is explained in help of command SET output into console window on running set /? within a command prompt window.
I have 15 input files and a map that says which input file should each output line come from. The input files all look the same:
1,some numbers
2,some numbers
...
2000,some numbers
The map file looks like
1, filename1
2, filename1
3, filename7
...
2000, filename4
I want to create one output file that consists of 2000 lines which were copied from the input files based on the map i.e. output lines 1 and 2 were copied from filename1, line 3 was copied from filename3, ... and line 2000 was copied from filename4.
Can you please help me figure out how to use the map?
I think my code should be something like
for i = 1 to 2000
currentInputFileName = (read i-th line from the map
to figure out input file name)
findstr "%i," /b %currentInputFileName% > %outputFile%
next i
Thanks for any help
Your approach is correct, but you have a couple minor errors: the number of line must be taken from the first token in the lines in map.txt file, and the filename is the second token. You could use your same code, but in such a case you must eliminate line numbers from map.txt file. Otherwise, you may eliminate the counter variable from the code and the space between the number and the filename in map.txt file.
Besides, you use a couple constructs that are particularly slow:
call subroutine is slow. Is better to place the subroutine code inside the for and use Delayed Expansion to get the variables.
The append redirection >> is slow because the output file is open and closed each time that a line is appended to it. It is much faster a normal > redirection.
This code should run much faster than the original; it uses the original format in map.txt file: 1,filename1, etc.
#echo off
(FOR /f "tokens=1,2 delims=," %%G IN (map.txt) DO (
findstr /b "%%G," "%%H"
)) > output.txt
I ended up writing it like this
#echo off
SET count=1
FOR /f %%G IN (map.txt) DO (call :subroutine "%%G")
GOTO :eof
:subroutine
findstr /b "%count%," %1 >> output.txt
set /a count+=1
GOTO :eof
Not sure if this is optimal in terms of speed. My input files actually contain not 2000 lines but 2000 blocks, each block 120 lines
1,1,some numbers
1,2,some numbers
...
1,120,some numbers
...
2000,120,some numbers
Is there a way to write this code to run faster?
I am trying to create a batch file that will edit a .csv and remove the first column, and any summary lines contained in the file. I am, however, fairly new to programming batch files, so I am not sure the best way to start this, and it would be great if you could include a basic explanation of how the code works so I can be self-sustaining in the future!
,Type,Date,Num,Name,Memo,Member,Clr,Split,Alias,Value,Balance
ABB - Egypt,,,,,,,,,,,
ElAin EL-Sokhna,,,,,,,,,,,
,Invoice,09-06-10,12005,ABB - EL-Sokhna,,Accounts Receivable,,Training Income,15000,,15000
,Invoice,09-14-11,12005,ABB - EL-Sokhna,“ElAin EL-Sokhna“ Trainer for OTS Application: First two weeks,Training Income,,Accounts,,150001,0
Total ElAin EL-Sokhna,,,,,,,,,241194,210400,301794
ABB - Egypt - Other,,,,,,,,,,,
There are various iterations of this file, as they come from a monthly report, I need to remove the first (empty) column, and any rows that look like ABB - Egypt,,,,,,,,,,, or Total ElAin EL-Sokhna,,,,,,,,,241194,210400,301794
So the output should be:
Type,Date,Num,Name,Memo,Member,Clr,Split,Alias,Value,Balance
Invoice,09-06-10,12005,ABB - EL-Sokhna,,Accounts,,Training Income,15000,,15000
Invoice,09-14-11,13002,ABB - EL-Sokhna,“ElAin EL-Sokhna“ Trainer for OTS Application: First two weeks,Training Income,,Accounts,,150001,0
Thanks for the input!
EDIT: It seems I wasn't clear enough in my OP (Sorry, first time here).
There are two processes that need to happen here, in every file the first column must be deleted, and any lines that are either title lines ABB - Egypt,,,,,,,,,,, or summary lines Total ElAin EL-Sokhna,,,,,,,,,241194,210400,301794 need to be removed.
All lines that need to be kept will be mostly filled in, such as ,Type,Date,Num,Name,Memo,Member,Clr,Split,Alias,Value,Balance or ,Invoice,09-06-10,12005,ABB - EL-Sokhna,,Accounts Receivable,,Training Income,15000,,15000 Notice that, as in the second line, it is possible for there to be some missing values in them, so doing a search for something like ",," will not work.
Batch is a terrible language for modifying text files. There are a great many special cases that require arcane knowledge to work around the problem. You may have a script that seems to do what you want, and then some wrinkle appears in your data, and the entire script may have to be redesigned.
With regard to your specific problem, it appears to me that you only want to preserve rows that begin with a comma, meaning the first column is empty. Of those remaining rows, you want to remove the first (empty) column.
Assuming none of the rows you want to keep have an empty value for the second column, then there is a really trivial solution:
#echo off
>"%~1.new" (for /f "delims=, tokens=*" %%A in ('findstr "^," %1') do echo %%A)
move /y "%~1.new" %1 >nul
The script expects the file to be passed as the first and only argument. So if your script is named "fixCSV.bat", and the file to be modified is "c:\test\file.csv", then you would use:
fixCSV "c:\test\file.csv"
The %1 expands to the value of the first argument, and %~1 is the same, except it also strips any enclosing quotes that may or may not be present.
The FINDSTR command reads the file and writes out only lines that begin with a comma. The FOR /F command iterates each line of output. The "delims=, tokens=*" options effectively strip all leading commas from each line, and the result is in variable %%A, which is then ECHOed. The entire construct is enclosed in parentheses and stdout is redirected to a temporary file. Finally, the temporary file is moved over top of the original file, thus replacing it.
If the 2nd column may be empty, then the result will be corrupted because it removes all leading commas (both columns 1 and 2 in this case). The script must be more complicated to compensate. You would need to set a variable and then use delayed expansion to get the sub-string, skipping the first character. But delayed expansion will corrupt expansion of the %%A variable if it contains the ! character. So delayed expansion must be toggled on and off. You are beginning to see what I mean by lots of special cases.
#echo off
setlocal disableDelayedExpansion
>"%~1.new" (
for /f "delims=" %%A in ('findstr "^," %1') do (
set "ln=%%A"
setlocal enableDelayedExpansion
echo !ln:~1!
endlocal
)
)
move /y "%~1.new" %1 >nul
As the batch scripts become more complicated, they become slower and slower. It may not be an issue for most files, but if the file is really large (say hundreds of megabytes) then it can become an issue.
I almost never use pure batch to modify text files anymore. Instead, I use a hybrid JScript/batch utility that I wrote called JREPL.BAT. The utility is pure script that runs natively on any Windows machine from XP onward. JREPL.BAT is able to efficiently modify text files using regular expression replacement. Regular expressions can appear to be mysterious, but they are well worth the investment in learning.
Assuming you have JREPL.BAT somewhere within your PATH, then the following command is all that you would need:
jrepl "^,(.*)" "$1" /jmatch /f "yourFile.csv" /o -
The /F option specifies the file to read.
The /O option with value of - specifies that the output should replace the original file.
The /JMATCH option specifies that each replacement value is written out to a new line. All other text is dropped.
The first argument is the search expression. It matches any line that begins with a comma, and everything after that is captured in a variable named $1.
The second argument specifies the replacement value, which is simply the captured value in variable $1.
A way will be to define all your rules in a variable which will be used against
findstr. The rules must be defined like this :
/c:"String which exclude the line" /c:"Another string which exclude the Line" /c: "etc.."
This rules must be exact (That they can't be found in a line who must stay).
For the empty first colonne you can use a substitution the way i made it in the code with
,Type=Type
,Invoice=Invoice
Test.bat :
#echo off&cls
setlocal enabledelayedexpansion
Rem The rules
set $String_To_Search=/c:"ABB - Egypt," /c:"Total ElAin El-Sokhna," /c:"ElAin EL-Sokhna," /c:"ABB - Egypt - Other,"
for /f "delims=" %%a in (test.csv) do (
set $line=%%a
Rem the substitutions for the first Column
set $Line=!$Line:,Type=Type!
set $line=!$Line:,Invoice=Invoice!
Rem the test and the ouput if nothing was found
echo !$Line! | findstr /i %$String_To_Search% >nul || echo !$Line!
))>Output.csv
I used a file test.csv for my test.
The ouput is redirected to Output.csv
Perhaps is this what you want?
#echo off
setlocal EnableDelayedExpansion
for /F "delims=" %%a in (input.csv) do (
set "line=%%a"
if "!line:~0,1!" equ "," echo !line:~1!
)
When a problem is not enough explained we can only guess the missing details. In this case, I assumed that you just want the lines that start with comma, deleting it. The output is the same as your output example...
EDIT: Output example added
Type,Date,Num,Name,Memo,Member,Clr,Split,Alias,Value,Balance
Invoice,09-06-10,12005,ABB - EL-Sokhna,,Accounts Receivable,,Training Income,15000,,15000
Invoice,09-14-11,12005,ABB - EL-Sokhna,“ElAin EL-Sokhna“ Trainer for OTS Application: First two weeks,Training Income,,Accounts,,150001,0
I would start here to learn this: How can you find and replace text in a file using the Windows command-line environment?
It covers many details of substitution from Windows command line and many ways to do it, some requiring only what's built into Windows, and some requiring other downloadable software.
Magoo is right, more criteria is needed, but there might be enough information in the linked page for you to get past the main hurdles.
#ECHO OFF
SETLOCAL
(FOR /f "tokens=*delims=," %%a IN ('findstr /b /l "," q28079306.txt') DO ECHO %%a)>newfile.txt
GOTO :EOF
I used a file named q28079306.txt containing your data for my testing.
Produces newfile.txt
My batch file terminates prematurely after I assign the first environmental variable (script output below). I've tried turning echo on, using errorlevels, sending the output to a text file, and checking syntax. I've spent several hours researching debugging batch scripts, but I have finally hit a brick wall.
Script's Goal: Search each directory name of the user's Program Files, looking for common antivirus programs. I realize that it would be easiest iterate through an array of antivirus names for this purpose, but I want to keep it simple for now.
#echo off
::variables
set AntiVirus1="Initial Value"
IF NOT ERRORLEVEL 0 echo %ERRORLEVEL%
else echo "env. variable created successfully."
for /d %%f in (""%ProgramFiles%\*"") do (
{
IF NOT ERRORLEVEL 0 echo %ERRORLEVEL%
echo "%%f"
if exist /i "*McAfee*" < %%f %AntiVirus1%="McAfee"
::find "Norton" < %%f
::find "Comodo" < %%f
::find "AVG" < %%f
}
echo %AntiVirus1%
#pause
Output of this script:
C:\Users\Matt\Desktop>set AntiVirus1="Initial Value"
C:\Users\Matt\Desktop>
Can someone point me to what I'm doing wrong?
UPDATE Corrected script, now working but returning incorrect results:
::#echo off
::variables
set AntiVirus1="Initial Value"
IF NOT ERRORLEVEL 0 (echo %ERRORLEVEL%) ELSE echo "env. variable created successfully."
echo Checking Program Files...
for /d %%f in ("%ProgramFiles%\*") do (
echo "%%f"
if %%f=="*adobe*" set AntiVirus1="adobe"
)
echo %AntiVirus1% found
#pause
First of all, ELSE must be on the same line with IF or on the same line with the closing parenthesis that pertains to IF. In you particular case you should change your first IF...ELSE command like this:
IF NOT ERRORLEVEL 0 (ECHO %ERRORLEVEL%) ELSE ECHO "env. variable created successfully."
or like this:
IF NOT ERRORLEVEL 0 (
ECHO %ERRORLEVEL%
) ELSE ECHO "env. variable created successfully."
(Capitalisation and indentation are perfectly optional.)
Other issues:
Duplicated quotation marks in the FOR loop header:
for /d %%f in (""%ProgramFiles%\*"") do (
should be
for /d %%f in ("%ProgramFiles%\*") do (
Braces ({, }) around the loop body. They are not part of the loop syntax (in fact, they are not part of batch scripting syntax at all), so should be dropped.
No closing parenthesis matching the opening one after DO. It should be added on a separate line after the loop body.
Incorrect use of ::-style comments in the loop body. They are not allowed inside bracketed blocks. Use REM instead.
UPDATE
In batch scripting, testing for a substring is done somewhat unusually. You'll need another environment variable and you'll also need to enable delayed expansion. The latter is not really connected with the comparison, but it is needed because the comparison is going to be performed within a bracketed block.
Here's your new script modified, with the changes highlighted:
::#echo off
::variables
set AntiVirus1="Initial Value"
IF NOT ERRORLEVEL 0 (echo %ERRORLEVEL%) ELSE echo "env. variable created successfully."
SETLOCAL EnableDelayedExpansion
echo Checking Program Files...
for /d %%f in ("%ProgramFiles%\*") do (
echo "%%f"
SET "folder=%%f"
if /I NOT "!folder:adobe=!"=="!folder!" set AntiVirus1="adobe"
)
echo %AntiVirus1% found
#pause
Here's a bit of explanation.
The ! syntax is a delayed expansion equivalent of % and is used with environment variables only, not with loop variables and not with command line parameters. Delayed expansion is needed because we are in a bracketed block. A bracketed block is parsed entirely before it starts executing, so all %var% expressions are expanded (evaluated) before the block starts and are not changed throughout the block's execution. That cannot suit us because we need to assign different values to a variable during the block's execution, and the values must be read within the block. Delayed expansion, as follows from the name, delays the expansion of a variable until the actual execution of every single command that references that variable. Because immediate expansion can still be used alongside delayed expansion, a different syntax is introduced, which is ! around variable names, instead of %.
!folder:adobe=! means evaluate folder replacing every occurrence of adobe with an empty string. The result of this expression is then compared to the (unchanged) value of folder. If there's a match, then the replacement didn't occur, which means there was no adobe in the value of folder in the first place. In this case we should do nothing. But if there was not a match, i.e. if the modified value didn't match the unmodified one, then we should set the AntiVirus1 variable. This is why there's NOT in front of the comparison.
The /I option simply means case-insensitive comparison.