I have to combine csv files into 1 file using a batch file.
The problem is that I only need the third row from each file.
How can I extract a specific row from a csv file in the windows command line / batch file?
I have found skip as command to skip lines from start of file, but I need to also skip the files from after the line I need.
This is my current code (after converting from xls to csv):
for %%i in (headers.csv) do (
for /f "delims=" %%j in ('type "%%i"') do echo %%j >> combined.csv
)
for %%i in (file*.csv) do (
for /f "skip=2 delims=" %%j in ('type "%%i"') do echo %%j >> combined.csv
)
for %%i in (headers.csv) do (
set "flag=Y"
for /f "delims=" %%j in ('type "%%i"') do if defined flag set "flag="&echo %%j >> combined.csv
)
for %%i in (file*.csv) do (
set "flag=Y"
for /f "skip=2 delims=" %%j in ('type "%%i"') do if defined flag set "flag="&echo %%j >> combined.csv
)
flag is set to something each time a new file is chosen. The first 2 lines of that file are then skipped, and on the third flag is cleared and the line reproduced. Once flag is cleared (becomed not-defined) the echo won't happen.
if defined works on the current (ie run-time) value of the variable, not the initial (parse-time) value.
#echo off
setlocal enableextensions disabledelayedexpansion
(
for /f "tokens=1,* delims=:" %%a in ('
findstr /n "^" headers.csv file*.csv ^| findstr /r /c:"^[^:]*:3:"
') do (
set "line=%%b"
setlocal enabledelayedexpansion
echo(!line:~2!
endlocal
)
) > combined.csv
It will use findstr to retrieve the full content from all the indicated files (I was not sure and I've also included the headers file), with lines numbered. As multiple files are requested, the output from this first findstr command will be in the format
filename.ext:lineNumber:linedata
This is filtered by a second findstr using a regular expression to only retrieve the third line from each file.
These lines will be processed by the for loop, using the colon as a delimiter to remove the file name. To avoid problems with lines that could start with a colon, the line number and the aditional colon are removed with a substring operation.
If in the final code the reference to the files forces findstr to include in the output the full path to the files, the tokens clause needs to be changed to tokens=2,* to face the colon in the drive name.
Related
I am trying to write a batch script that reads 18th line from a .log file and outputs the same. The .log file name varies each time. abc_XXXX.log where xxxx are process IDs. Below is the code I am trying to run to achieve this.
:Test1
set "xprvar=" for /F "skip=17 delims=" %%p in (abc*.log) do (echo %%p& goto
break)
:break
pause
goto END
set var=anyCommand doesn't work. It just sets the var to the literal string.
The usage of afor /f is the right way, just the variable assignment works different:
for /F "skip=17 delims=" %%p in ('dir /b abc*.log') do ( set "xprvar=%%p"& goto break )
There is also an option using FindStr
#Echo Off
For /F "Tokens=1-2* Delims=:" %%A In ('FindStr/N "^" "abc_*.log" 2^>Nul'
) Do If %%B Equ 18 Echo %%A:%%C
Pause
The above example Echoes the <filename>:<18th line content>, but there's no reason in the appropriate situation why you couldn't change that to read:
#Echo Off
For /F "Tokens=1-2* Delims=:" %%A In ('FindStr/N "^" "abc_*.log" 2^>Nul'
) Do If %%B Equ 18 Set "xprvar=%%C"
If there is more than one matching filename in the directory, the variable would be set to the content in the last file parsed.
#ECHO Off
SETLOCAL
FOR %%f IN (abc*.log) DO (
SET "reported="
FOR /f "skip=17delims=" %%p IN (%%f) DO IF NOT DEFINED reported (
ECHO %%p
SET "reported=Y"
)
)
Assign each filename in turn to %%f.
For each filename found, clear the reported flag then read the file, skipping the first 17 lines. echo the 18th line and set the reported flag so that the remainder of the lines are not echoed.
#Echo off&SetLocal EnableExtensions EnableDelayedExpansion
cd "C:\Documents and Settings\John\Desktop\New\Interest\f2"
Pushd "C:\Documents and Settings\John\Desktop\New\Interest\f2"
Set Line#=26
Set /A LOfs=24 -1, Len=34 - LOfs
For %%A in (*.txt) do For /F "tokens=1* delims=:" %%B in (
'Findstr /N ".*" "%%A" ^|Findstr "^%Line#%:"'
) do if %errorlevel% == 0 Set "Line=%%C"&Ren "%%~fA" "!Line:~%LOfs%,%Len%! - %%A!""
Popd
In the above I am trying to change the filename of files in a directory with text in it at a certain position.
If line 26 is blank do nothing and do not change filename.
I have gone wrong somewhere and am going round in circles.
Can anyone help?
Thank you.
You don't state how your script fails, but I can see some potential problems. I also see possible simplifications.
You certainly don't need both CD and PUSHD
I got rid of the numeric variables and included the number literals in the actual code. You can revert back to variables if you want.
You don't need the outer FOR loop. FINDSTR can search multiple files when using wildcards in the file name, and then it will include the filename, followed by : in the output. So if you add the /N option, output will have the form filename:line#:text. You can then adjust the 2nd FINDSTR to return only the correct line numbers.
It is not enough to ignore blank lines. Your rename only works if there is at least one valid file name character after the 23rd character. Filenames cannot include :, *, ?, /, \, <, >, or |. (I may have missed some). I adjusted the FOR /F delims and the FINDSTR search to compensate.
FOR variable expansion like %%A will corrupt values if they contain ! and delayed expansion is enabled. ! is a valid character in file names. So the delayed expansion must be toggled on and off within the loop.
I believe the following will do what you want. The code below will simply echo the rename commands. Remove the ECHO before the ren once it gives the correct results.
#echo off
setlocal disableDelayedExpansion
pushd "C:\Documents and Settings\John\Desktop\New\Interest\f2"
for /f "tokens=1,3 delims=:*?\/<>|" %%A in (
'findstr /n "^" "*.txt" ^| findstr "^[^:]*:26:.......................[^:*?\\/<>|]"'
) do (
set "old=%%A"
set "line=%%B"
setlocal enableDelayedExpansion
ECHO ren "!old!" "!line:~23,11! - !old!"
endlocal
)
popd
An slightly different method to Daves:
#Echo Off
Set "SrcDir=%UserProfile%\Desktop\New\Interest\f2"
Set "Mask=*.txt"
Set "Line#=26"
Set "LOfs=23"
Set "Len=11"
If /I Not "%CD%"=="%SrcDir%" Pushd "%SrcDir%"2>Nul&&(Set _=T)||Exit/B
For /F "Tokens=1-2* Delims=:" %%A In ('FindStr/N "^" "%Mask%" 2^>Nul'
) Do If "%%B"=="%Line#%" If Not "%%~C"=="" (Set "Line=%%C"
SetLocal EnableDelayedExpansion
If Not "!Line:~%LOfs%,%Len%!"=="" (
If Not Exist "!Line:~%LOfs%,%Len%! - %%A" (
Ren "%%A" "!Line:~%LOfs%,%Len%! - %%A"))
EndLocal)
If "_"=="T" PopD
This method don't require findstr.exe nor toggle setlocal/endlocal, so it should run faster. Also, it avoids to re-process any already renamed file changing the plain for %%A by a for /F combined with dir command.
#Echo off
SetLocal EnableDelayedExpansion
cd "C:\Documents and Settings\John\Desktop\New\Interest\f2"
Set /A Line#=26, LOfs=24 -1, Len=34 - LOfs
For /F "delims=" %%A in ('dir /A-D /B *.txt') do (
rem Read the desired line from this file
(for /L %%i in (1,1,%Line#%) do set "Line=" & set /P "Line=") < "%%A"
if defined Line ECHO Ren "%%~fA" "!Line:~%LOfs%,%Len%! - %%A"
)
Note also that when this Batch file ends the current directory is automatically recovered to the current one when setlocal command was executed, so pushd/popd commands are not needed either.
I am creating a code that strips through different MAC addresses randomly, but cannot figure out how to do this. My thought on how to approach this is to randomize or rearrange the order of the MAC address in the text file with this script, but I cannot figure out how to do this with a batch file. How this will work is that it will read "maclist.txt", then create a new temp file with the random order "maclist_temp.txt", that will be the rearranged file. Then, it will pull this randomized file in order.
I have tried Google and searching the web, but I haven't found anything too useful. I'm still actively looking, but any advice would be extremely useful.
Something as simple as extracting and deleting a random line and then adding to the bottom might work. Randomization would be better though, but I want to keep the original list. Something like:
Make a temp copy of maclist.txt called maclist_temp.txt
Take one random MAC address, remove it from maclist_temp.txt
Readd it to the bottom
That is all I want, but any suggestions are welcome.
You may try this batch file to help you to shuffle your maclist.txt. The usage of the batch code is
C:\> type list.txt | shuffle.bat > maclist_temp.txt
Here are the contents of shuffle.bat:
#ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
SET TmpFile=tmp%RANDOM%%RANDOM%.tmp
TYPE NUL >%Tmpfile%
FOR /F "tokens=*" %%i IN ('MORE') DO SET Key=!RANDOM!!RANDOM!!RANDOM!000000000000& ECHO !Key:~0,15!%%i>> %TmpFile%
FOR /F "tokens=*" %%i IN ('TYPE %TmpFile% ^| SORT') DO SET Line=%%i&ECHO.!Line:~15!
::DEL %TmpFile%
ENDLOCAL
After issuing the above command, maclist_temp.txt will contain a randomized list of MAC addresses.
Hope this helps.
Here is a simpler method to randomize/randomise a file, no temp files needed. You can even reuse the same input filename.
Limitations are: blank lines and line starting with ; will be skipped, and lines starting with = will have all leading = signs stripped and ^ characters are doubled.
#echo off
setlocal
for /f "delims=" %%a in (maclist.txt) do call set "$$%%random%%=%%a"
(for /f "tokens=1,* delims==" %%a in ('set $$') do echo(%%b)>newmaclist.txt
endlocal
I really like foxidrive's approach. Nevertheless I want to provide a solution with all the listed limitations eliminated (although cmd-related restrictions like file sizes < 2 GiB and line lengths < ~ 8 KiB remain).
The key is delayed expansion which needs to be toggled to not lose explamation marks. This solves all the potential problems with special characters like ^, &, %, !, (, ), <, >, | and ".
The counter index has been implemented in order not to lose a single line of the original text file, which could happen without, because random may return duplicate values; with index appended, the resulting variable names $$!random!.!index! are unique.
The findstr /N /R "^" command precedes every line of the original file with a line number followed by a colon. So no line appears empty to the for /F loop which would ignore such. The line number also implicitly solves the issue with leading semicolons, the default eol character of for /F.
Finally, everything up to and including the first colon (remember the said prefix added by findstr) is removed from every line before being output, hence no more leading equal-to signs are dismissed.
So here is the code:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set /A "index=0"
for /f "delims=" %%a in ('findstr /N /R "^" "%~dpn0.lst"') do (
setlocal EnableDelayedExpansion
for /F %%b in ("$$!random!.!index!") do (
endlocal
set "%%b=%%a"
)
set /A "index+=1"
)
> "%~dpn0.new" (
for /f "delims=" %%a in ('set $$') do (
set "item=%%a"
setlocal EnableDelayedExpansion
echo(!item:*:=!
endlocal
)
)
endlocal
exit /B
This seems to work. Feed it a command line parameter of the file to randomize.
#echo off
setlocal EnableDelayedExpansion
rem read the number of lines in the file
rem the find prepends the line number so we catch blank lines
for /f "delims=" %%n in ('find /c /v "" %1') do set "len=%%n"
set len=!len:*: =!
rem echo %1 has %len% lines
rem Relocate as many lines as there are lines in the file
for /l %%j in (1 1 !len!) do (
rem echo starting round %%j
rem geta random number between 1 and the number of lines in the file
set /a var=!random! %% !len! + 1
rem echo relocating line !var!
rem make sure there is no temp file
if exist %1.temp del %1.temp
rem read each line of the file, write any that don't match and then write the one that does
<%1 (
for /l %%i in (1 1 !len!) do (
rem if it is the target line then save it
if %%i == !var! (
set /p found=
rem echo saving !found!
)
rem if it is the target line then write it
if not %%i == !var! (
set /p other=
rem echo writing !other!
echo !other!>> %1.temp
)
)
rem now write the target line at the end
rem echo appending !found!
echo !found!>> %1.temp
)
rem replace the original with the temp version
move %1.temp %1>nul
)
rem print the result
type %1
Place in cmd file
for /f "tokens=2 delims=/" %%m in ('cmd /e:on /v:on /c "for /f %%f in (maclist.txt) do #echo !random!/%%f" ^| sort') do echo %%m
It spawns a cmd which reads the mac list in the inner for, prefixes a random value and a slash to the mac and sorts the list. Then this list is splitted in the outter for using the slash as delimiter and printing the mac address.
I am looking to find a batch or VBS solution to strip out lines in a program generated text file with the extension of .trs.
In every .trs file created there is a line that contains the word 'labour'. I need every line after the line that contains the word labour to be deleted.
The .trs files are all stored in c:\export
I have searched for this but some of the commands were well over my head. Could anyone be so kind as to offer me a cut and paste open of the whole batch file, please.
I believe this is the code you are looking for (in a batch file) to remove all of the lines above the word "labour". Let me know if modifications need to be made to the code (such as if there are more than one instance of "labour" in the file).
#echo OFF
setLocal EnableDelayedExpansion
cd C:\export
for /f "delims=" %%I in ('findstr /inc:"labour" "test.trs"') do (
set /A"line=%%I"
)
set count=0
for /f "delims=" %%A in (test.trs) do (
If !count! GEQ %line% goto ExitLoop
echo %%A >>temp.txt
set /A count+=1
echo !count!
)
:ExitLoop
type temp.txt > test.trs
del temp.txt
endlocal
OUTPUT:
test.trs (BEFORE changes)
this
is
a
labour
test
of
the
results
test.trs (AFTER changes)
this
is
a
Here is an alternate method to process every .trs file in "C:\export":
#echo off
if not exist "C:\export\*.trs" goto :EOF
if exist "C:\export\queue.tmp" del /q "C:\export\queue.tmp"
for /f "tokens=*" %%A in ('dir /b "C:\export\*.trs"') do (
for /f "tokens=1,2 delims=:" %%B in ('findstr /inc:"labour" "C:\export\%%A" ^| findstr /n .*') do if "%%B" equ "1" set LineNumber=%%C
for /f "tokens=1* delims=:" %%D in ('findstr /n .* "C:\export\%%A"') do if %%D lss %LineNumber% echo.%%E>>"C:\export\queue.tmp"
move /y "C:\export\queue.tmp" "C:\export\%%A">NUL
)
First, I do some error checking to avoid things that would break the script. Next, I pull a list of .trs files stored in C:\export, and loop through each file.
I use 'findstr /inc:"labour" "C:\export\%%A"' to get the line number where "labour" is found in the current file, then pipe it into 'findstr /n .*' to number the results in case more than one match is found.
I then use a for loop with "tokens=1,2 delims=:" to find the first result (if "%%B" equ "1") and store the line number (set LineNumber=%%C).
Next, I use 'findstr /n .* "C:\export\%%A"' to read every line of the file, "tokens=1* delims=:" to separate out the line numbers again, then copy all the data to a temp file until %LineNumber% has been reached. This method of reading the file (using findstr and numbering the lines) also ensures that no blank lines will be skipped by the for loop.
Finally, I replace the original file with the temp file, then loop through to the next file.
I tried to keep the above code as slimmed down as possible. Here is the same script with formatting, comments, visual feedback, and user-definable variables:
#echo off
::Set user-defined Variables
set FilePath=C:\export
set FileType=*.trs
set Keyword=labour
::Check for files to process and exit if none are found
if not exist "%FilePath%\%FileType%" echo Error. No files to process.&goto :EOF
::Delete temp file if one already exists
if exist "%FilePath%\queue.tmp" del /q "%FilePath%\queue.tmp"
::List all files in the above specified destination, then process them one at a time
for /f "tokens=*" %%A in ('dir /b "%FilePath%\%FileType%"') do (
::Echo the text without a line feed (so that "Done" ends up on the same line)
set /p NUL=Processing file "C:\export\%%A"... <NUL
::Search the current file for the specified keyword, and store the line number in a variable
for /f "tokens=1,2 delims=:" %%B in ('findstr /inc:"%Keyword%" "%FilePath%\%%A" ^| findstr /n .*') do (
if "%%B" equ "1" set LineNumber=%%C
)>NUL
::Output all data from the current file to a temporary file, until the line number found above has been reached
for /f "tokens=1* delims=:" %%D in ('findstr /n .* "%FilePath%\%%A"') do (
if %%D lss %LineNumber% echo.%%E>>"%FilePath%\queue.tmp"
)>NUL
::Replace the current file with the processed data from the temp file
move /y "%FilePath%\queue.tmp" "%FilePath%\%%A">NUL
echo Done.
)
This DOS batch script is stripping out the blank lines and not showing the blank lines in the file even though I am using the TYPE.exe command to convert the file to make sure the file is ASCII so that the FIND command is compatible with the file. Can anyone tell me how to make this script include blank lines?
#ECHO off
FOR /F "USEBACKQ tokens=*" %%A IN (`TYPE.exe "build.properties" ^| FIND.exe /V ""`) DO (
ECHO --%%A--
)
pause
That is the designed behavior of FOR /F - it never returns blank lines. The work around is to use FIND or FINDSTR to prefix the line with the line number. If you can guarantee no lines start with the line number delimiter, then you simply set the appropriate delimiter and keep tokens 1* but use only the 2nd token.
::preserve blank lines using FIND, assume no line starts with ]
::long lines are truncated
for /f "tokens=1* delims=]" %%A in ('type "file.txt" ^| find /n /v ""') do echo %%B
::preserve blank lines using FINDSTR, assume no line starts with :
::long lines > 8191 bytes are lost
for /f "tokens=1* delims=:" %%A in ('type "file.txt" ^| findstr /n "^"') do echo %%B
::FINDSTR variant that preserves long lines
type "file.txt" > "file.txt.tmp"
for /f "tokens=1* delims=:" %%A in ('findstr /n "^" "file.txt.tmp"') do echo %%B
del "file.txt.tmp"
I prefer FINDSTR - it is more reliable. For example, FIND can truncate long lines - FINDSTR does not as long as it reads directly from a file. FINDSTR does drop long lines when reading from stdin via pipe or redirection.
If the file may contain lines that start with the delimiter, then you need to preserve the entire line with the line number prefix, and then use search and replace to remove the line prefix. You probably want delayed expansion off when transferring the %%A to an environment variable, otherwise any ! will be corrupted. But later within the loop you need delayed expansion to do the search and replace.
::preserve blank lines using FIND, even if a line may start with ]
::long lines are truncated
for /f "delims=" %%A in ('type "file.txt" ^| find /n /v ""') do (
set "ln=%%A"
setlocal enableDelayedExpansion
set "ln=!ln:*]=!"
echo(!ln!
endlocal
)
::preserve blank lines using FINDSTR, even if a line may start with :
::long lines >8191 bytes are truncated
for /f "delims=*" %%A in ('type "file.txt" ^| findstr /n "^"') do (
set "ln=%%A"
setlocal enableDelayedExpansion
set "ln=!ln:*:=!"
echo(!ln!
endlocal
)
::FINDSTR variant that preserves long lines
type "file.txt" >"file.txt.tmp"
for /f "delims=*" %%A in ('findstr /n "^" "file.txt.tmp"') do (
set "ln=%%A"
setlocal enableDelayedExpansion
set "ln=!ln:*:=!"
echo(!ln!
endlocal
)
del "file.txt.tmp"
If you don't need to worry about converting the file to ASCII, then it is more efficient to drop the pipe and let FIND or FINDSTR open the file specified as an argument, or via redirection.
There is another work around that completely bypasses FOR /F during the read process. It looks odd, but it is more efficient. There are no restrictions with using delayed expansion, but unfortunately it has other limitations.
1) lines must be terminated by <CR><LF> (this will not be a problem if you do the TYPE file conversion)
2) lines must be <= 1021 bytes long (disregarding the <CR><LF>)
3) any trailing control characters are stripped from each line.
4) it must read from a file - you can't use a pipe. So in your case you will need to use a temp file to do your to ASCII conversion.
setlocal enableDelayedExpansion
type "file.txt">"file.txt.tmp"
for /f %%N in ('find /c /v "" ^<"file.txt.tmp"') do set cnt=%%N
<"file.txt.tmp" (
for /l %%N in (1 1 %cnt%) do(
set "ln="
set /p "ln="
echo(!ln!
)
)
del "file.txt.tmp"
I wrote a very simple program that may serve as replacement for FIND and FINDSTR commands when they are used for this purpose. My program is called PIPE.COM and it just insert a blank space in empty lines, so all the lines may be directly processed by FOR command with no further adjustments (as long as the inserted space don't cares). Here it is:
#ECHO off
if not exist pipe.com call :DefinePipe
FOR /F "USEBACKQ delims=" %%A IN (`pipe ^< "build.properties"`) DO (
ECHO(--%%A--
)
pause
goto :EOF
:DefinePipe
setlocal DisableDelayedExpansion
set pipe=´)€ì!Í!ŠÐŠà€Ä!€ü.t2€ü+u!:æu8²A€ê!´#€ì!Í!².€ê!´#€ì!Í!²+€ê!´#€ì!Í!Šò€Æ!´,€ì!Í!"Àu°´LÍ!ëÒ
setlocal EnableDelayedExpansion
echo !pipe!>pipe.com
exit /B
EDIT: Addendum as answer to new comment
The code at :DefinePipe subroutine create a 88 bytes program called pipe.com, that basically do a process equivalent to this pseudo-Batch code:
set "space= "
set line=
:nextChar
rem Read just ONE character
set /PC char=
if %char% neq %NewLine% (
rem Join new char to current line
set line=%line%%char%
) else (
rem End of line detected
if defined line (
rem Show current line
echo %line%
set line=
) else (
rem Empty line: change it by one space
echo %space%
)
)
goto nextChar
This way, empty lines in the input file are changed by lines with one space, so FOR /F command not longer omit they. This works "as long as the inserted space don't cares" as I said in my answer.
Note that the pipe.com program does not work in 64-bits Windows versions.
Antonio
Output lines including blank lines
Here's a method I developed for my own use.
Save the code as a batch file say, SHOWALL.BAT and pass the source file as a command line parameter.
Output can be redirected or piped.
#echo off
for /f "tokens=1,* delims=]" %%a in ('find /n /v "" ^< "%~1"') do echo.%%ba
exit /b
EXAMPLES:
showall source.txt
showall source.txt >destination.txt
showall source.txt | FIND "string"
An oddity is the inclusion of the '^<' (redirection) as opposed to just doing the following:
for /f "tokens=1,* delims=]" %%a in ('find /n /v "" "%~1"') do echo.%%ba
By omitting the redirection, a leading blank line is output.
Thanks to dbenham, this works, although it is slightly different than his suggestion:
::preserve blank lines using FIND, no limitations
for /f "USEBACKQ delims=" %%A in (`type "file.properties" ^| find /V /N ""`) do (
set "ln=%%A"
setlocal enableDelayedExpansion
set "ln=!ln:*]=!"
echo(!ln!
endlocal
)
As mentioned in this answer to the above question, it doesn't seem that lines are skipped by default using for /f in (at least) Windows XP (Community - Please update this answer by testing the below batch commands on your version & service pack of Windows).
EDIT: Per Jeb's comment below, it seems that the ping command, in at least Windows XP, is
causing for /f to produce <CR>'s instead of blank lines (If someone knows specifically why, would
appreciate it if they could update this answer or comment).
As a workaround, it seems that the second default delimited token (<space> / %%b in the example)
returns as blank, which worked for my situation of eliminating the blank lines by way of an "parent"
if conditional on the second token at the start of the for /f, like this:
for /f "tokens=1,2*" %%a in ('ping -n 1 google.com') do (
if not "x%%b"=="x" (
{do things with non-blank lines}
)
)
Using the below code:
#echo off
systeminfo | findstr /b /c:"OS Name" /c:"OS Version"
echo.&echo.
ping -n 1 google.com
echo.&echo.
for /f %%a in ('ping -n 1 google.com') do ( echo "%%a" )
echo.&echo.&echo --------------&echo.&echo.
find /?
echo.&echo.
for /f %%a in ('find /?') do ( echo "%%a" )
echo.&echo.
pause
.... the following is what I see on Windows XP, Windows 7 and Windows 2008, being the only three versions & service packs of Windows I have ready access to: