Batch to insert a line in a text file - batch-file

My goal is to make a script which is able to insert a string under a specific line. This is what I have done:
SETLOCAL ENABLEDELAYEDEXPANSION
set outputFile=C:\Utilisateurs\a669884\Documents\script22.txt
set "_strInsert=import java.interceptor.ReportInterceptor;"
set "_strFind=import javax.interceptor.Interceptors;"
:Ask
echo Are you in the good folder (wlp-cas-provider)?(Y/N)
set INPUT=
set /P INPUT=Type input: %=%
If /I "%INPUT%"=="y" goto yes
If /I "%INPUT%"=="n" goto no
echo Incorrect input & goto Ask
:yes
for %%i in (*.java) DO (
FOR /F "usebackq delims=" %%A IN ("%%i") DO (
Echo %%A | Find "%_strFind%" && ECHO %%A>>"%outputFile%" && ECHO %_strInsert%>>"%outputFile%"
IF [!errorlevel!] == [1] ECHO %%A>>"%outputFile%"
)
MOVE /Y "%outputFile%" "%%i" && DEL /F /Q "%outputFile%"
)
:no
echo Oh no! Go to the right folder and comeback
cls
pause
The main idea is to copy the modified text in a temporary file and paste it in the original file. After that, the temporary file is deleted.
The code is working perfectly when it's done on only one file. With the for in order to do it on all .java files in a folder, it's not working anymore. The temporary file script22.txt is not deleted anymore and the text is piled up in it.

How about something along these lines: (untested)
#ECHO OFF
FOR %%A IN ("%CD%") DO IF /I NOT "%%~nxA"=="wlp-cas-provider" (
ECHO=Please change directory to wlp-cas-provider
ECHO= before running this script again.
ECHO=
ECHO=Press any key to exit ...
PAUSE>NUL
GOTO :EOF)
SET "_strFind=import javax.interceptor.Interceptors;"
SET "_strInsert=import java.interceptor.ReportInterceptor;"
SETLOCAL ENABLEDELAYEDEXPANSION
SET NL=^
SET "str2Add=%_strFind%!NL!%_strInsert%"
FOR /F "TOKENS=1-2* DELIMS=:" %%A IN ('FINDSTR/BLINC:"%_strFind%" *.java 2^>NUL'
) DO (SET/A "i=1+%%B"
FINDSTR/BLINC:"%_strInsert%" "%%A"|FINDSTR/B "!i!:">NUL
IF ERRORLEVEL 1 (( FOR /F "TOKENS=1* DELIMS=:" %%D IN ('FINDSTR/N $ "%%A"'
) DO IF %%D EQU %%B (ECHO=!str2Add!) ELSE ECHO=%%E)>>"%%A.tmp"))
FOR %%A IN (*.java.tmp) DO IF EXIST "%%~nA" (DEL "%%~nA" && REN "%%A" "%%~nA")

There is no heavy logical error in your script why the loop should fail. I suspect it is the pipe | that is causing the unexpected behaviour, because it initiates a new command prompt (cmd) instance for either side, where the left one receives an already expanded string (echo %%A), which may contain characters that have special meanings to cmd.
Here is a possible improvement of your script -- see all the explanatory remarks (rem):
#echo off
setlocal DisableDelayedExpansion
rem // Define constants here:
set "targetFolder=C:\Utilisateurs\wlp-cas-provider"
set "fileMask=*.java"
set "outputFile=C:\Utilisateurs\a669884\Documents\script22.txt"
set "_strInsert=import java.interceptor.ReportInterceptor;"
set "_strFind=import javax.interceptor.Interceptors;"
rem // Enumerate all matching files:
for %%I in ("%targetFolder%\%fileMask%") do (
rem /* Redirect the output of the whole `for /F` loop at once;
rem this improves the performance, and there is no appending `>>`,
rem so it does not matter whether the file already exists before: */
> "%outputFile%" (
rem /* Use `findstr /N` to precede every line by its line number
rem and a colon, so no line appears empty to `for /F` any more;
rem remember that `for /F` ignores empty lines otherwise: */
for /F "delims=" %%A in ('findstr /N "^" "%%~I"') do (
rem // Store the current line string with line number prefix:
set "line=%%A"
rem // Toggle delayed expansion to maintain exclamation marks:
setlocal EnableDelayedExpansion
rem // Remove the line number prefix and the colon:
set "line=!line:*:=!"
rem // Return the original line unconditionally:
echo(!line!
rem /* Explicitly initiate another child `cmd` instance on the
rem left side of the pipe with delayed expansion enabled;
rem escape the delayed expansion like `^^!` not to be
rem processed by the parent `cmd` instance, so the child
rem `cmd` instance receives the variable `!line!` only;
rem therefore strings potentially containing special
rem characters are expanded at the very latest point: */
(cmd /V /C echo(^^!line^^!| find "!_strFind!" > nul) && (
rem /* The string to search has been found, so return the
rem string to insert after it at this point: */
echo(!_strInsert!
)
endlocal
)
)
rem // Move the result file over the currently iterated one:
move /Y "%outputFile%" "%%~I" > nul
)
endlocal
exit /B
This allows literally all characters to appear in the search string. Note that quotation marks " need to be doubled to be processed correctly by find. If you want the whole line to match the search string, replace the find command line by findstr /X /C:"!_strFind!". In this case, if quotation marks appear in the search string, escape them like \"; literal backslashes need to be doubled like \\. Add the /I option to find or findstr if you want to search case-insensitively.
You can avoid the block containing the pipe...:
(cmd /V /C echo(^^!line^^!| find "!_strFind!" > nul) && (
echo(!_strInsert!
)
...when you use if statements to compare strings.
For example, use this instead if the line is expected to contain the search string:
if not "!line!"=="!line:*%_strFind%=!" echo(!_strInsert!
This always does a case-insensitive search because of the used sub-string replacement approach. This method may fail if any of the following characters occur in the search string: ^, !, %, =, ".
If the whole line is expected to match the search string, use this instead:
if "!line!"=="!_strFind!" echo(!_strInsert!
This does a case-sensitive search, unless you add the /I switch to the if statement.

Related

Batch script to add a line of text to files

I am a novice and found some info here on howto make a script that write a text line to files. The result seems fine (also for multiple files), except that the text ends up at the bottom and I need it at the top.
#echo off
set "$New_line=TEXT"
for /r "C:\" %%A in (*.txt) do >"%%A" echo %$New_line%
#ECHO Off
SETLOCAL
rem The following settings for the source directory, destination directory, target directory,
rem batch directory, filenames, output filename and temporary filename [if shown] are names
rem that I use for testing and deliberately include names which include spaces to make sure
rem that the process works using such names. These will need to be changed to suit your situation.
SET "sourcedir=u:\your files"
set "$New_line=TEXT"
:: a filename that simply won't exist in the subtree and doesn't end ".txt"
SET "dummyfilename=##qqyhnggqvxjr.#$#"
FOR /F "delims=" %%b IN ('dir /b /s/a-d "%sourcedir%\*.txt"') DO (
SET "skipme="
FOR /F "delims=:" %%c IN ('FINDSTR /n /L /C:"%$new_line%" "%%b"') DO IF "%%c" equ "1" SET "skipme=Y"
IF NOT DEFINED skipme (
>"%%~dpb%dummyfilename%" ECHO %$new_line%
TYPE "%%b">>"%%~dpb%dummyfilename%"
MOVE /y "%%~dpb%dummyfilename%" "%%b" >NUL
)
)
GOTO :EOF
I really don't believe you'd want to prepend the text to every .txt file on your c: drive. You should set "sourcedir" to start the processing on a small sample subtree.
The findstr processing looks for %$new_line% as the first line of each file selected by the outer for loop and sets skipme if that condition is found. In this way, the processing is skipped for files that have already been processed.
Re: use of closing parenthesis in added line:
If there are closed-parentheses in the line to be inserted, you need to "escape" the ) with a caret ^ thus : set "$New_line=#Firm_Begin(16368^)"
The reason for this is that batch will render the escaped-) as
IF NOT DEFINED skipme (
>"%%~dpb%dummyfilename%" ECHO #Firm_Begin(16368^)
TYPE "%%b">>"%%~dpb%dummyfilename%"
MOVE /y "%%~dpb%dummyfilename%" "%%b" >NUL
)
and recognise the ^) as an escaped-) hence treat the ) as if it was any regular text character.
Without the escape character, batch interprets the command as
IF NOT DEFINED skipme (
>"%%~dpb%dummyfilename%" ECHO #Firm_Begin(16368)
TYPE "%%b">>"%%~dpb%dummyfilename%"
MOVE /y "%%~dpb%dummyfilename%" "%%b" >NUL
)
This has two consequences.
For files where skipme is not defined, #Firm_Begin(16368 is sent to the dummy file as the first and only line, and the text of the entire file is then added and then fortunately the resultant is moved over the original file so that the dummy file does not exist.
In files where skipme is defined however, only the ...ECHO %$new_line% is skipped. The original file is then typed into the non-existent dummy file, and then the dummyfile overwrites the original - so no damage done....
...Or is there?
findstr...C:"%$new_line%"... discriminates whether to select the file to have the foreword added or not, via the flag skipme. Unfortunately, the value of $new_line now contains ^ and in this instance the ^ does not escape the ) because the /c:"expression" is quoted (as it has to be since the string-value contains spaces), so its contents are taken literally. findstring will search for the literal string #Firm_Begin(16368^), not #Firm_Begin(16368).
cure for this is to remove any ^ from $new_line, so
...'FINDSTR /n /L /C:"%$new_line:^=%" "%%b"'...
and add the /x switch as a refinement so that the string must match the entire line, hence
...
FOR /F "delims=" %%b IN ('dir /b /s/a-d "%sourcedir%\*.txt"') DO (
SET "skipme="
FOR /F "delims=:" %%c IN ('FINDSTR /n /x /L /c:"%$new_line:^=%" "%%b"') DO IF "%%c" equ "1" SET "skipme=Y"
IF NOT DEFINED skipme (
>"%%~dpb%dummyfilename%" ECHO %$new_line%
TYPE "%%b">>"%%~dpb%dummyfilename%"
MOVE /y "%%~dpb%dummyfilename%" "%%b" >NUL
)
)
GOTO :EOF
which should cure the problem.
But then there's the clean-up, where \x can be used to detect files which have #Firm_Begin(16368 as the first line, not #Firm_Begin(16368).
[this part I haven't tested, and unfortunately have no time to test atm]
FOR /F "delims=:" %%c IN ('FINDSTR /n /x /L /c:"%$new_line:~0,-3" "%%b"') DO IF "%%c" equ "1" echo incorrect headline in "%%b"
should indicate the files that suffer that condition. All that would need to be done for any such file detected would be to initialise the dummy file with the correct top line and then use >>"%dummyfile%" for /f "usebackqskip=1delims=" %%L in ("%%b") do echo %%L to append the contents of the file %%b , all but the first line to the dummy file.

Delete Last 6 Rows From Multiple .txt Files

I have never done a batch file before. I have a few dozen .txt files sitting in a folder (ex. C:\files).
The files all end with 6 rows of text that need to be deleted. A sample would be (note spaces in first line):
var...
'ascending';...
'LIT-xxx,LIT-xxx...
setfunction...
0.33...
getdate...
Additionally, I would like the "new" files to overwrite the current files so that the file names and directory do not change.
abs 10.txt
him 4.txt
lab 18.txt
The following code snippet does exactly what you want, deleting the last six lines from text files:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem // Define constants here:
set "FILES=C:\files\*.txt" & rem // (specify file location and pattern)
set /A "LINES=-6" & rem /* (specify number of lines to delete;
rem positive number: delete from begin,
rem negative number: delete from end) */
rem // Standard `for` loop to resolve file pattern:
for %%F in ("%FILES%") do (
rem // Get the count of lines for the current file:
for /F %%N in ('^< "%%~F" find /C /V ""') do set "COUNT=%%N"
rem // Initialise a line index:
set /A "INDEX=-LINES"
rem /* Enumerate all lines of the current file, preserving empty ones
rem by preceding each with a line number, so no line appears empty
rem to the `for /F` loop; the line number is split off later on;
rem in addition, the current file is emptied after being read: */
for /F "delims=" %%L in ('
findstr /N /R "^" "%%~F" ^& ^> "%%~F" break
') do (
rem // Increment index, get text of currently iterated line:
set /A "INDEX+=1" & set "LINE=%%L"
rem // Toggle delayed expansion to preserve exclamation marks:
setlocal EnableDelayedExpansion
rem // Check index value and write to current file conditionally:
if !INDEX! GTR 0 if !INDEX! LEQ !COUNT! (
rem // Split off line number from line text:
>> "%%~F" echo(!LINE:*:=!
)
endlocal
)
)
endlocal
exit /B
This approach does not use temporary files in order to avoid any name conflicts. However, due to the fact that there are multiple file write operations for every single file, the overall performance is a bit worse than when writing all data to a temporary file at once and moving it back onto the original file.
Backup your original files to a different backup folder, then run this script:
#echo off
setlocal enabledelayedexpansion
pushd "%temp%\Test"
for %%G in ("*.txt") do (set "break="
(for /f "delims=|" %%H in (%%~G) do (
if not defined break (
echo %%H | findstr /r /b /c:"[ ]*var.*" >nul && set break=TRUE || echo %%H )
)) >> %%~nG_mod.txt
del %%~G & ren %%~nG_mod.txt %%G )
popd
exit /b
It assumed:
your 6 rows of text always start from [any number of spaces]var[any text] row, as you posted in the question, where only one string of such kind is present in any file
other 5 bottom rows don't need to match in every file
you save the files to filter in %temp%\Test, and there are no other unrelated files in that dir.

Batch: Replacing a line by another line

I know similar questions had been asked many time, but I cannot workout something fitting my needs...
So I have a file, containing a secret key, the line is starting like :
SECRET_KEY =
The secret key is a non-fixed length.
I want to find this line, to save it somewhere, to replace the all file containing the secret key by another file (very similar, but with a different secret key which I don't want), and to reintroduce the original secret key in the new file!
So for the moment I have this code:
echo Saving the actual SECRET_KEY...
findstr "SECRET_KEY" somefile.py > SECRET_KEY.txt
echo Copying source code files...
xcopy %~dp0\somepath %~dp0\someotherpath /EY
This part is working, as I have stored the original secret key in a text file, and replaced the file by another (together with other files, directories structures, etc...)
So now how can I find again the line starting by SECRET_KEY and replace the all line by the content of SECRET_KEY.txt?
EDIT:
I made some progress I think:
echo Saving the SECRET_KEY...
findstr "SECRET_KEY" inputfile.py > SECRET_KEY.txt
echo Copying source code files...
xcopy %~dp0\somepath %~dp0\someotherpath /EY
echo Restoring the SECRET_KEY...
setlocal EnableDelayedExpansion
for /F "delims=" %%a in ('findstr /I "SECRET_KEY" SECRET_KEY.txt') do set "secret_key=%%a"
echo !secret_key!
#echo off
set file=inputfile.py
set newline=!secret_key!
set insertline=23
set output=outputfile.py
(for /f "tokens=1* delims=[]" %%a in ('find /n /v "##" ^< "%file%"') do (
if "%%~a"=="%insertline%" (
echo !newline!
) ELSE (
echo.%%b
)
)) > %output%
I can then delete SECRET_KEY.TXT, inputfile.py, and rename outputfile.py into inputfile.py
Most likely not the most efficent way, but it works...
EXCEPT for one thing! I have a lot a special characters in this inputfile, that's why I used !secret_key! for example...
But I also have a lot of:
[some stuff]
and in the outputfile, it appears as:
[some stuff
So I guess I'm close, but I can't find a delimiter which will allow every single character to be outputed!
The following script reads the line containing SECRET_KEY from file otherfile.py at first; then it gets the number of the line of the input file inputfile.py where SECRET_KEY is placed; afterwards it reads the file inputfile.py line by line, check each line number whether it matches the previously found one; if a match is found, the previously gathered key from file otherfile.py is returned, so the line is replaced; otherwise, the line returned unedited; the output data is written to file outputfile.py:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem // Define constants here:
set "INFILE=inputfile.py"
set "OUTFILE=outputfile.py"
set "KEYFILE=otherfile.py"
set "KEYWORD=SECRET_KEY"
rem // Retrieve key from a file:
set "KEY="
setlocal EnableDelayedExpansion
for /F delims^=^ eol^= %%K in ('findstr /L /C:"!KEYWORD!" "%KEYFILE%"') do (
endlocal
set "KEY=%%K"
rem // Toggle delayed expansion inside of the loop to not lose `!`:
setlocal EnableDelayedExpansion
rem // `goto` ensures to take *first* key; remove it to take *last* one;
goto :CONT_KEY
)
:CONT_KEY
endlocal
rem // Retrieve line number where key is found:
set "LRPL="
setlocal EnableDelayedExpansion
for /F "delims=:" %%M in ('findstr /N /L /C:"!KEYWORD!" "%INFILE%"') do (
endlocal
set "LRPL=%%M"
rem // Toggle delayed expansion inside of the loop to not lose `!`:
setlocal EnableDelayedExpansion
rem // `goto` ensures to take *first* key; remove it to take *last* one;
goto :CONT_NUM
)
:CONT_NUM
endlocal
rem // Move line by line from one file to another and replace key:
> "%OUTFILE%" (
rem /* Use `findstr` to precede every line with its line number plus `:`,
rem in order no line to appear empty to `for /F`, as this ignores such;
rem the prefix (up to and including the first `:`) is removed later: */
for /F "delims=" %%L in ('findstr /N /R "^" "%INFILE%"') do (
set "LINE=%%L"
rem // Extract the prefixed number of the current line:
for /F "delims=:" %%N in ("%%L") do set "LNUM=%%N"
rem // Toggle delayed expansion inside of the loop to not lose `!`:
setlocal EnableDelayedExpansion
rem /* Check current line number against previously retrieved one
rem and replace line in case of a match: *//
if !LNUM! EQU !LRPL! (echo(!KEY!) else (echo(!LINE:*:=!)
endlocal
)
)
endlocal
exit /B
Note that the KEYFILE and INFILE variables (defined at the top of the script) may be set to the same file path. However, OUTFILE must be different from INFILE; otherwise the referred file will be emptied.
This covers only the tasks of storing the SECRET_KEY line of a file to a variable and of exchanging the line containing SECRET_KEY in a file by the previously stored one and write the result to another file.

how to remove duplicate entry from the text file using batch script or vb script?

how can I remove the duplicate entry from the text file using batch script. All i want to remove the duplicates before "=" sign and "%%" is exist in every single text file. Text file look likes below
%%B05AIPS_CDDOWNLOAD_IBDE_UNC=\\%%B05AIPS_UPLOAD_NODE.\F$\DATA\IPSL\CDFILES\B05_NAG\CD\INCOMING
%%B05AIPS_CDDOWNLOAD_FTS_UNC=\\%%B05AIPS_UPLOAD_NODE.\B05_NAG\FTS\To_Clearpath\%%DATE_CCYYMMDD.
%%B05AIPS_CDDOWNLOAD_FTS_UNC=%%B05AIPS_CDDOWNLOAD_FTS_UNC.
I got about 30 plus different text files which contains above kind of entries and want to remove the duplicate line and want to keep the first occurrence. Remember duplicate line should be identified before "=" sign only and removal required for the entire line.Each of the different text files have got "%%" sign. Please guide me if there is way to do through batch script or vbscript? Thanks
Here is a simple batch-file solution; let us call the script rem-dups.bat. Supposing your input file is test.txt and your output file is result.txt, you need to provide these files as command line arguments, so you need to call it by: rem-dups.bat "test.txt" "results.txt". Here is the script:
#echo off
setlocal EnableExtensions EnableDelayedExpansion
set "INFILE=%~1"
set "OUTFILE=%~2"
if not defined INFILE exit /B 1
if not defined OUTFILE set "OUTFILE=con"
for /F "usebackq tokens=1,* delims==" %%K in ("%INFILE%") do (
set "LEFT=%%K"
set "RIGHT=%%L"
set "LEFT=!LEFT:*%%%%=__!"
rem Remove `if` query to keep last occurrence:
if not defined !LEFT! set "!LEFT!=!RIGHT!"
)
> "%OUTFILE%" (
for /F "delims=" %%F in ('set __') do (
set "LINE=%%F"
echo(!LINE:*__=%%%%!
)
)
endlocal
exit /B
The script is based on the fact that there cannot occur duplicate environment variables, that are such with equal names.
This code only works if the following conditions are fulfilled:
the file content is treated in a case-insensitive manner;
the order of lines in the output file does not matter;
the partial strings before the first = sign start with %% and contain at least one more character other than %;
the partial strings before the first = contain only characters which may occur within environment variable names, besides the leading %%;
the partial strings after the first = must not be empty;
the partial strings after the first = must not start with = on their own;
no exclamation marks ! are allowed within the file, because they may get lost or lead to other unexpected results;
Here is an alternative method using a temporary file:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "INFILE=%~1"
set "OUTFILE=%~2"
if not defined INFILE exit /B 1
if not defined OUTFILE set "OUTFILE=con"
set "TEMPFILE=%TEMP%\%~n0_%RANDOM%.tmp"
> "%TEMPFILE%" break
> "%OUTFILE%" (
for /F usebackq^ delims^=^ eol^= %%L in ("%INFILE%") do (
for /F tokens^=1^,*^ delims^=^=^ eol^= %%E in ("%%L") do (
> nul 2>&1 findstr /I /X /L /C:"%%E" "%TEMPFILE%" || (
echo(%%L
>> "%TEMPFILE%" echo(%%E
)
)
)
)
> nul 2>&1 del "%TEMPFILE%"
endlocal
exit /B
Every unique (non-empty) token left to the first = sign is written to a temporary file, which is searched after having read each line from the input file. If the token is already available in the temporary file, the line is skipped; if not, it is written to the output file.
The file content is treated in a case-insensitive manner, unless you remove the /I switch from the findstr command.
Update: Improved Scripts
Here are two scripts which are improved so that no special character can bring them to fail. They do not use temporary files. Both scripts remove lines with duplicate keywords (such is the partial string before the first = sign).
This script keeps the first line in case of duplicate keywords have been encountered:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "INFILE=%~1"
set "OUTFILE=%~2"
if not defined INFILE exit /B 1
if not defined OUTFILE exit /B 1
> "%OUTFILE%" break
for /F usebackq^ delims^=^ eol^= %%L in ("%INFILE%") do (
for /F tokens^=1^ delims^=^=^ eol^= %%E in ("%%L") do (
set "LINE=%%L"
set "KEY=%%E"
setlocal EnableDelayedExpansion
if not "!LINE:~,1!"=="=" (
set "KEY=!KEY: = !"
set "KEY=!KEY:\=\\!" & set "KEY=!KEY:"=\"!"
more /T1 "%OUTFILE%" | > nul 2>&1 findstr /I /M /B /L /C:"!KEY!=" || (
>> "%OUTFILE%" echo(!LINE!
)
)
endlocal
)
)
endlocal
exit /B
This script keeps the last line in case of duplicate keywords have been encountered:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "INFILE=%~1"
set "OUTFILE=%~2"
if not defined INFILE exit /B 1
if not defined OUTFILE exit /B 1
> "%OUTFILE%" (
for /F delims^=^ eol^= %%L in ('findstr /N /R "^" "%INFILE%"') do (
set "LINE=%%L"
for /F "delims=:" %%N in ("%%L") do set "LNUM=%%N"
setlocal EnableDelayedExpansion
set "LINE=!LINE:*:=!"
if defined LINE if not "!LINE:~,1!"=="=" (
for /F tokens^=1^ delims^=^=^ eol^= %%E in ("!LINE!") do (
setlocal DisableDelayedExpansion
set "KEY=%%E"
setlocal EnableDelayedExpansion
set "KEY=!KEY: = !"
set "KEY=!KEY:\=\\!" & set "KEY=!KEY:"=\"!"
more /T1 +!LNUM! "%INFILE%" | > nul 2>&1 findstr /I /M /B /L /C:"!KEY!=" || (
echo(!LINE!
)
endlocal
endlocal
)
)
endlocal
)
)
endlocal
exit /B
For both scripts, the following rules apply:
the order of lines with non-duplicate keywords is maintained;
empty lines are ignored and therefore removed;
empty keywords, meaning lines starting with =, are ignored and therefore removed;
non-empty lines that do not contain an = at all are treated as they would be ended with an = for the check for duplicates, hence the entire line is used as the keyword;
for the check for duplicates, each TAB character is replaced by a single SPACE;
every line that is transferred to the returned file is copied from the original file without changes (hence the aforementioned attachment of = or replacement of TAB is not reflected there);
the check for duplicates is done in a case-insensitive manner, unless you remove the /I switch from the findstr command;
Amendment: Processing Multiple Files
All of the above scripts are designed for processing a single file only. However, if you need to process multiple files, you could simply write a wrapper that contains a for loop enumerating all the input files and calls one of the scripts above (called rem-dups.bat) for every item -- like this:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem Define constants here:
set "INPATH=D:\Data\source" & rem (location of input files)
set "OUTPATH=D:\Data\target" & rem (location of output files)
set INFILES="source.txt" "test*.txt" & rem (one or more input files)
set "OUTSUFF=_no-dups" & rem (optional suffix for output file names)
set "SUBBAT=%~dp0rem-dups.bat"
pushd "%INPATH%" || exit /B 1
for %%I in (%INFILES%) do if exist "%%~fI" (
call "%SUBBAT%" "%%~fI" "%OUTPATH%\%%~nI%OUTSUFF%%%~xI"
)
popd
endlocal
exit /B
You must not specify the same locations for the input and output files. If you want to overwrite the original input files, you need to write the modified output files to another location first, then you can move them back to the source location -- supposing you have set OUTSUFF in the wrapper script to an empty string (set "OUTSUFF=" instead of set "OUTSUFF=_no-dups"). The command line to overwrite the original input files would be: move /Y "D:\Data\target\*.*" "D:\Data\source".
You could read the file into Excel without splitting it into multiple columns. Use Excel functionality to eliminate duplicates and save it back. You could do all this in VBScript.
Create an Excel Object
Loop
Load text file
Remove duplicates
Save text file
Until there are no more files
Dispose of the Excel Object
Code for the individual pieces should be easily available on the web. Do ask for any additional, specific, pointers you might need.

How to randomly rearrange lines in a text file using a batch file

I am creating a code that strips through different MAC addresses randomly, but cannot figure out how to do this. My thought on how to approach this is to randomize or rearrange the order of the MAC address in the text file with this script, but I cannot figure out how to do this with a batch file. How this will work is that it will read "maclist.txt", then create a new temp file with the random order "maclist_temp.txt", that will be the rearranged file. Then, it will pull this randomized file in order.
I have tried Google and searching the web, but I haven't found anything too useful. I'm still actively looking, but any advice would be extremely useful.
Something as simple as extracting and deleting a random line and then adding to the bottom might work. Randomization would be better though, but I want to keep the original list. Something like:
Make a temp copy of maclist.txt called maclist_temp.txt
Take one random MAC address, remove it from maclist_temp.txt
Readd it to the bottom
That is all I want, but any suggestions are welcome.
You may try this batch file to help you to shuffle your maclist.txt. The usage of the batch code is
C:\> type list.txt | shuffle.bat > maclist_temp.txt
Here are the contents of shuffle.bat:
#ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
SET TmpFile=tmp%RANDOM%%RANDOM%.tmp
TYPE NUL >%Tmpfile%
FOR /F "tokens=*" %%i IN ('MORE') DO SET Key=!RANDOM!!RANDOM!!RANDOM!000000000000& ECHO !Key:~0,15!%%i>> %TmpFile%
FOR /F "tokens=*" %%i IN ('TYPE %TmpFile% ^| SORT') DO SET Line=%%i&ECHO.!Line:~15!
::DEL %TmpFile%
ENDLOCAL
After issuing the above command, maclist_temp.txt will contain a randomized list of MAC addresses.
Hope this helps.
Here is a simpler method to randomize/randomise a file, no temp files needed. You can even reuse the same input filename.
Limitations are: blank lines and line starting with ; will be skipped, and lines starting with = will have all leading = signs stripped and ^ characters are doubled.
#echo off
setlocal
for /f "delims=" %%a in (maclist.txt) do call set "$$%%random%%=%%a"
(for /f "tokens=1,* delims==" %%a in ('set $$') do echo(%%b)>newmaclist.txt
endlocal
I really like foxidrive's approach. Nevertheless I want to provide a solution with all the listed limitations eliminated (although cmd-related restrictions like file sizes < 2 GiB and line lengths < ~ 8 KiB remain).
The key is delayed expansion which needs to be toggled to not lose explamation marks. This solves all the potential problems with special characters like ^, &, %, !, (, ), <, >, | and ".
The counter index has been implemented in order not to lose a single line of the original text file, which could happen without, because random may return duplicate values; with index appended, the resulting variable names $$!random!.!index! are unique.
The findstr /N /R "^" command precedes every line of the original file with a line number followed by a colon. So no line appears empty to the for /F loop which would ignore such. The line number also implicitly solves the issue with leading semicolons, the default eol character of for /F.
Finally, everything up to and including the first colon (remember the said prefix added by findstr) is removed from every line before being output, hence no more leading equal-to signs are dismissed.
So here is the code:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set /A "index=0"
for /f "delims=" %%a in ('findstr /N /R "^" "%~dpn0.lst"') do (
setlocal EnableDelayedExpansion
for /F %%b in ("$$!random!.!index!") do (
endlocal
set "%%b=%%a"
)
set /A "index+=1"
)
> "%~dpn0.new" (
for /f "delims=" %%a in ('set $$') do (
set "item=%%a"
setlocal EnableDelayedExpansion
echo(!item:*:=!
endlocal
)
)
endlocal
exit /B
This seems to work. Feed it a command line parameter of the file to randomize.
#echo off
setlocal EnableDelayedExpansion
rem read the number of lines in the file
rem the find prepends the line number so we catch blank lines
for /f "delims=" %%n in ('find /c /v "" %1') do set "len=%%n"
set len=!len:*: =!
rem echo %1 has %len% lines
rem Relocate as many lines as there are lines in the file
for /l %%j in (1 1 !len!) do (
rem echo starting round %%j
rem geta random number between 1 and the number of lines in the file
set /a var=!random! %% !len! + 1
rem echo relocating line !var!
rem make sure there is no temp file
if exist %1.temp del %1.temp
rem read each line of the file, write any that don't match and then write the one that does
<%1 (
for /l %%i in (1 1 !len!) do (
rem if it is the target line then save it
if %%i == !var! (
set /p found=
rem echo saving !found!
)
rem if it is the target line then write it
if not %%i == !var! (
set /p other=
rem echo writing !other!
echo !other!>> %1.temp
)
)
rem now write the target line at the end
rem echo appending !found!
echo !found!>> %1.temp
)
rem replace the original with the temp version
move %1.temp %1>nul
)
rem print the result
type %1
Place in cmd file
for /f "tokens=2 delims=/" %%m in ('cmd /e:on /v:on /c "for /f %%f in (maclist.txt) do #echo !random!/%%f" ^| sort') do echo %%m
It spawns a cmd which reads the mac list in the inner for, prefixes a random value and a slash to the mac and sorts the list. Then this list is splitted in the outter for using the slash as delimiter and printing the mac address.

Resources