Batch script to add a line of text to files - batch-file

I am a novice and found some info here on howto make a script that write a text line to files. The result seems fine (also for multiple files), except that the text ends up at the bottom and I need it at the top.
#echo off
set "$New_line=TEXT"
for /r "C:\" %%A in (*.txt) do >"%%A" echo %$New_line%

#ECHO Off
SETLOCAL
rem The following settings for the source directory, destination directory, target directory,
rem batch directory, filenames, output filename and temporary filename [if shown] are names
rem that I use for testing and deliberately include names which include spaces to make sure
rem that the process works using such names. These will need to be changed to suit your situation.
SET "sourcedir=u:\your files"
set "$New_line=TEXT"
:: a filename that simply won't exist in the subtree and doesn't end ".txt"
SET "dummyfilename=##qqyhnggqvxjr.#$#"
FOR /F "delims=" %%b IN ('dir /b /s/a-d "%sourcedir%\*.txt"') DO (
SET "skipme="
FOR /F "delims=:" %%c IN ('FINDSTR /n /L /C:"%$new_line%" "%%b"') DO IF "%%c" equ "1" SET "skipme=Y"
IF NOT DEFINED skipme (
>"%%~dpb%dummyfilename%" ECHO %$new_line%
TYPE "%%b">>"%%~dpb%dummyfilename%"
MOVE /y "%%~dpb%dummyfilename%" "%%b" >NUL
)
)
GOTO :EOF
I really don't believe you'd want to prepend the text to every .txt file on your c: drive. You should set "sourcedir" to start the processing on a small sample subtree.
The findstr processing looks for %$new_line% as the first line of each file selected by the outer for loop and sets skipme if that condition is found. In this way, the processing is skipped for files that have already been processed.
Re: use of closing parenthesis in added line:
If there are closed-parentheses in the line to be inserted, you need to "escape" the ) with a caret ^ thus : set "$New_line=#Firm_Begin(16368^)"
The reason for this is that batch will render the escaped-) as
IF NOT DEFINED skipme (
>"%%~dpb%dummyfilename%" ECHO #Firm_Begin(16368^)
TYPE "%%b">>"%%~dpb%dummyfilename%"
MOVE /y "%%~dpb%dummyfilename%" "%%b" >NUL
)
and recognise the ^) as an escaped-) hence treat the ) as if it was any regular text character.
Without the escape character, batch interprets the command as
IF NOT DEFINED skipme (
>"%%~dpb%dummyfilename%" ECHO #Firm_Begin(16368)
TYPE "%%b">>"%%~dpb%dummyfilename%"
MOVE /y "%%~dpb%dummyfilename%" "%%b" >NUL
)
This has two consequences.
For files where skipme is not defined, #Firm_Begin(16368 is sent to the dummy file as the first and only line, and the text of the entire file is then added and then fortunately the resultant is moved over the original file so that the dummy file does not exist.
In files where skipme is defined however, only the ...ECHO %$new_line% is skipped. The original file is then typed into the non-existent dummy file, and then the dummyfile overwrites the original - so no damage done....
...Or is there?
findstr...C:"%$new_line%"... discriminates whether to select the file to have the foreword added or not, via the flag skipme. Unfortunately, the value of $new_line now contains ^ and in this instance the ^ does not escape the ) because the /c:"expression" is quoted (as it has to be since the string-value contains spaces), so its contents are taken literally. findstring will search for the literal string #Firm_Begin(16368^), not #Firm_Begin(16368).
cure for this is to remove any ^ from $new_line, so
...'FINDSTR /n /L /C:"%$new_line:^=%" "%%b"'...
and add the /x switch as a refinement so that the string must match the entire line, hence
...
FOR /F "delims=" %%b IN ('dir /b /s/a-d "%sourcedir%\*.txt"') DO (
SET "skipme="
FOR /F "delims=:" %%c IN ('FINDSTR /n /x /L /c:"%$new_line:^=%" "%%b"') DO IF "%%c" equ "1" SET "skipme=Y"
IF NOT DEFINED skipme (
>"%%~dpb%dummyfilename%" ECHO %$new_line%
TYPE "%%b">>"%%~dpb%dummyfilename%"
MOVE /y "%%~dpb%dummyfilename%" "%%b" >NUL
)
)
GOTO :EOF
which should cure the problem.
But then there's the clean-up, where \x can be used to detect files which have #Firm_Begin(16368 as the first line, not #Firm_Begin(16368).
[this part I haven't tested, and unfortunately have no time to test atm]
FOR /F "delims=:" %%c IN ('FINDSTR /n /x /L /c:"%$new_line:~0,-3" "%%b"') DO IF "%%c" equ "1" echo incorrect headline in "%%b"
should indicate the files that suffer that condition. All that would need to be done for any such file detected would be to initialise the dummy file with the correct top line and then use >>"%dummyfile%" for /f "usebackqskip=1delims=" %%L in ("%%b") do echo %%L to append the contents of the file %%b , all but the first line to the dummy file.

Related

Can I use the variable name specified in the do syntax immediately? In batch file [duplicate]

This question already has an answer here:
Variables are not behaving as expected
(1 answer)
Closed 3 months ago.
I want to find file list PATH1
and overwrite it if there is a file with the same name on PATH2.
#echo off
SET PATH1="FIND_PATH"
SET PATH2="OVERWRITE_PATH"
for /f "delims=" %%A in ('dir /b %PATH1%') do (
dir /s /b /r %PATH2% | find /i "%%A" > list.txt
SET /p PATH3<=list.txt
move %PATH1%%%A %PATH3%
)
At line 7,
If possible, it would be nice to store it in a variable without going through list.txt in the middle.
I perform "ECHO %PATH3%" outside the for syntax, the last saved item is output.
but I perform "ECHO %PATH3%" inside the do syntax, Outputs an error.
how can I do.
thx.
Here's my analysis of your code:
for /f "delims=" %%A in ('dir /b %PATH1%') do (
rem %%A contains filename + extension of each file AND DIRECTORY in path1
dir /s /b /r %PATH2% | find /i "%%A" > list.txt
rem LIST.txt contains full pathnames of all filenames AND DIRECTORIES
rem where "%%A" forms part of the full pathname
SET /p PATH3<=list.txt
rem PATH3 contains the first line of list.txt
move %PATH1%%%A %PATH3%
rem move `path1%%A` (a file or directoryname - but without the `\` separator)
rem to the first (probably filename) found
)
Comment : first time in using batch for the last 40 years that I've ever seen the /r swich used in a dir statement...
So - we have to make some reasonable assumptions as to the intent of this code.
Tip: Use set "var=value" for setting string values - this avoids problems caused by trailing spaces. Don't assign " or a terminal backslash or Space. Build pathnames from the elements - counterintuitively, it is likely to make the process easier. If the syntax set var="value" is used, then the quotes become part of the value assigned.
#echo off
SET "PATH1=FIND_PATH"
SET "PATH2=OVERWRITE_PATH"
for /f "delims=" %%b in ('dir /b /a-d "%PATH1%"') do (
for /f "delims=" %%u in ('dir /s /b /a-d /r "%PATH2%\%%b" 2^>nul') do (
ECHO move "%PATH1%\%%A" "%%u"
)
)
[1] Change to better set syntax
[2] Prefer to avoid ADFNPSTXZ (in either case) as metavariables (loop-control variables) as ADFNPSTXZ are metavariable-modifiers which can lead to difficult-to-find bugs (See for/f from the prompt for documentation)
[3] Include /a-d to exclude directory names from dir lists
[4] Include %%b in the filemask for the inner for to search for that exact name only in the destination directory-tree
[5] The 2^>nul suppresses an error message should no files be found in the destination-tree matching the filename. The caret (^) is required to tell cmd that the > is part of the dir command, not the for
[6] Append the filename found (%%b) to the source directoryname with required separator
[7] %%u will contain the full pathnames of the matching files in the destination directory-tree
[8] The resultant command is merely echoed for verification. After verification, remove the echo keyword to actually execute the move.
Untested
Always verify against a test directory before applying to real data.
The effect would be to find all of the filenames in the source directory and move it over all matching filenames in the destination directory tree.
LOGICAL FLAW
a "move" command would move a file a.txt over the first a.txt in the destination-tree. Then the source file no longer exists (as it's been moved) so if a second a.txt is located in the destination-tree, then the move will fail.
Consider this further modification:
#echo off
SET "PATH1=FIND_PATH"
SET "PATH2=OVERWRITE_PATH"
for /f "delims=" %%b in ('dir /b /a-d "%PATH1%"') do (
set "moved="
for /f "delims=" %%u in ('dir /s /b /a-d /r "%PATH2%\%%b" 2^>nul') do (
set "moved=y"
ECHO COPY "%PATH1%\%%A" "%%u"
)
if defined moved (
ECHO del "%PATH1%\%%A"
) else (
ECHO "%PATH1%\%%A" - no matches in tree "%PATH2%" found
ECHO MOVE "%PATH1%\%%A" "%PATH2%"
)
)
This change sets moved to nothing for each source file found, then sets it to y if any matching filename is found n the destination tree.
If the file has been copied, then delete the source file, otherwise complain and move it to the destination-tree-root.

Remove the last line from the text file (including empty lines) using batch

My text file
Header
A,B,C,D,E
F,G,H,I,J
K,L,M,N,O
Footer
I want remove the Footer and also empty lines below Footer (Empty lines are not static) And my expected output
Header
A,B,C,D,E
F,G,H,I,J
K,L,M,N,O
I tried the below code, But it removing the last empty line alone.
set row=
for /F "delims=" %%j in (file.txt) do (
if defined row echo.!row!>> newfile.txt
set row=%%j
)
This would be a generic way to perform the task; If you want something different consider providing more specific information and example file(s):
#Echo Off
SetLocal DisableDelayedExpansion
Set "SrcFile=file.txt"
If Not Exist "%SrcFile%" Exit /B
Copy /Y "%SrcFile%" "%SrcFile%.bak">Nul 2>&1||Exit /B
( Set "Line="
For /F "UseBackQ Delims=" %%A In ("%SrcFile%.bak") Do (
SetLocal EnableDelayedExpansion
If Defined Line Echo !Line!
EndLocal
Set "Line=%%A"))>"%SrcFile%"
EndLocal
Exit /B
You should change your filename on line 4 to match your actual source file's name. If you are happy with the result you can optionally delete the .bak file, (which is a backup of the original file, saved for safety).
Note: The resultant file will end with the normal CRLF, (i.e there will be a blank line at the bottom).
What about a simple:
FINDSTR /R /I /V "^$ Footer" file.txt>>newfile.txt
?
/R means regexp
/I means case insensitive
/V means EXCLUDE matches, instead of showing
in the double quotes are (separated by space) all the regexps that will be applied:
^$ matches empty lines
Footer matches the exact word Footer
pretty straight forward: read until you hit the footer, then quit (removes the Footer and anything after):
#echo off
for /F "delims=" %%j in (file.txt) do (
echo %%j|findstr /b "Footer" >nul && goto :done
echo %%j>> newfile.txt
)
:done

searching for text file(s) and adding a line in them with batch file

I am trying to write a batch file that can be used for the following task - search for certain text files and adding a text line in them - below the specified line. I am trying to use the following approach:
dir /s /a /b "%SystemDrive%\config*.ini" >> %userprofile%\temp.txt
for /F "tokens=* delims=," %%G IN (%userprofile%\temp.txt) DO (echo "%%G" >> %userprofile%\temp2.txt)
for /F "tokens=* delims=," %%G IN (%userprofile%\temp2.txt) DO (
SETLOCAL ENABLEDELAYEDEXPANSION
cd %%~dG%%~pG
set inputFile=%%~nG%%~xG
set outputFile=in.tmp
set _strInsert=TCPPortNumber=870
set _strFind=Random=No
FOR /F "usebackq delims=" %%A IN ("%inputFile%") DO (
Echo %%A | Find "%_strFind%" && ECHO %%A>>"%outputFile%" && ECHO %_strInsert%>>"%outputFile%"
IF [!errorlevel!] == [1] ECHO %%A>>"%outputFile%"
)
MOVE /Y "%outputFile%" "%inputFile%" && DEL /F /Q "%outputFile%"
Searching for config*.ini file(s) and writing results into temp.txt. Inserting quotes in case there are spaces in file paths for each line in temp.txt -> temp2.txt. Then for each file path in temp2.txt trying to insert TCPPortNumber=870 below the line Random=No. If I use a "hard" file path this approach works, but with reading the files path from temp2.txt it gives me an error: The system cannot find the file specified.
Can anyone help?
SETLOCAL ENABLEDELAYEDEXPANSION
dir /s /a /b "%SystemDrive%\config*.ini" >> %userprofile%\temp.txt
set outputFile=in.tmp
set _strInsert=TCPPortNumber=870
set _strFind=Random=No
for /F "usebackqdelims=" %%G IN ("%userprofile%\temp.txt") DO (
pushd %%~dpG
FOR /F "usebackq delims=" %%A IN ("%%~nxG") DO (
ECHO %%A>>"%outputFile%"
Echo %%A | Find "%_strFind%" && >>"%outputFile%" ECHO %_strInsert%
)
MOVE /Y "%outputFile%" "%%~nxG"
popd
)
endlocal
The setlocal command should not be within the loop (unless it is matched by an endlocal) because it is not a switch, but establishes a frame, hence within the loop, cmd is establishing many nested local environments.
The three constant strings also should appear outside of the loop since their values do not change.
The second tempfile is not required as it is simply the first with each line quoted.
Since %temp% may contain spaces, the for/f ... %%Gneeds to have the filename quoted, and consequently theusebackqoption. The line is to be accepted into%%Gin its entirity, sodelims=` is required, turning delimiters off.
The substring selectors may be combined, so ~dp delivers the drive and path and ~nx the name and extension.
In the original code, inputfile was being set up, but %inputfile% would be resolved to the value of inputfile at the time the entire block (parenthesised sequence of lines) was parsed not at run-time as the values change through execution of the code. This is the commonly-encountered delayed expansion problem - hundreds of references here on SO.
Since what appears to be needed is to insert a line after a particular line, we can simply reproduce each line then see whether the insertion is required, which simplifies the code. The placement of the redirector to before the echo overcomes the Windows-NT characteristic of selecting the device to be redirected by preceding the redirector with a device number.
Finally, move /y forces the replacement of the input file with the output file, so the output file will no longer exist
I haven't actually tried this code - you should execute it against a small copied subtree of the real data and evaluate it before relying on it.
small fix - changed original cd to pushd and added corresponding popd also removed superfluous %
PUSHD changes the current directory to that specified. POPD restores the original current-directory.
The extra % was a keying error (my normal machine is ill and I have to use the laptop - and boy, do I loathe the keyboard...)

copy & rename files to other folder

I have some of log files formatted like this "name.log"
I would like to copy those from one folder to another folder like
xcopy /y "C:\Folder1" "D:\Folder2"
And i need to rename file with created date of original file (no copy file) so that the text file in Folder2 would be like "name yyyymmddhhmm.log" if some file has the same name (date of creation) it will be overwritten.
The code:
set Source=C:\Users\user1\Desktop\Folder1
set Dest=D:\Folder2
if not exist %Dest% md %Dest%
for /F %%a in ('dir /b "%Source%\*.txt"') do call :Sub %%a
goto :eof
:Sub
set "filename=%1"
for /F %%s in ("%Source%\%1") do if %%~zs==0 goto :eof
set "datepart="
FOR /F "tokens=1-5 delims=/-: " %%a IN ('dir /tc "%filename%" ^| findstr "%filename%"') DO (
IF "%%c" neq "" SET "datepart=%%c%%a%%b%%d%%e"
)
FOR /F %%a IN ("%filename%") DO (
set "NewName=%%~na %datepart%%%~xa"
)
xcopy /y "%Source%\%filename%" "%Dest%\%NewName%*"
GOTO :EOF
The problem is that If I don't put the .bat in the same folder that origin files (Folder1),the files aren't change name. For example, if it is out, the files change name with old name and one white space.
The command windows tell me that it doesn't find the file when it get the creation date.
If I put the script into folder1 it works well.
On the other hand, if I execute the script with "Task Scheduler" I have the same problem. The files are copied but without date of creation.
What do I need to solve this problem?
#ECHO OFF
SETLOCAL
set Source=C:\Users\user1\Desktop\Folder1
set Dest=D:\Folder2
set "Source=u:\sourcedir\t w o"
set "Dest=u:\destdir"
if not exist "%Dest%" md "%Dest%"
for /F "delims=" %%k in ('dir /b "%Source%\*.log"') do call :Sub "%%k"
goto :eof
:Sub
SET "newname=%~1"
for /F "delims=" %%s in ("%Source%\%~1") do (if %%~zs==0 goto :eof
FOR /F "tokens=1-5 delims=/-: " %%a IN ('dir /tc "%Source%\%~1" ^| findstr "%~1"') DO (
IF "%%c" neq "" SET "newname=%%~ns %%c%%a%%b%%d%%e%%~xs"
)
)
ECHO(xcopy /y "%Source%\%~1" "%Dest%\%NewName%"
GOTO :EOF
The required XCOPY commands are merely ECHOed for testing purposes. After you've verified that the commands are correct, change ECHO(XCOPY to XCOPY to actually copy the files. Append >nul to suppress report messages (eg. 1 file copied)
This may seem quite a radical change, but actually it really isn't.
First issue is that I overrode your directory settings with directories to suit my system. The syntax SET "var=value" (where value may be empty) is used to ensure that any stray trailing spaces are NOT included in the value assigned. set /a can safely be used "quoteless".
Using quotes in the md command makes the procedure immune to "paths containing separators" - I test using spaces in paths and filenames because that appears to be a popular thing to do.
I changed the directory-scan metavariable from %%a to %%k to avoid confusion with the %%a in the subroutine. Your text says that you are starting with &.log files, but your filemask was *.txt so I changed it to *.log. Quoting the parameter delivered to :Sub means the procedure will receive the entire name of the file if it contains spaces.
Within the subroutine, it would appear that yowant no name-change if the %%c part from the dir/tc scan is empty. %~1 is the supplied filename minus the quotes.
The outer loop in %%s : I added delims= to cater for spaces in filenames and used %~1 in preference to %filename%
Within the %%s block, %%s refers to the file, so you can use %%s and its modified forms like %%~zs to refer to that file's characteristics - which unfortunately do not include create-date (%%~ts contains the last-update date - you may be able to use that in te following line rather than dir and findstr)
Then as #aschipfi suggested, include the source directory in the dir otherwise the dir takes place on the current directory.
FOR /F "tokens=1-5 delims=/-: " %%a IN ("%%~ts") DO (
should work for you if you can use last-update-date in place of create-date.
So - if %%c is not empty, set the new name to (the name part of the file in %%s)+space+the date string+(the extension in %%s)
And then do the xcopy - using the old name unless it was modified.

how to remove duplicate entry from the text file using batch script or vb script?

how can I remove the duplicate entry from the text file using batch script. All i want to remove the duplicates before "=" sign and "%%" is exist in every single text file. Text file look likes below
%%B05AIPS_CDDOWNLOAD_IBDE_UNC=\\%%B05AIPS_UPLOAD_NODE.\F$\DATA\IPSL\CDFILES\B05_NAG\CD\INCOMING
%%B05AIPS_CDDOWNLOAD_FTS_UNC=\\%%B05AIPS_UPLOAD_NODE.\B05_NAG\FTS\To_Clearpath\%%DATE_CCYYMMDD.
%%B05AIPS_CDDOWNLOAD_FTS_UNC=%%B05AIPS_CDDOWNLOAD_FTS_UNC.
I got about 30 plus different text files which contains above kind of entries and want to remove the duplicate line and want to keep the first occurrence. Remember duplicate line should be identified before "=" sign only and removal required for the entire line.Each of the different text files have got "%%" sign. Please guide me if there is way to do through batch script or vbscript? Thanks
Here is a simple batch-file solution; let us call the script rem-dups.bat. Supposing your input file is test.txt and your output file is result.txt, you need to provide these files as command line arguments, so you need to call it by: rem-dups.bat "test.txt" "results.txt". Here is the script:
#echo off
setlocal EnableExtensions EnableDelayedExpansion
set "INFILE=%~1"
set "OUTFILE=%~2"
if not defined INFILE exit /B 1
if not defined OUTFILE set "OUTFILE=con"
for /F "usebackq tokens=1,* delims==" %%K in ("%INFILE%") do (
set "LEFT=%%K"
set "RIGHT=%%L"
set "LEFT=!LEFT:*%%%%=__!"
rem Remove `if` query to keep last occurrence:
if not defined !LEFT! set "!LEFT!=!RIGHT!"
)
> "%OUTFILE%" (
for /F "delims=" %%F in ('set __') do (
set "LINE=%%F"
echo(!LINE:*__=%%%%!
)
)
endlocal
exit /B
The script is based on the fact that there cannot occur duplicate environment variables, that are such with equal names.
This code only works if the following conditions are fulfilled:
the file content is treated in a case-insensitive manner;
the order of lines in the output file does not matter;
the partial strings before the first = sign start with %% and contain at least one more character other than %;
the partial strings before the first = contain only characters which may occur within environment variable names, besides the leading %%;
the partial strings after the first = must not be empty;
the partial strings after the first = must not start with = on their own;
no exclamation marks ! are allowed within the file, because they may get lost or lead to other unexpected results;
Here is an alternative method using a temporary file:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "INFILE=%~1"
set "OUTFILE=%~2"
if not defined INFILE exit /B 1
if not defined OUTFILE set "OUTFILE=con"
set "TEMPFILE=%TEMP%\%~n0_%RANDOM%.tmp"
> "%TEMPFILE%" break
> "%OUTFILE%" (
for /F usebackq^ delims^=^ eol^= %%L in ("%INFILE%") do (
for /F tokens^=1^,*^ delims^=^=^ eol^= %%E in ("%%L") do (
> nul 2>&1 findstr /I /X /L /C:"%%E" "%TEMPFILE%" || (
echo(%%L
>> "%TEMPFILE%" echo(%%E
)
)
)
)
> nul 2>&1 del "%TEMPFILE%"
endlocal
exit /B
Every unique (non-empty) token left to the first = sign is written to a temporary file, which is searched after having read each line from the input file. If the token is already available in the temporary file, the line is skipped; if not, it is written to the output file.
The file content is treated in a case-insensitive manner, unless you remove the /I switch from the findstr command.
Update: Improved Scripts
Here are two scripts which are improved so that no special character can bring them to fail. They do not use temporary files. Both scripts remove lines with duplicate keywords (such is the partial string before the first = sign).
This script keeps the first line in case of duplicate keywords have been encountered:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "INFILE=%~1"
set "OUTFILE=%~2"
if not defined INFILE exit /B 1
if not defined OUTFILE exit /B 1
> "%OUTFILE%" break
for /F usebackq^ delims^=^ eol^= %%L in ("%INFILE%") do (
for /F tokens^=1^ delims^=^=^ eol^= %%E in ("%%L") do (
set "LINE=%%L"
set "KEY=%%E"
setlocal EnableDelayedExpansion
if not "!LINE:~,1!"=="=" (
set "KEY=!KEY: = !"
set "KEY=!KEY:\=\\!" & set "KEY=!KEY:"=\"!"
more /T1 "%OUTFILE%" | > nul 2>&1 findstr /I /M /B /L /C:"!KEY!=" || (
>> "%OUTFILE%" echo(!LINE!
)
)
endlocal
)
)
endlocal
exit /B
This script keeps the last line in case of duplicate keywords have been encountered:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "INFILE=%~1"
set "OUTFILE=%~2"
if not defined INFILE exit /B 1
if not defined OUTFILE exit /B 1
> "%OUTFILE%" (
for /F delims^=^ eol^= %%L in ('findstr /N /R "^" "%INFILE%"') do (
set "LINE=%%L"
for /F "delims=:" %%N in ("%%L") do set "LNUM=%%N"
setlocal EnableDelayedExpansion
set "LINE=!LINE:*:=!"
if defined LINE if not "!LINE:~,1!"=="=" (
for /F tokens^=1^ delims^=^=^ eol^= %%E in ("!LINE!") do (
setlocal DisableDelayedExpansion
set "KEY=%%E"
setlocal EnableDelayedExpansion
set "KEY=!KEY: = !"
set "KEY=!KEY:\=\\!" & set "KEY=!KEY:"=\"!"
more /T1 +!LNUM! "%INFILE%" | > nul 2>&1 findstr /I /M /B /L /C:"!KEY!=" || (
echo(!LINE!
)
endlocal
endlocal
)
)
endlocal
)
)
endlocal
exit /B
For both scripts, the following rules apply:
the order of lines with non-duplicate keywords is maintained;
empty lines are ignored and therefore removed;
empty keywords, meaning lines starting with =, are ignored and therefore removed;
non-empty lines that do not contain an = at all are treated as they would be ended with an = for the check for duplicates, hence the entire line is used as the keyword;
for the check for duplicates, each TAB character is replaced by a single SPACE;
every line that is transferred to the returned file is copied from the original file without changes (hence the aforementioned attachment of = or replacement of TAB is not reflected there);
the check for duplicates is done in a case-insensitive manner, unless you remove the /I switch from the findstr command;
Amendment: Processing Multiple Files
All of the above scripts are designed for processing a single file only. However, if you need to process multiple files, you could simply write a wrapper that contains a for loop enumerating all the input files and calls one of the scripts above (called rem-dups.bat) for every item -- like this:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem Define constants here:
set "INPATH=D:\Data\source" & rem (location of input files)
set "OUTPATH=D:\Data\target" & rem (location of output files)
set INFILES="source.txt" "test*.txt" & rem (one or more input files)
set "OUTSUFF=_no-dups" & rem (optional suffix for output file names)
set "SUBBAT=%~dp0rem-dups.bat"
pushd "%INPATH%" || exit /B 1
for %%I in (%INFILES%) do if exist "%%~fI" (
call "%SUBBAT%" "%%~fI" "%OUTPATH%\%%~nI%OUTSUFF%%%~xI"
)
popd
endlocal
exit /B
You must not specify the same locations for the input and output files. If you want to overwrite the original input files, you need to write the modified output files to another location first, then you can move them back to the source location -- supposing you have set OUTSUFF in the wrapper script to an empty string (set "OUTSUFF=" instead of set "OUTSUFF=_no-dups"). The command line to overwrite the original input files would be: move /Y "D:\Data\target\*.*" "D:\Data\source".
You could read the file into Excel without splitting it into multiple columns. Use Excel functionality to eliminate duplicates and save it back. You could do all this in VBScript.
Create an Excel Object
Loop
Load text file
Remove duplicates
Save text file
Until there are no more files
Dispose of the Excel Object
Code for the individual pieces should be easily available on the web. Do ask for any additional, specific, pointers you might need.

Resources