Remove duplicate value in csv on combine - batch-file

Question

I encountered the below code online and modified this on my need.
I just wanna ask since i am new to batch file if there is a way to remove duplicate values after the combine.
#echo off
ECHO Set working directory
pushd %~dp0
ECHO Deleting existing combined file
del combined.csv
setlocal ENABLEDELAYEDEXPANSION
set cnt=1
for %%i in (*.csv) do (
if !cnt!==1 (
for /f "delims=" %%j in ('type "%%i"') do echo %%j >> combined.csv
) else if %%i NEQ combined.csv (
for /f "skip=1 delims=" %%j in ('type "%%i"') do echo %%j >> combined.csv
)
set /a cnt+=1
)

Answer 1

#ECHO OFF
SETLOCAL
ECHO Set working directory
pushd %~dp0
ECHO Deleting existing combined file
del combined.csv
set "flag="
for %%i in (*.csv) do if %%i NEQ combined.csv (
IF DEFINED flag (
findstr /l /x /v /g:combined.csv "%%i">#.vsc
TYPE #.vsc >>combined.csv
) ELSE (
COPY "%%i" combined.csv >nul
SET flag=y
)
)
DEL #.vsc /F /Q
POPD
GOTO :EOF
This may suit you better.
It uses a simple setlocal rather than the delayedexpansion version, initialising flag to empty then setting it within the loop and using if defined which works on the run-time value of flag.
First time through, it simply copies the detected source file to combined.csv and then sets flag to a value so it's ow defined
each other time through, findstr outputs those lines in the source file %%i that /v do not /x exactly match /l literally /g:filename any line in the combined.txt file TO a tempfile I'be nominated as #.vsc (name not important). Then that file is appended to combined.csv
Consequently, provided any particular .csv is free of duplicate lines within itself, the combined.csv will also be free of duplicate lines.
Since the header line is evidently identical in every file, the initial copy of the first file will place the header into combined.csv and hence findstr will neatly exclude it thereafter.
Revision to combat evil unicode:
#ECHO OFF
SETLOCAL
ECHO Set working directory
pushd %~dp0
ECHO Deleting existing combined file
del combined.csv
set "flag="
for %%i in (*.csv) do if %%i NEQ combined.csv (
(FOR /f "delims=" %%j IN ('type "%%i"') DO ECHO %%j)>#.vsc
IF DEFINED flag (
findstr /l /x /v /g:combined.csv "#.vsc" >##.vsc
TYPE ##.vsc>>combined.csv
) ELSE (
REN #.vsc combined.csv
SET flag=y
)
)
DEL #.vsc /F /Q
DEL ##.vsc /F /Q
POPD
GOTO :EOF
I suspect that the problem is using UNICODE within your files. Cutting-and-pasting your data showed that it was unicode.
The for /f... ceremony reads unicode and produces ASCII, hence this version first simply converts to ASCII using your familiar technique then operates on the converted file #.vsc. findstr does not appreciate outputting to the same file as it's attempting to read as /g:, so a further tempfile ##.vsc is used for the findstr output.
Note that the unicode characters between (header) Last modified and date and also elsewhere will be replaced by question-marks.