compare file content in folder using windows batch - batch-file

Question

I've been struggling with this for several days... there is one folder with a lot txt files with random names that are generated from server timestamps, but content of files must not be identical for two files in that folder! any ideas? my only option is using windows batch

Answer 1

#ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir"
PUSHD "%sourcedir%"
FOR %%a IN (*.*) DO (
FOR %%c IN (*.*) DO IF /i "%%~nxa" lss "%%~nxc" IF "%%~za"=="%%~zc" (
FC "%%a" "%%c" >NUL
IF NOT ERRORLEVEL 1 ECHO "%%a" and "%%c" are identical
)
)
GOTO :EOF
You would need to change the setting of sourcedir and of the filemask *.* to suit your circumstances.
Revision for only-one-mention-of-a-duplicate-file
#ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir"
PUSHD "%sourcedir%"
FOR %%a IN (*.*) DO (
SET "reported="
FOR %%c IN (*.*) DO IF NOT DEFINED reported IF /i "%%~nxa" lss "%%~nxc" IF "%%~za"=="%%~zc" (
FC /b "%%a" "%%c" >NUL
IF NOT ERRORLEVEL 1 ECHO "%%a" and "%%c" are identical&SET reported=Y
)
)
GOTO :EOF
I've also added /b to fc to allow for non-text files.

Answer 2

The solution below process the list of file names just once, so it should run faster.
#echo off
setlocal EnableDelayedExpansion
for %%a in (*.txt) do (
if not defined size[%%~Za] (
set size[%%~Za]="%%a"
) else (
set newName="%%a"
for %%b in (!size[%%~Za]!) do (
fc "%%a" %%b >NUL
if not errorlevel 1 (
echo "%%a" and %%b are identical
set "newName="
)
)
if defined newName set "size[%%~Za]=!size[%%~Za]! !newName!"
)
)
If two files are identical, the name of the second one is not saved in the lists, so it is not compared again vs. other same size files. If no more than two files may be identical, then this method could be modified so the name of the first file be also removed from the lists (below the echo ... are identical command), so the method be even faster.
If you may download a third party program that calculate the MD5 checksum, then it may be used to check if two files are identical instead of fc command as foxidrive suggested. This would be faster because the MD5 checksum of each file would be calculated just once and stored in another array (with the file name as index).

Answer 3

As this code uses certutil, this will work only for windows Vista or later versions of the OS. This will check for duplicates in files of the same size and will only read each involved file only once.
#echo off
rem Configure environment
setlocal enableextensions disabledelayedexpansion
rem Where to search for files
set "folder=%cd%"
rem We need a temporary file to hold the size sorted list of files
set "tempFile=%temp%\%~nx0.%random%%random%%random%.tmp"
rem Change to target folder and work from here
pushd "%folder%"
rem Retrieve the list of files with its size and set a environment variable
rem named as the size of the file. The value of this variable will hold the
rem number of files with this size
(for /f "delims=" %%a in ('dir /a-d /b /os *') do (
echo \%%~za\%%a\
set /a "sz_%%~za+=1"
)) >"%tempFile%"
rem Retrieve the list of sizes that happens more than one time
for /f "tokens=2,3 delims=_=" %%a in ('set sz_') do if %%b gtr 1 if %%a gtr 0 (
rem Retrive the list of files with the indicated size
setlocal
for /f "tokens=1,2 delims=\" %%c in ('findstr /l /b /c:"\%%a\\" "%tempFile%"') do (
set "hash="
for /f "skip=1 delims=" %%e in ('certutil -hashfile "%%d"') do if not defined hash (
rem For each file, compute its hash. This hash is used as a variable name.
rem If the variable is defined, a previous file has the same size and hash
rem so it is a duplicate
set "hash=1"
if defined "%%e" (
<nul set /p ".=%%d = "
setlocal enabledelayedexpansion
echo(!"%%e"!
endlocal
) else (
rem Store the name of the file in a variable named as the hash of the file
set ""%%e"=%%d"
)
)
)
endlocal
rem This inner setlocal/endlocal ensures there is no collision between hashes for
rem files with different sizes
)
rem Cleanup
popd
del /q "%tempFile%" >nul 2>nul
endlocal
edited For a simplified version with no temporary file (the list is created in memory) while still reading only the needed files only once each file, AND as demanded a more readable output
edited again to correct a problem with the output of different groups of duplicated for the same file size
#echo off
setlocal enableextensions disabledelayedexpansion
set "folder=%~1"
if not defined folder set "folder=%cd%"
pushd "%folder%"
for /f "delims=" %%a in ('dir /a-d /b /os *') do (
set /a "sz_%%~za+=1"
setlocal enabledelayedexpansion
for /f "delims=" %%b in ("!fl_%%~za! ") do (endlocal & set "fl_%%~za=%%b "%%a"")
)
for /f "tokens=2,3 delims=_=" %%a in ('set sz_') do if %%b gtr 1 (
setlocal & setlocal enabledelayedexpansion
for /f "delims=" %%c in ("!fl_%%a!") do (
endlocal
for %%d in (%%~c) do (
if %%a equ 0 ( set "hash=0" ) else (
set "hash="
for /f "skip=1 delims=" %%e in ('certutil -hashfile "%%~d"') do if not defined hash set "hash=%%e"
)
setlocal enabledelayedexpansion
for /f "delims=" %%e in ("!hash!") do if defined hash_"%%~e" (
for /f "delims=" %%z in ("!hash_"%%~e"!") do (endlocal & set "hash_"%%~e"=%%z"%%~d";")
) else (
endlocal & set "hash_"%%~e"="%%~d"="
)
)
)
for /f "tokens=1,* delims==" %%c in ('set hash_ 2^>nul^|find ";"') do (
set "first=1"
for %%e in (%%d) do if defined first (set "first=" & echo(%%e) else (echo( = %%e)
)
endlocal
)
popd
endlocal
exit /b