Remove duplicate files in a directory - batch-file

Question

I have a working script, (below), which compares files, by size only, and if one or more of the files have different size, it shows on the .bat screen.
#echo off
setlocal EnableDelayedExpansion
rem Group all file names by size
for /R %%a in (*.*) do (
set "size[%%~Za]=!size[%%~Za]!,%%~Fa"
)
rem Show groups that have more than one element
for /F "tokens=2,3* delims=[]=," %%a in ('set size[') do (
if "%%c" neq "" echo [%%a]: %%b,%%c
)
pause
But I need more!
I also need the script delete duplicate files, keeping just one. For example, if there are 3 files of the same size but with different names, the script should choose one to keep, deleting the other 2.
#LotPings, i tried with 2 .pdf files, and appeared this message in the pic below:
#LotPings
Now it recognized fine! But it doesn't deleting the duplicate, just recognizing.... Look the image below:

Answer 1

I think treating files with same size as same content is a bit naive.
If in doubt better do a binary compare with fc.
Your way of stuffing %%~Fa into a string without quoting every single name possibly containing spaces can be problematic in further processing.
If the files are identical you could create links.
You know that for /r will recurse into subdirs?
The following batch will process the size[] array differently passing the size and content to a subroutine which keeps the first file and (only echos) the del command.
:: Q:\Test\2018\06\06\SO_50723488.cmd
#echo off & setlocal
rem Group all file names by size
for /R %%a in (*.*) do call set size[%%~Za]=%%size[%%~Za]%%,"%%~Fa"
rem Process groups
for /F "tokens=2* delims=[]=," %%a in ('set size[') do Call :Sub %%a %%b
pause
Goto :Eof
:Sub
If "%~3"=="" (Set "size[%1]="&goto :EOf)
Echo ========================================
Echo processing %*
Echo Keep %2
Shift&shift
:loop
Echo Del %1
if not "%~2"=="" (shift&goto :loop)
Sample output:
> Q:\Test\2018\06\06\SO_50723488.cmd
========================================
processing 0 "Q:\Test\2018\05\03\Text3.txt","Q:\Test\2018\05\27\log.html"
Keep "Q:\Test\2018\05\03\Text3.txt"
Del "Q:\Test\2018\05\27\log.html"
========================================
processing 14 "Q:\Test\2018\05\03\Text1.txt","Q:\Test\2018\05\15\HP-G1610-20180515.json"
Keep "Q:\Test\2018\05\03\Text1.txt"
Del "Q:\Test\2018\05\15\HP-G1610-20180515.json"
========================================
processing 21 "Q:\Test\2018\05\12\text1.txt","Q:\Test\2018\05\12\text2.txt"
Keep "Q:\Test\2018\05\12\text1.txt"
Del "Q:\Test\2018\05\12\text2.txt"
========================================
processing 22 "Q:\Test\2018\05\20\2.txt","Q:\Test\2018\05\22\version.ps1"
Keep "Q:\Test\2018\05\20\2.txt"
Del "Q:\Test\2018\05\22\version.ps1"
========================================
processing 288 "Q:\Test\2018\05\11\RawData.txt","Q:\Test\2018\05\23\SO_50478080.ps1"
Keep "Q:\Test\2018\05\11\RawData.txt"
Del "Q:\Test\2018\05\23\SO_50478080.ps1"

Answer 2

The following script removes duplicate files in the current directory. It leaves one file, and removes rest of its duplicates. I developed the script for Windows 10, so it might not work for other Windows versions.
The advantage of this script is it not only checks the last modified date and time, but also the file size. Let's say you have a text file with *** symbols? you can turn them all ### symbols, and still occupy the same space. The script checks the last modified date, and time along with its size to ensure both the files are identical, otherwise it doesn't remove any file.
setlocal enabledelayedexpansion
echo(
echo "This program removes duplicate files in the current directory."
echo "Warning: Take a backup of the current directory to avoid accidents"
PAUSE
echo(
set path=%cd%
set /A count=0
for %%a in (*.*) do (
for %%b in (*.*) do (
if exist "%%b" if %%a neq %%b (
if "%%~Ta %%~Za" equ "%%~Tb %%~Zb" (
echo "%%a File is deleted"
set /A count+=1
del "%%a"
)
)
)
)
echo(
echo "---------"
echo "!count! Duplicate Files were deleted in the current directory"
echo "---------"
echo(
endlocal
PAUSE
If you have any problems report me at
https://github.com/donqq/Duplicate-File-Remover/