Page 1 of 1

Recommended duplicate file finder? (binary file compare)

Posted: Sat Jul 16, 2011 11:34 pm
by AbRASiON
Does anyone have a recommended application to scan multiple drives and find duplicate files?
Bonus points for something which presents the data in a way which shows entire folders which are the same.

I've used DFL in the past (DOS) and duplic8 for about 10 years but I need something more powerful.

Re: Recommended duplicate file finder? (binary file compare)

Posted: Sun Jul 17, 2011 1:19 pm
by drsauced
We've been using Beyond Compare for a couple of years at work, and it works fabulously.

http://www.scootersoftware.com/

Re: Recommended duplicate file finder? (binary file compare)

Posted: Sun Jul 17, 2011 3:43 pm
by bdwilcox
drsauced wrote:
We've been using Beyond Compare for a couple of years at work, and it works fabulously.

http://www.scootersoftware.com/

Do you know if Beyond Compare can compare not only the file itself, but also its metadata?

Re: Recommended duplicate file finder? (binary file compare)

Posted: Mon Jul 18, 2011 1:06 am
by drsauced
Ah, no it doesn't seem to be able to compare metadata for pictures. That'd be pretty useful...

I'll send them an email tomorrow and see what they say.

Re: Recommended duplicate file finder? (binary file compare)

Posted: Tue Jul 19, 2011 2:16 am
by Dirge
Beyond Compare looks like the Rolls Royce of duplicate file finders. Personally I am too cheap to pay for software like that, as I probably will only use it once or twice.

Here are some free alternatives, but you get what you paid for.

Auslogics Duplicate File Finder

CloneSpy

Duplicate Cleaner

Duplicate Files Finder

AllDUP

Re: Recommended duplicate file finder? (binary file compare)

Posted: Mon Aug 08, 2011 6:48 am
by AbRASiON
Tested clonespy, awful.
Working on dupe files finder and dupe cleaner now

Re: Recommended duplicate file finder? (binary file compare)

Posted: Wed Jan 11, 2012 3:49 am
by Arclight
AbRASiON wrote:
Does anyone have a recommended application to scan multiple drives and find duplicate files?
Bonus points for something which presents the data in a way which shows entire folders which are the same.

I've used DFL in the past (DOS) and duplic8 for about 10 years but I need something more powerful.


Sorry to hijack the thread but i was wondering how would you get so many duplicate files that would determine you to use a program to clean them up?

Re: Recommended duplicate file finder? (binary file compare)

Posted: Wed Feb 22, 2012 12:38 pm
by bitcat70
Here is something I used in the past: http://www.araxis.com/merge/index.html They offer a free 30 day trial.

Re: Recommended duplicate file finder? (binary file compare)

Posted: Wed Feb 22, 2012 1:37 pm
by LaChupacabra
What system are you running and what file attributes are you looking to compare? This could be a pretty simple script depending on what you're looking for

Re: Recommended duplicate file finder? (binary file compare)

Posted: Thu Feb 23, 2012 2:35 pm
by LaChupacabra
Just wanted to drop this in here. I whipped this up in Powershell (guessing you are using Windows). I created 2 directories with random files in them, then created similar folders in the directories (some the same and some different) and then moved files into those directories. Here's what I came up with

$var1 = @(Get-ChildItem c:\test1 -recurse)
$var2 = @(Get-ChildItem c:\test2 -recurse)
Compare-Object $var1 $var2 -includeequal -excludedifferent -passthru


Pretty straightforward really. What this does is builds an array from whatever directories you chose (in this case 2 folders called test1 and test2 in C:), goes the through those folders and any subfolders it finds, then compares the second directory against the first. This will find any files that are the same, regardless of folder, and put them into a list formated by directory. If you want to output it to a file, simply add this to the end

$var1 = @(Get-ChildItem c:\test1 -recurse)
$var2 = @(Get-ChildItem c:\test2 -recurse)
Compare-Object $var1 $var2 -includeequal -excludedifferent -passthru > c:\results.txt


and you will get a file in C: named results.txt that will have the results saved. It's quick and free. Best of all you can modify it however you'd like and achieve different results (try removing the -includeequal -excludedifferent and -passthru). Hope it helps!

Re: Recommended duplicate file finder? (binary file compare)

Posted: Sat Dec 08, 2012 6:55 am
by sjl
What I would do is run a hashing algorithm - MD5 or SHA1, either is good - across everything. I'd then run a compare on all files with matching hashes. No matching hash means no duplicates - guaranteed. Matching hash could just be a collision (albeit pretty unlikely), hence the full compare.

Won't pick up on pictures that are identical but for metadata (or that are just resized versions), but it'll catch just about everything else.

Re: Recommended duplicate file finder? (binary file compare)

Posted: Sat Dec 08, 2012 8:07 am
by nanoflower
I like the Duplicate File Finder from Rashid Hoda, http://rashid-hoda.software.informer.com/ That one has always worked well for me and you can do comparisons on combinations of name, binary contents, size plus the ability to choose directories and exclude directories.

Re: Recommended duplicate file finder? (binary file compare)

Posted: Fri Mar 22, 2013 3:53 am
by kroker
If you haven't seen this already, here is a good list of free duplicate file finders: http://www.techsupportalert.com/best-fr ... tector.htm