Recommended duplicate file finder? (binary file compare)

All things storage here: hard drives, DVD RW drives, little wicker baskets.

Moderators: morphine, Steel

Recommended duplicate file finder? (binary file compare)

Postposted on Sat Jul 16, 2011 11:34 pm

Does anyone have a recommended application to scan multiple drives and find duplicate files?
Bonus points for something which presents the data in a way which shows entire folders which are the same.

I've used DFL in the past (DOS) and duplic8 for about 10 years but I need something more powerful.
AbRASiON
Gerbil Team Leader
 
Posts: 274
Joined: Tue Mar 18, 2003 8:06 pm

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Sun Jul 17, 2011 1:19 pm

We've been using Beyond Compare for a couple of years at work, and it works fabulously.

http://www.scootersoftware.com/
Calm seas never made a skilled mariner.
drsauced
Graphmaster Gerbil
 
Posts: 1463
Joined: Mon Apr 21, 2003 1:38 pm
Location: Here!

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Sun Jul 17, 2011 3:43 pm

drsauced wrote:We've been using Beyond Compare for a couple of years at work, and it works fabulously.

http://www.scootersoftware.com/

Do you know if Beyond Compare can compare not only the file itself, but also its metadata?
bdwilcox
Graphmaster Gerbil
 
Posts: 1261
Joined: Mon Apr 21, 2003 12:21 pm

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Mon Jul 18, 2011 1:06 am

Ah, no it doesn't seem to be able to compare metadata for pictures. That'd be pretty useful...

I'll send them an email tomorrow and see what they say.
Calm seas never made a skilled mariner.
drsauced
Graphmaster Gerbil
 
Posts: 1463
Joined: Mon Apr 21, 2003 1:38 pm
Location: Here!

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Tue Jul 19, 2011 2:16 am

Beyond Compare looks like the Rolls Royce of duplicate file finders. Personally I am too cheap to pay for software like that, as I probably will only use it once or twice.

Here are some free alternatives, but you get what you paid for.

Auslogics Duplicate File Finder

CloneSpy

Duplicate Cleaner

Duplicate Files Finder

AllDUP
FDISK /MBR
Dirge
Gerbil Jedi
 
Posts: 1543
Joined: Thu Feb 19, 2004 3:08 am
Location: New Zealand

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Mon Aug 08, 2011 6:48 am

Tested clonespy, awful.
Working on dupe files finder and dupe cleaner now
AbRASiON
Gerbil Team Leader
 
Posts: 274
Joined: Tue Mar 18, 2003 8:06 pm

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Wed Jan 11, 2012 3:49 am

AbRASiON wrote:Does anyone have a recommended application to scan multiple drives and find duplicate files?
Bonus points for something which presents the data in a way which shows entire folders which are the same.

I've used DFL in the past (DOS) and duplic8 for about 10 years but I need something more powerful.


Sorry to hijack the thread but i was wondering how would you get so many duplicate files that would determine you to use a program to clean them up?
nVidia video drivers FAIL, click for more info
Disclaimer: All answers and suggestions are provided by an enthusiastic amateur and are therefore without warranty either explicit or implicit. Basically you use my suggestions at your own risk.
Arclight
Gerbil Elite
 
Posts: 696
Joined: Tue Feb 01, 2011 3:50 am

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Wed Feb 22, 2012 12:38 pm

Here is something I used in the past: http://www.araxis.com/merge/index.html They offer a free 30 day trial.
bitcat70
Gerbil
 
Posts: 40
Joined: Fri Jan 28, 2011 3:02 pm

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Wed Feb 22, 2012 1:37 pm

What system are you running and what file attributes are you looking to compare? This could be a pretty simple script depending on what you're looking for
LaChupacabra
Gerbil First Class
Gold subscriber
 
 
Posts: 133
Joined: Tue Dec 30, 2008 10:59 pm

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Thu Feb 23, 2012 2:35 pm

Just wanted to drop this in here. I whipped this up in Powershell (guessing you are using Windows). I created 2 directories with random files in them, then created similar folders in the directories (some the same and some different) and then moved files into those directories. Here's what I came up with

Code: Select all
$var1 = @(Get-ChildItem c:\test1 -recurse)
$var2 = @(Get-ChildItem c:\test2 -recurse)
Compare-Object $var1 $var2 -includeequal -excludedifferent -passthru


Pretty straightforward really. What this does is builds an array from whatever directories you chose (in this case 2 folders called test1 and test2 in C:), goes the through those folders and any subfolders it finds, then compares the second directory against the first. This will find any files that are the same, regardless of folder, and put them into a list formated by directory. If you want to output it to a file, simply add this to the end

Code: Select all
$var1 = @(Get-ChildItem c:\test1 -recurse)
$var2 = @(Get-ChildItem c:\test2 -recurse)
Compare-Object $var1 $var2 -includeequal -excludedifferent -passthru > c:\results.txt


and you will get a file in C: named results.txt that will have the results saved. It's quick and free. Best of all you can modify it however you'd like and achieve different results (try removing the -includeequal -excludedifferent and -passthru). Hope it helps!
LaChupacabra
Gerbil First Class
Gold subscriber
 
 
Posts: 133
Joined: Tue Dec 30, 2008 10:59 pm

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Sat Dec 08, 2012 6:55 am

What I would do is run a hashing algorithm - MD5 or SHA1, either is good - across everything. I'd then run a compare on all files with matching hashes. No matching hash means no duplicates - guaranteed. Matching hash could just be a collision (albeit pretty unlikely), hence the full compare.

Won't pick up on pictures that are identical but for metadata (or that are just resized versions), but it'll catch just about everything else.
sjl
Gerbil
 
Posts: 65
Joined: Tue Dec 07, 2004 5:14 pm

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Sat Dec 08, 2012 8:07 am

I like the Duplicate File Finder from Rashid Hoda, http://rashid-hoda.software.informer.com/ That one has always worked well for me and you can do comparisons on combinations of name, binary contents, size plus the ability to choose directories and exclude directories.
nanoflower
Gerbil First Class
 
Posts: 185
Joined: Wed Mar 04, 2009 1:10 pm

Re: Recommended duplicate file finder? (binary file compare)

Postposted on Fri Mar 22, 2013 3:53 am

If you haven't seen this already, here is a good list of free duplicate file finders: http://www.techsupportalert.com/best-fr ... tector.htm
kroker
Gerbil In Training
 
Posts: 8
Joined: Sun Feb 21, 2010 3:26 am


Return to Storage

Who is online

Users browsing this forum: Exabot [Bot] and 5 guests