Recommendations for file sync and backup.

The place for all kinds of software for all kinds of operating systems.

Moderator: Dposcorp

Recommendations for file sync and backup.

Postposted on Sat Jan 08, 2011 1:32 am

I have a laptop, main desktop, and external HDD with 220gb, 1tb, and .5tb, respectively. File organization has been tedious and difficult and I've always backed things up manually but I'm getting fairly sick of having to do so. The problem partially stems from my slight lack of good file organization as well as the clutter that comes in from random downloads everyday, everything from unconverted video downloads to application installers. The items I need to back up are located in quite a variety of folders; anything from My Documents (for games) to folders residing on the desktop, to folders under Program Files need to be backed up, synced, or even combined from time to time. Another challenge is the fact that different items and programs reside on the different systems; for example, I have fewer games installed on the laptop as it is very low spec and not used for gaming, while the desktop has everything and serves as a "command center". The external HDD needs only the "important" files as it is only half the capacity of the desktop's.

Truth is, I'm not even exactly sure what I need. A good start however, would be a program that recognizes a "master system" when setting up initial folders. An example of a folder setup I'd need would be having a folder within the desktop's Program Files folder synced to a similar folder on the laptop (it is a program installation files folder but also contains everything that pertains to that program, like a 3D modeler installation folder that also contains all models and textures), and also synced with a folder allocated for it on the external HDD.

I suppose what I really need is a ton of Dropboxes, but each Dropbox is a actually a typical folder that gets synced, but having them sync over the network instead of the internet when possible is preferable. (Each system can already access each other through Windows networking/drive sharing.) Having the choice to sync between both would be ideal. The program should be able to perform as many operations between merging folders as possible, and automation like Dropbox's is preferable I suppose. My computer usage habits are usually such that only one system is active at a time, and the external HDD is usually off (it is attached to the desktop currently) so the syncronizations should occur when two or more such systems (including the external HDD) happen to be active at a time, or they should occur in realtime like Dropbox's.

I'll continue Googling and see what I can find, but it takes a lot of time to find out what different programs do, so if someone can make recommendations for what I have and need, I'd greatly appreciate it. Thanks.
C-A_99
Gerbil First Class
 
Posts: 151
Joined: Tue Apr 06, 2010 9:46 pm

Re: Recommendations for file sync and backup.

Postposted on Sat Jan 08, 2011 4:18 am

A couple of thoughts from someone who is on a similar quest. In list form!

    You need something which not only mirrors your data but tracks file versions ala Time Machine.

    Verification is super important to me, because there is nothing worse than making a corrupt backup.

    I don't want my backups to be stored in a proprietary format. If I want to access my files later in life I don't want to be stymied by and special restrictions.

    A cross platform solution able to run on both Windows and Linux would be bloody awesome
FDISK /MBR
Dirge
Gerbil Jedi
 
Posts: 1553
Joined: Thu Feb 19, 2004 3:08 am
Location: New Zealand

Re: Recommendations for file sync and backup.

Postposted on Sat Jan 08, 2011 4:12 pm

How would such a time machine work? Tracking multiple versions of several files can really take up a lot of space, but I suppose for files designated as "works in progress", it would be useful.

Accessibility is a given, which I mentioned above. I'm not backing up so much as I am file-syncing, as the external HDD must be accessible on different machines, and obviously a backup file on my laptop would be useless if I can't do anything with it on the laptop.

With my own file syncing needs, cross-platform should be easy as cake to implement. As I said above, when backing up files from, say, my documents, program files, and other system-exlusive folders, I need them to "map" to different folders on the external HDD, and even from my laptop to desktop, since my desktop runs x64 and the program files folder I work with has the "(x86)" after it.

I suppose the best way for this to work is to designate directory mappings between all folders. For example, I have a folder on the external HDD called "mydocs", which "my documents" would map to. It needs to be better than that however, as certain folders are important to back up, say, The Sims 2 game files, whereas Battlefield 2 profiles are not important because they can be recovered. (So in that case I would map the folder for TS2 on the computer to the directory for TS2 under "mydocs", while ignoring the directory for BF2.)
C-A_99
Gerbil First Class
 
Posts: 151
Joined: Tue Apr 06, 2010 9:46 pm

Re: Recommendations for file sync and backup.

Postposted on Sat Jan 08, 2011 4:45 pm

C-A_99 wrote:How would such a time machine work? Tracking multiple versions of several files can really take up a lot of space, but I suppose for files designated as "works in progress", it would be useful.

Well, on *NIX you can tell the rsync tool to create a backup where any files which have not changed since the previous backup are stored as a link to the previously backed up version instead of making a new copy. If you name the top level backup folders according to the date of each backup, it is very easy to get back to the state of the file system on any previous date. Each backup folder appears to contain a complete snapshot of all the files on that date; but files which do not change from backup to backup are stored as links to a single copy.

Yes, if you have large files that change frequently, it eats a lot of space. You may need to occasionally go into the backup folders and delete large files that you don't need any more.

It is unclear to me how well this linking functionality is supported in the various Windows ports of rsync.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37830
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Recommendations for file sync and backup.

Postposted on Mon Jan 17, 2011 1:49 am

Oh, and I just remembered an issue I forgot earlier; file renamings. It may not be easy to do nor likely to exist in any backup/sync programs (that is, the ability to handle renamed files and update/sync accordingly), but it would be immensely helpful. I suppose the best or common solution to this is for the program to delete a file from the secondary folder (or the folder that is to be updated) if its corresponding file in the primary folder (the folder that needs to be backed up or copied over otherwise) disappears. (Such as due to renaming or deletion).

When I have time, I'll look over some rsync ports on Windows and give one of them a try.
C-A_99
Gerbil First Class
 
Posts: 151
Joined: Tue Apr 06, 2010 9:46 pm

Re: Recommendations for file sync and backup.

Postposted on Mon Jan 30, 2012 1:14 am

Alright, extreme necro-posting here, but I've procrastinated on this until I finally decided on exactly what I could use. (Plus I'd rather keep it all to this topic than create a new one.)

I stopped bothering with this until just recently, then I assembled an old XP machine together to use as a server. Ideally, I want something that's pretty much the exact same as Dropbox, but hosted by me instead of them. (The main issue with this is that I'll have to DMZ the server machine and find a way to get it out on the web... And hope my ISP doesn't constantly change my IP.) I've looked into or worked with the following programs:

- Syncrify: A client-server kind of program that uses rsync internally. (Lots of them do I guess, including Dropbox.) The server configuration page simply doesn't allow me to log in after I enter my email and password. Useless until this issue is resolved.
- Windows Live Mesh/Sync: Not sure if I can host locally instead of using the SkyDrive cloud.
- Synctoy: Not a networking based sync, only does folder to folder. Not sure how it handles if drive names change (i.e. if a particular external drive goes from G:\ to H:\). Only allows two sync destinations. Takes more time figuring out which parts of the files changed than it does actually syncing the files. Not a good solution for me.
- SparkleShare: No Windows version yet = not yet of use for me. (Though it is in development.)
- Syncany: In development, but no releases yet that I can find.
- OwnCloud: Seems to lack automation like Dropbox; Server requires Linux but that's not a problem if only the server requires it.
- SugarSync, SpiderOak, etc.: Seems to have no way of hosting it.

The list is non-exhaustive of what I've looked at. In general, the most common problems I'm seeing are clients lacking of support for Windows (Don't care too much if Linux required for server.) or lack of being able to self-host (or otherwise being able to dictate storage capacity by how much actual storage I have, instead of by paying a monthly fee; I need a free solution).

I'll report back if I find something. After using Dropbox for a while, I can say that what I need is just a stretch of it, just need to be able to sync more folders than just the Dropbox, map them accordingly across systems (i.e. C:\Program Files\[program] may need to correspond to G:\Program Files(x86)\[program] on another system) and to be able to do so on my own server. I guess this mapping feature isn't absolutely necessary, but it'd be very useful to have. At this point I shouldn't even care anyway if I can get something fairly basic working.
C-A_99
Gerbil First Class
 
Posts: 151
Joined: Tue Apr 06, 2010 9:46 pm


Return to General Software

Who is online

Users browsing this forum: No registered users and 3 guests