Personal computing discussed

Moderators: morphine, Steel

 
BIF
Gold subscriber
Minister of Gerbil Affairs
Topic Author
Posts: 2408
Joined: Tue May 25, 2004 7:41 pm

Setting up a Syncronized Folder ... Thingy...

Fri Apr 19, 2019 4:28 pm

Hello!

I have a Synology DS918+ NAS with (currently) a 10 TB drive in it. No RAID yet, of course. I'm setting up a couple of "DropBox-like" syncronization/replication folders, and would like to ask some questions about the way I should go about populating everything.

One of my first projects is to do this with my "downloads" partitions. My objective is to consolidate both PCs' downloads partitions onto one of the PCs (the workstation first) so that I'll have a "master" partition from which to start. Next, I'll set up replication between that PC and the NAS, and will wait for everything to get replicated up to the NAS. Once everything is uploaded to the NAS, I plan to partner the other PC (the laptop) with the NAS and let everything replicate from the NAS down to the laptop.

When all is done, I'll take one last backup of the "old" download folder on the laptop, then delete it and keep the new syncronized folder.

Questions:

Q1 - In my grand plan above, have I forgotten something critical?

Q2a- Going forward, for backups...do I need to back up the downloads folder on all three devices (laptop, workstation, and NAS)?

Q2b - And if so, how do I ensure that my backups are consistent between the three devices?

Q2B - Or can I get away with backing up the folder only on one of those devices, such as the workstation or the NAS?
 
Redocbew
Gold subscriber
Gerbil Jedi
Posts: 1918
Joined: Sat Mar 15, 2014 11:44 am

Re: Setting up a Syncronized Folder ... Thingy...

Fri Apr 19, 2019 4:45 pm

It sounds like what you really want is a shared network drive that's mapped to a directory on the NAS. If it's just for storage and performance isn't critical that would be a lot simpler than syncing a local directory across multiple systems.

Otherwise, you need to decide if you want the syncing to happen in real time like the Dropbox client does it, or if just syncing once a night at 3AM or something is good enough. If you want it to happen in real time then you need something which gets installed on both the NAS and each of your sync'd systems that monitors the disk, and manages sending files to and from the NAS when they're updated. I believe Synology offers applications which can do that; they call it "drive" or "cloud station" or something. I just do backups nightly with rsync so I haven't really looked into that.
Do not meddle in the affairs of archers, for they are subtle and you won't hear them coming.
 
Dposcorp
Silver subscriber
Minister of Gerbil Affairs
Posts: 2737
Joined: Thu Dec 27, 2001 7:00 pm
Location: Detroit, Michigan

Re: Setting up a Syncronized Folder ... Thingy...

Fri Apr 19, 2019 5:00 pm

Do you have a off site location for any of the storage (NAS, PC, laptop?)
You need that for true disaster prep.
 
BIF
Gold subscriber
Minister of Gerbil Affairs
Topic Author
Posts: 2408
Joined: Tue May 25, 2004 7:41 pm

Re: Setting up a Syncronized Folder ... Thingy...

Fri Apr 19, 2019 5:20 pm

Thanks, this is a good start.

Redocbew wrote:
It sounds like what you really want is a shared network drive that's mapped to a directory on the NAS. If it's just for storage and performance isn't critical that would be a lot simpler than syncing a local directory across multiple systems.
For this one, it is just storage, but I won't always be within my WIFI range (so I'm thinking local is probably best), and really I've been keeping 1TB of downloads files on both machines all this time anyway, so it's not really an increase in my storage expectations.

Otherwise, you need to decide if you want the syncing to happen in real time like the Dropbox client does it, or if just syncing once a night at 3AM or something is good enough. If you want it to happen in real time then you need something which gets installed on both the NAS and each of your sync'd systems that monitors the disk, and manages sending files to and from the NAS when they're updated. I believe Synology offers applications which can do that; they call it "drive" or "cloud station" or something. I just do backups nightly with rsync so I haven't really looked into that.
Yes, Cloud Station is the old one..."Drive" is the new one. I've already set up and tested the replication using Drive. I actually like it a lot, plus it keeps logs of all files being changed, added, deleted, and the status of replicating those changes.

I'm thinking locally accessible partitions would be good here (particularly on the laptop) because I won't always be at home. And my use-case is pretty simple: Most of my download files are applicable to both machines anyway.

For example, if I download an ISO file on the workstation in support of a beta team that I'm on, I'm probably going to need to install it on the laptop too. Maybe not immediately, but probably "at some point soon". If I download a new version of Filezilla's installer on one machine, I'm going to eventually want to use it on the other. Probably 99% of what I do will be "the same thing for both computers."

For the laptop, synchronization does not need to occur while I'm on travel or otherwise not connected to my WIFI network. But I do need access to those files when I'm away from home, and I'd like to not be limited by transmission speeds or the possibility that my house has a power or internet outage and my NAS is down. So local + replication is the way, I think.

I understand the issues with deletions...that deletions on one of the devices will get replicated to the others, resulting in the replicant versions of those files also being deleted on all of the devices once replication resumes. I'm okay with that, because I need to reduce repetitiveness (me deleting, adding, changing files multiple times on multiple devices). Plus, I will be taking backups on a regular basis anyhow (I just need a workable strategy).

If this works okay with my download files, then I'll implement similar setups for my photos (I need to figure out how to combine my iCloud photos with those on my PCs), data documents (kind of like my own version of "Google Docs", videos, website source, and basically everything that I might need access to, regardless which device I'm on.
 
BIF
Gold subscriber
Minister of Gerbil Affairs
Topic Author
Posts: 2408
Joined: Tue May 25, 2004 7:41 pm

Re: Setting up a Syncronized Folder ... Thingy...

Fri Apr 19, 2019 5:23 pm

Dposcorp wrote:
Do you have a off site location for any of the storage (NAS, PC, laptop?)
You need that for true disaster prep.


This project is to make my data access more convenient/more available. It's not intended for disaster recovery. I already use Macrium Reflect to take backups, and that is completely separate from this thread. I do need to get better about cycling my backup drives to and from an offsite location, however; but then who doesn't?

Thanks!
 
Redocbew
Gold subscriber
Gerbil Jedi
Posts: 1918
Joined: Sat Mar 15, 2014 11:44 am

Re: Setting up a Syncronized Folder ... Thingy...

Fri Apr 19, 2019 5:27 pm

The nice thing about DSM is that it makes replicating your files from the NAS up to S3 or some other cloud storage really easy. It does tend to get clogged every now and then if you give it a ton of stuff to chew on though, so take that for what it's worth.
Do not meddle in the affairs of archers, for they are subtle and you won't hear them coming.
 
BIF
Gold subscriber
Minister of Gerbil Affairs
Topic Author
Posts: 2408
Joined: Tue May 25, 2004 7:41 pm

Re: Setting up a Syncronized Folder ... Thingy...

Fri Apr 19, 2019 5:36 pm

Redocbew wrote:
The nice thing about DSM is that it makes replicating your files from the NAS up to S3 or some other cloud storage really easy. It does tend to get clogged every now and then if you give a ton of stuff to chew on though, so take that for what it's worth.
Good to know, thanks. Yep, I plan to just work one project at a time. Downloads folder first, then the next thing which will probably be my photos and documents.

Actually, thinking about it, "everything" is a document. Even a photo. Even a music project with its project file and all the separate .wav files and stuff. Even a website's source files. Even a 3D graphic file scene or a CAD file with my newest idea for a fully enclosed tube train. Okay, I don't have that last one, but it's still a great idea.

Anyway, they're all "documents", even if in my mind some are more and others are less. Hmmm, more to consider.
 
Duct Tape Dude
Gerbil Elite
Posts: 719
Joined: Thu May 02, 2013 12:37 pm

Re: Setting up a Syncronized Folder ... Thingy...

Fri Apr 19, 2019 5:54 pm

Have you considered an actual syncing tool like SyncThing (https://syncthing.net/) or Resilio Sync (https://www.resilio.com/individuals-sync/) ?

I use Resilio among several devices to sync a few TB of data across several folders, and it just works. Sufficient conflict resolution for deletions and all, straightforward permissions, simple discovery anywhere, etc. The Pro version lets you do "selective sync" which is like OneDrive/Dropbox's placeholder files. It basically makes your NAS into private cloud storage on the go. Can't recommend it enough (though remember these enable file syncing, not backup).

For actual backups you should have your NAS sync to a cloud service or something offsite, in case something takes out your entire setup.
 
BIF
Gold subscriber
Minister of Gerbil Affairs
Topic Author
Posts: 2408
Joined: Tue May 25, 2004 7:41 pm

Re: Setting up a Syncronized Folder ... Thingy...

Fri Apr 19, 2019 6:06 pm

Duct Tape Dude wrote:
Have you considered an actual syncing tool like SyncThing (https://syncthing.net/) or Resilio Sync (https://www.resilio.com/individuals-sync/) ?

I use Resilio among several devices to sync a few TB of data across several folders, and it just works. Sufficient conflict resolution for deletions and all, straightforward permissions, simple discovery anywhere, etc. The Pro version lets you do "selective sync" which is like OneDrive/Dropbox's placeholder files. It basically makes your NAS into private cloud storage on the go. Can't recommend it enough (though remember these enable file syncing, not backup).

For actual backups you should have your NAS sync to a cloud service or something offsite, in case something takes out your entire setup.
Nice! But for now I'm planning to use Synology's built-in syncing tool. As I mentioned in a prior post, it logs everything. It even identifies the user who did something (edited this file, added that file). Unless it gives me grief, I'll stick with the Synology tool. And yes, this project is for Syncing, not backup.

I'm not sure yet how I'll handle the backups for the NAS itself, but I definitely need to plan for (at a minimum) backing up the NAS's OS and configuration files. Not sure how one backs up "sync'd" files, especially if the PCs they're synced with are already being backed up (I alluded to this in one of my previous posts).

Who is online

Users browsing this forum: No registered users and 1 guest