Personal computing discussed

Moderators: renee, Steel, notfred

 
dashbarron
Gerbil XP
Topic Author
Posts: 343
Joined: Tue Aug 11, 2009 8:49 am

Media Replication

Tue Apr 16, 2019 6:57 am

Thought I'd cull the opinions of the masses here to see where I should start.

I have a handful of physical servers across the internet that contain a lot of documents, media, photos, music, videos. How do I setup replication between all of these? As in, new things are added, they all get a copy . What's the software I can use to schedule this out (or use built-in windows function to run the scheduler) in the dead of night?

Is DFS in Windows server something to look at? Or is it considerable overkill? I see there's a product called Allway Sync which I've installed but haven't tried to setup yet. And then I was thinking Veeam has a free community version of backup and replication, though again haven't looked at it yet.

Thanks ahead of time. I usually get these problems solved through a slue of your suggestions.
Last edited by dashbarron on Tue Apr 16, 2019 8:10 am, edited 1 time in total.
 
Chrispy_
Maximum Gerbil
Posts: 4670
Joined: Fri Apr 09, 2004 3:49 pm
Location: Europe, most frequently London.

Re: Media Replication

Tue Apr 16, 2019 7:51 am

I've used DFS replication in the past and it falls afoul of several real-world scenarios unless you commit to "The Microsoft Way" of data management. FWIW, this "Microsoft Way" of data management is a moving target of best-practices and outdated documentation, using customers as paying lab rats. It still feels like it's very much tied to the Sharepoint collaboration era of publishing from multiple users in multiple locations with version-control gaffer-taped onto the side as a mandatory afterthought. In the right situation it's amazing but I've never found it to do the simple job it's supposed to do when just trying to use it on enormous datasets in real-time.

I manage around 100TB of cross-site replication using scheduled Robocopy mirroring, which is both bomb-proof reliable and costs nothing, but it's not bit-level replication, only file-level replication, so it's bandwidth intensive. If you have huge datasets with regular changes in large files over low bandwidth, I would look at byte-level or block-level replication software instead, and to answer those questions we probably need to know more about what platform your storage is on. Rsync will work at byte-level if that's an option for you.

Veem's really good, but also possibly overkill. I haven't used the community edition so I don't know what its limitations are or how well it works. Hopefully someone else can comment on that.
Congratulations, you've noticed that this year's signature is based on outdated internet memes; CLICK HERE NOW to experience this unforgettable phenomenon. This sentence is just filler and as irrelevant as my signature.
 
dashbarron
Gerbil XP
Topic Author
Posts: 343
Joined: Tue Aug 11, 2009 8:49 am

Re: Media Replication

Tue Apr 16, 2019 8:24 am

Chrispy_ wrote:
I've used DFS replication in the past and it falls afoul of several real-world scenarios unless you commit to "The Microsoft Way" of data management. FWIW, this "Microsoft Way" of data management is a moving target of best-practices and outdated documentation, using customers as paying lab rats. It still feels like it's very much tied to the Sharepoint collaboration era of publishing from multiple users in multiple locations with version-control gaffer-taped onto the side as a mandatory afterthought. In the right situation it's amazing but I've never found it to do the simple job it's supposed to do when just trying to use it on enormous datasets in real-time.

I manage around 100TB of cross-site replication using scheduled Robocopy mirroring, which is both bomb-proof reliable and costs nothing, but it's not bit-level replication, only file-level replication, so it's bandwidth intensive. If you have huge datasets with regular changes in large files over low bandwidth, I would look at byte-level or block-level replication software instead, and to answer those questions we probably need to know more about what platform your storage is on. Rsync will work at byte-level if that's an option for you.

Veem's really good, but also possibly overkill. I haven't used the community edition so I don't know what its limitations are or how well it works. Hopefully someone else can comment on that.


Robocopy as in the Powershell command? This could work, as I don't need any bit-level changes, and for what documents change a few extra Kb/s won't matter. Everything is on a Windows machine and new servers will be. Storage a bit mangled right now, but the goal is a RAID card and mirrored drives on a SATA or SAS interface for everything. Lots of data protection between the daily syncs and at least one machine that has daily backups.
 
Chrispy_
Maximum Gerbil
Posts: 4670
Joined: Fri Apr 09, 2004 3:49 pm
Location: Europe, most frequently London.

Re: Media Replication

Tue Apr 16, 2019 6:04 pm

Yep.

https://ss64.com/nt/robocopy.html - you can call it from task scheduler and include switches and logging with ease.

For replication you probably want to use the /MIR switch. As always, test on a sacrificial copy of things first - it's destructively powerful if you screw up the switches and paths.
Congratulations, you've noticed that this year's signature is based on outdated internet memes; CLICK HERE NOW to experience this unforgettable phenomenon. This sentence is just filler and as irrelevant as my signature.
 
Flying Fox
Gerbil God
Posts: 25690
Joined: Mon May 24, 2004 2:19 am
Contact:

Re: Media Replication

Wed Apr 17, 2019 2:49 am

To me, the more interesting problem seems to be how to cross the different networks. Assuming this is not some business-type site-to-site network and more like between 2 home-based computers connected to the public internet, you will need a bit more secure way to have 1 connecting to the other. Opening up port 139 over the wild dangerous internet is pretty much a no-no. Some sort of VPN may usually be required and I'm very interested in an easy to set up but decently secure solution for that. At this stage it is a bit beyond my level of networking-fu. :-?
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
 
dashbarron
Gerbil XP
Topic Author
Posts: 343
Joined: Tue Aug 11, 2009 8:49 am

Re: Media Replication

Thu Apr 18, 2019 1:02 pm

Flying Fox wrote:
To me, the more interesting problem seems to be how to cross the different networks. Assuming this is not some business-type site-to-site network and more like between 2 home-based computers connected to the public internet, you will need a bit more secure way to have 1 connecting to the other. Opening up port 139 over the wild dangerous internet is pretty much a no-no. Some sort of VPN may usually be required and I'm very interested in an easy to set up but decently secure solution for that. At this stage it is a bit beyond my level of networking-fu. :-?


Well you got me there. I didn't really think about this. I have a VPN, but not sure if it's static-enough to setup a semi-permanent solution or not.

I know there's a program called Shrewsoft for doing those connections, but I'm a complete novice when it comes to things like this.
 
Redocbew
Minister of Gerbil Affairs
Posts: 2495
Joined: Sat Mar 15, 2014 11:44 am

Re: Media Replication

Thu Apr 18, 2019 1:59 pm

In Linux I use rsync over SSH in a nightly cron job. That's how I replicate my home directory to the NAS each night. In Windows I'm not sure how you'd authenticate with the remote host using something like robocopy. It might be something just as simple, but I haven't tried it.
Do not meddle in the affairs of archers, for they are subtle and you won't hear them coming.
 
chuckula
Minister of Gerbil Affairs
Posts: 2109
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Media Replication

Thu Apr 18, 2019 2:03 pm

Redocbew wrote:
In Linux I use rsync over SSH in a nightly cron job. That's how I replicate my home directory to the NAS each night. In Windows I'm not sure how you'd authenticate with the remote host using something like robocopy. It might be something just as simple, but I haven't tried it.


Came looking for rsync.
Left happy.

Incidentally, there are some options that let rsync "cheat" by looking at timestamp and file size information to skip files that have the same timestamp and file size. While less rigorous than actually using the rsync algorithm to ensure the contents of a local and remote file are identical, it's ok to use in many real-world situations where changes to files are expected to also change these bits of metadata.
4770K @ 4.7 GHz; 32GB DDR3-2133; Officially RX-560... that's right AMD you shills!; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
 
dashbarron
Gerbil XP
Topic Author
Posts: 343
Joined: Tue Aug 11, 2009 8:49 am

Re: Media Replication

Tue Apr 23, 2019 12:39 pm

chuckula wrote:
Redocbew wrote:
In Linux I use rsync over SSH in a nightly cron job. That's how I replicate my home directory to the NAS each night. In Windows I'm not sure how you'd authenticate with the remote host using something like robocopy. It might be something just as simple, but I haven't tried it.


Came looking for rsync.
Left happy.

Incidentally, there are some options that let rsync "cheat" by looking at timestamp and file size information to skip files that have the same timestamp and file size. While less rigorous than actually using the rsync algorithm to ensure the contents of a local and remote file are identical, it's ok to use in many real-world situations where changes to files are expected to also change these bits of metadata.


Linux based solutions have never been my strong suit. I always try and end up feeling disappointed and overwhelmed. So I try to stick to Windows solutions; extra time to track down Linux problems hasn't been my luxury unfortunately.

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On