fast network/storage hardware

[Turkina]

Oh wise gerbils, I am in a bit of a quandary on a project and not sure how to proceed. I have a server which needs to stream a massive amount of (video) data to 2 different hosts. Our initial approach was to broadcast it via UDP over a standard gigabit ethernet, but this doesn't give us enough bandwidth (the hosts are in close physical proximity and can easily be connected to the same switch/etc). We figure we need 1.2 - 1.5 gigabits/second sustained. We don't have a ton of money to throw at the problem, but could spare maybe $1k or so.

We have identified the following options:
link aggregation - using 2 gigE ports (and a switch with high enough crossbar bandwidth) to try to get the ~1.5 Gbps we need.
10 gigabit ethernet - definitely enough bandwidth, but probably too expensive ($4k for a switch at least?)

Maybe we are barking up the wrong tree though - for instance are there any networked storage devices with multiple eSATA ports we might be able to use? Since we have only 1 host writing, maybe some kind of fast networked storage would be better (like the old days of multiport SCSI drives)?

Any advice?

[sjl]

In my experience, link aggregation tends to not work as you would think it works. A single connection from host A to host B will tend to use the one Ethernet adapter; the second (and subsequent) adapters will only be used for separate connections, rather than providing throughput to a single high-bandwidth stream. This is fine for my purposes (I'm a backup/recovery sysadmin; link aggregation works just fine in that situation, since there are multiple connections from dozens, if not hundreds, of backup clients), but for you, I'd not recommend it without testing to make sure it does what you expect - and not spending money without verifying that that testing matches your scenario(s).

Infiniband might be an option, or possibly IP over fibre channel - hardware costs are going to be the killer for you; $1k for this sort of host-to-host throughput really isn't much money. "Networked storage", these days, basically means fibre channel; I'm pretty sure there aren't any storage vendors that will do what you want on any other system, not at the sort of throughput you're talking about.

You could also check the size of your MTU; bumping it up might help with your throughput, without needing to spend big bucks on hardware.

[just brew it!]

Do you have your choice of OS, or do one (or both) ends of the connection need to be Windows-based? Linux includes network interface bonding capability; I believe this can be set up to aggregate multiple gigabit NICs into a single logical pipe. But both ends would need to be running Linux, and you'd need to make sure you've got enough bandwidth from the CPU to/from the NICs themselves (all of them need to be PCIe -- no classic PCI).

What are you using for storage? Since you say it is a "massive" amount of video I assume it's spinning hard drives (not SSDs). The hard drives themselves could also be a bottleneck (especially as they get towards the inner tracks) unless you're running a RAID array.

Edit: Doing this the "right" way still requires support from the network switch (port grouping). There's an "adaptive load balance" mode that doesn't require special support from the switch, but this seems to be less of a sure bet.

Edit 2: ...and the adaptive load balance mode suffers from the issue sjl mentions. So at the very least, you'll need a switch that supports port grouping or link aggregation. Or multiple direct point-to-point links (no switch) between the machines in question.

[highlandr]

Would it be possible to do 10Gb cards in each machine and hardwire them together? This seems like the cheapest and easiest solution, without bothering with bonding or expensive switches.

JBI's right to ask about hard drives too. I will say that just last week we were able to saturate a Gigabit link with a 4 drive RAID5 array, pushing out an 11 gig image to 24 clients. Those were only 7200 rpm drives, too, so there is some room if you go with the traditional 10K or 15K drives. (Or SSD, but that's not an option if you need the space.)

[Flatland_Spider]

What software are you using, and what are you streaming? You might be able to solve this problem with software.

[just brew it!]

Would it be possible to do 10Gb cards in each machine and hardwire them together? This seems like the cheapest and easiest solution, without bothering with bonding or expensive switches.

Actually, I think that ends up costing significantly more than a pile of PCIe Gb NICs and a low-end LACP-capable switch. 4X 10Gb NICs will easily blow his 1K budget even without a 10Gb switch.

[Usacomp2k3]

You can get a PCI-E 4gb Fibre NIC for as low as $325/ea. How flexible is the budget?

[notfred]

You've got to watch out on link aggregation for how the traffic is split. Some hash it so as sjl says it will split on a per-stream, whilst others round-robin which will spread the load but potentially result in out of order frames requiring more endpoint processing to reorder.

[Turkina]

Thanks for the replies guys

There might be *some* wiggle room in the budget (maybe we could go to $4k for instance if we were sure it would buy us the performance we need).

As for OS - the team writing the streamer server is unfortunately using Windows at the moment. We are trying to get them to switch over to linux (RHEL or Fedora) so we can do link bonding more easily.

As for disks - there is a pile of 500 GB SSDs we have access to, and our system runs will only be an hour or so tops - typically less than 30 minutes. This is what got me thinking about some kind of multiport NAS solution, since we have plenty of fast storage already.

Would it be possible to do 10Gb cards in each machine and hardwire them together? This seems like the cheapest and easiest solution, without bothering with bonding or expensive switches.

That is an interesting idea - I have seen 10 Gb cards for under $500 each, so for $2k total that might not be a tough sell. I assume we'd wire them together with crossover cables right?

You can get a PCI-E 4gb Fibre NIC for as low as $325/ea.

I didn't realize the NICs were so cheap - what about a switch?

Edit : Maybe 10 GigE isn't out of our price range:
1x http://www.newegg.com/Product/Product.a ... 6833150165
3x http://www.newegg.com/Product/Product.a ... 6833106044
$2800 isn't terrible for that kind of throughput....

[Aphasia]

Remember to add trancievers for the switches and network cards, either fiber trancievers + cables, or active sfp-copper-cable. Cheapest is the cables which is what... somewhat between $100-200/each.

[Anarchist]

if the reason you need such high-bandwidth is in order to stream 1080P HD video perhaps you can save your $1K and down grade the video to 720P or lower.

[Mastax]

Just a small interjection from a non-expert:

Jumbo Frames

They won't solve all (read: any) of your problems, but for large transfers over short distances, they can be surprisingly effective.

[notfred]

If the video is being streamed properly it will be something like MPEG-2 Transport Streams which results in lots of little packets. Enabling jumbo frames wins you nothing in the case where you are sending at less than the MTU.

[Turkina]

It is video data, but in the form of raw frames (maybe some de-Bayering or colorspace conversion or whatever else the imagery geeks want to do with it). This is for an experiment test rig, so the high speed camera data is collected, and shipped off to our processing hosts which will analyze it as fast as possible (so downsampling really isn't much of an option). I think we settled on 10 GigE with something like the hardware I posted links to before. I think we can make the dollars work.