Jumbo Frames?

[gbcrush]

I think I understand the basic concept. Watered down: Transferring larger frames = faster throughput with large file transfers, and smaller overhead. Almost everything I've read about it says something like "to make this work, all devices across the network must support jumbo frames", but no one is giving me illustrative examples of "works/doesn't work"

For instance

#1 - My fiancee uses a laptop and connects wirelessly. Since that's not ethernet protocol, I don't have to worry right, I'm assuming that because she connects wirelessly to the router, and from there to the NAS or other PCs, if the router has jumbo frames turned on, we're ok.

#2 - Everything else in the house is wired on GigE, except the Xbox 360. That means I can turn Jumbo frames on for everything, except the 360. If I do that, does stuff break? I wasn't planning on streaming or transferring anything to it, but i suppose it is a possibility with the new NAS going in.

And along those lines, if I enable jumbo frames on everything I can, will the 360 not supporting this interfere with NAS-> HTPC communications. I can't imagine why, but I'm just guessing. Some of you gerbs have experience with this

...

And in case you're wondering, I'm setting up a NAS in my house this weekend. With a centralized place for backups and file transfers, I thought it might be time to visit the Jumbo Frames question.

[notfred]

Short answer, it's not worth it. It's not going to increase your throughput and likely break a bunch of stuff for you.

Longer answer.
The idea of jumbo frames is that there is a fixed amount of overhead processing that happens on each frame regardless of length, bigger frames means more data for the same overhead or less overhead for the same data. However decent modern NICs (e.g. Intel) are offloading a lot of the per-frame processing (e.g. through the use of a TCP Offload Engine), modern CPUs are so fast and modern storage is so comparatively slow that even if you are maxing your storage then your CPU and NIC can usually keep up easily.

The place where you want jumbo frames is on pure locally connected stuff with no routers, e.g. servers connected together or server connected to storage array. This way you have a very limited amount of stuff that talks to each other and it is easy to verify that it all supports jumbo frames. It's also likely to be running extremely heavy network traffic loads.

In the case of the two scenarios mentioned:
#1 Any jumbo frames on the Ethernet segment will hit the bridging block that connects the Ethernet and the Wifi together. At this point I'm guessing that the bridging block will go "Too big, drop it" as it would take knowledge of layer 3 stuff (IP) to fragment it properly and it's a layer 2 bridge (802.3 to 802.11). Hopefully whatever protocol you are running realises after a timeout that the jumbo frame got dropped and tries again with a normal frame, but that's delay, retransmission and hope that get the data through.

#2 TCP should negotiate MSS, but for other protocols like UDP you are relying on applications doing the right thing (see above), and some of them will not. Also remember going out to the Internet as well. You are now potentially relying on the tiny underpowered CPU in your router to perform fragmentation on any jumbo frames (it should do the fragmentation as it is working at layer 3). You're also relying on it clamping the MSS correctly to avoid getting in a world of hurt with people that block ICMP on their servers and break PMTUD.

[gbcrush]

Short answer,...
Longer answer.

See, why cant they say that sort of crap in the descriptions I read in manuals and such. Thanks notfred. That makes sense from an angle I wasnt considering.

[Aphasia]

See, why cant they say that sort of crap in the descriptions I read in manuals and such. Thanks notfred. That makes sense from an angle I wasnt considering.

Because then they wouldnt be able to sell all the fancy certifications that us networking people have to get for extra bling in our CV's.

[bcronce]

My home router supports jumbo frames on the switch. It's a dumb switch, so I can't turn it off.

[notfred]

Switches supporting jumbo frames are fine, that just means that they will switch packets up to that size. A dumb switch will never generate frames (apart from pause frames which are tiny anyway) so there are no worries there.

[emorgoch]

One thing that notfred didn't mention is that there's a limited bandwidth available on the line. With TOE and fast CPUs, that's generally a bottleneck. So to increase the amount of useful data you push down the line, you send larger packets. In a home network, there is potential for increasing transfer speed when moving large files, for things like web browsing, you won't notice any difference since you'll still be using "regular" sized packets going across the internet.

For more information, I personally like Jeff Atwood's write up about how Jumbo Frames work, and his experience implementing them. I'd also quickly mention that WiFi is still Ethernet, just down with electromagnetic radiation as the signalling mechanism rather than electrons going down a wire. All the principals of packets remain the same.

[Aphasia]

But the bandwidth limit of any line is still the same, no matter how you divide your data between headers and payloads inside the packets, so jumbo-frames have exactly zero impact on available bandwidth and its limit. A kink in the cable would do more for bandwidth in most cases.

The thing is that if he gets such a large difference, he has alot of other problems, may be a very old nick that doesnt off-load the checksum properly, etc. Any decent PC today, if you have hard-drive bandwidth available, shouldnt have any problem at all pushing 98%+ Utilization on a gigabit line. That is a bit over 100MB/s.

The thing is, copying files are not a good test of bandwidth, its a test of end to end application use. And that includes hard drive speed, interfaces, filesystem, transfer protocol, network and sharing protocol, etc.

If you look at that image, the first parts are a filecopy first from my fileserver, writing to and than reading from my secondary drive, a velociraptor. The second parts, are the same, but first writing to, than reading from my main drive, and SSD, and if you know your SSD's, you could probably guess the model. The result is... that I need to defragment my velociraptor, since it usually does 100MB/s + on sequential reads if it doesnt have to seek. My SSD does much better. The fileserver is a 4 drive raid 5 array capable of 250MB/s or thereabouts.

A better test is something like iperf that can generate TCP and UDP streams in memory and connect to an endpoint. It's open source btw. And if you need a graphical front-end I can recommend jperf which makes the result abit easier to present to people.

Even better is programs like chariot that we used at work that can tailor specific stream and patterns including errors to see that a network handles everything correctly. And even better than that are the nifty little dedicated boxes with 10Gbit fiber interfaces that you connect to the network that does the same thing, but just a heck of a lot more of it. That said, I watched a few people miss a tiny detail one time during such a setup and the result was, never try to enable port-mirroring on a switchport passing 10 gbits of traffic with 64 byte packets, that will make your tests go awry. On the other hand, it does does work decently with 1500byte packets.

[notfred]

One thing that notfred didn't mention is that there's a limited bandwidth available on the line. With TOE and fast CPUs, that's generally a bottleneck.

Nope, your storage subsystem is generally the limit once you are on GigE and as Aphasia said the bandwidth is constant.

The overhead bandwidth does decrease and hence the bandwidth available to the application level does increase with jumbo frames, but the % gain is too little to risk the breakage.

I'd also quickly mention that WiFi is still Ethernet, just down with electromagnetic radiation as the signalling mechanism rather than electrons going down a wire.

Not quite. 802.11 is based on the 802.3 principles but there are differences, especially in the area of jumbo frames.
Wikipedia mentions:The IEEE 802 standards committee does not recognize jumbo frames, as doing so would remove interoperability with existing Ethernet equipment and other 802 protocols, including 802.5 Token Ring and 802.11 Wireless LAN.

Even better is programs like chariot that we used at work that can tailor specific stream and patterns including errors to see that a network handles everything correctly. And even better than that are the nifty little dedicated boxes with 10Gbit fiber interfaces that you connect to the network that does the same thing, but just a heck of a lot more of it. That said, I watched a few people miss a tiny detail one time during such a setup and the result was, never try to enable port-mirroring on a switchport passing 10 gbits of traffic with 64 byte packets, that will make your tests go awry. On the other hand, it does does work decently with 1500byte packets.

Yup, used to use Ixia boxes heavily whilst I was the software developer responsible for all Cisco CRS Ethernet interfaces - 10GigE at 64bytes is over 14million packets per second. You need hardware handling to cope with that kind of throughput, to say nothing of the 100GE interface that I helped fix the demo for.

[Aphasia]

Ouch, 100GE is a whole other question, havent even gotten closed to something like that. Aggregated total of 70G to a single site, sure, yes, but single int 100G, not even close for me. Although I've heard a few stories from a friend that went down to Siemens in germany and viewed them trying to toast one of the better ciscos with traffic load, they had a row of racks with datagenerating servers to get enough troughput, never did get to hear if they managed to "break" it though.

Edit: defragged my velociraptor, and its now up to its speedy 110MB/s write speed again, although even lower than before on read, which is odd, but that may be the placement on disk.

[just brew it!]

But the bandwidth limit of any line is still the same, no matter how you divide your data between headers and payloads inside the packets, so jumbo-frames have exactly zero impact on available bandwidth and its limit. A kink in the cable would do more for bandwidth in most cases.

Umm...

The point is that jumbo frames reduce overhead by allowing a larger percentage of each frame to be devoted to payload. So even though the number of raw bits flowing down the wire doesn't change, more of those bits are user data; therefore the *effective* bandwidth (as seen by the application) goes up.

[Aphasia]

TLDR: In theory all is well, in practice, its most often isnt worth the bother. If you want to try it, enable it on both side and see if it makes a difference, but don't expect any miracles. It's easy enough to try out. Just enable it on the workstations, fileserver and have switching support, it usually works alright even with the other equipment if they follow standard. If something like a printer doesnt work, turn it off again.

---------------------

JBI - Point taken, and I never said it wasnt. just that the gain is so small it in most cases doesnt matter. So even if the theory is sound, the practice isnt neccessarily sound, and thats where it depends on your enviroment. The difference in transferred data should still be ~1.5% at the most or so due to the extra overhead bytes using TCP. Should be something like 140bytes per jumbo frame times. Without doing the math with pkts/s, it should add up to perhaps 1.5MB/s or so over a gigabit line if you have full utilization, and that is without any filesystem overhead. As I said in my last post, some of the problems is keeping a connection fed with data.

In practice, the gain in transfer rate is offset by the problems it can cause. Many recomendations about jumbo generally says that you want it supported on the full subnet you are on... which in theory often isn't doable, since many subnets often includes a printer, etc. And in home networks, once you start mixing in what falls under etc its routers/printers/nas/media servers/xbox'es/tv-sets/DVR-boxes you get very spotty support for jumbo. Then you have notfred's point #2 higher up of course.

If you are talking specialty cases like some storage implementations, there may be different uses for jumbo, but even more so super jumbo that goes up to 64KB of payload. Then you of course have the other hassles that depend if your networking equipment is store and forward or cut-through, which, over the long run in larger networks may have an impact since a larger frame takes more time on each store/forward. Of couse, cut-through has the problems with error-checking instead.