Page 1 of 1

gigabit ethernet throughput

Posted: Fri Nov 03, 2017 11:50 am
by captaintrav
I was noticing my file transfer speeds seem to be bottlenecked, so I tried doing some testing with nttcp and noticed somethings.

My server has a PCI gigabit Ethernet adapter, and that's going to be a bottleneck possibly, but, I can send faster (around 600mbit) than I can receive (450mbit), what would explain that?

This also affects another machine with a I believe realtek integrated Lan, but it should be PCIe. I can only seem to get ~1gbit in one direction, the other direction goes 800mbit (can't recall which is which)

Predictably, an old laptop with an Intel gigabit NIC can thrash ~1gbit in either direction, as can a machine with an Nvidia integrated LAN.

I guess if I can't find any ideas on either the realtek issue or increasing throughput of a PCI device I can buy some PCIe addin NICs? I want my gigabits! The PCI thing puzzles me because supposedly PCI should be able to do 133MBytes/sec but with my PCI Nic it's bottlenecked to 35MBytes/sec. I can see it being a shared bus and all, but next to nothing should be using PCI bandwidth on this machine.

Re: gigabit ethernet throughput

Posted: Fri Nov 03, 2017 12:32 pm
by SuperSpy
Remember that PCI can only do 133 MB/s total whereas Ethernet is full duplex aka 125 MB/s both up and down.

Re: gigabit ethernet throughput

Posted: Fri Nov 03, 2017 12:49 pm
by Usacomp2k3
What's the CPU usage? Cheap one's often rely on the CPU for calculations which can slow it down too.

Re: gigabit ethernet throughput

Posted: Fri Nov 03, 2017 1:40 pm
by notfred
You already know the answer:
captaintrav wrote:
Predictably, an old laptop with an Intel gigabit NIC can thrash ~1gbit in either direction,

Messing around with anything else is just wasting your time.

Re: gigabit ethernet throughput

Posted: Fri Nov 03, 2017 2:08 pm
by Aphasia
Any decently integrated NIC on the chipset should be able to get full speed. I usually get around 113-120MB/s (900-960mbps) worth, which pegs my NIC's at >90-96% Utilization during one way transfers. Also note that depending on what drives you have, if not an SSD 120MB/s is a fair bit of data that usually aren't reached on multi-file transfers, especially smaller files.

Re: gigabit ethernet throughput

Posted: Fri Nov 03, 2017 2:52 pm
by captaintrav
Aphasia wrote:
Any decently integrated NIC on the chipset should be able to get full speed. I usually get around 113-120MB/s (900-960mbps) worth, which pegs my NIC's at >90-96% Utilization during one way transfers. Also note that depending on what drives you have, if not an SSD 120MB/s is a fair bit of data that usually aren't reached on multi-file transfers, especially smaller files.


I agree, but this one seems to be a bit s#$!:
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03)

That said, at least it works, I had Realtek hardware in the past that wouldn't work reliably in Linux at all. I'm going to order up an Intel PCIe add-in card and test. I think I should be able to push 60-80MB/sec with the drives I have but haven't got there yet. Of course I'm just being picky since I have a large one-time amount of data to transfer, but what's the use being an 'enthusiast' if I can't be fussy? lol

Re: gigabit ethernet throughput

Posted: Fri Nov 03, 2017 2:55 pm
by captaintrav
notfred wrote:
You already know the answer:
captaintrav wrote:
Predictably, an old laptop with an Intel gigabit NIC can thrash ~1gbit in either direction,

Messing around with anything else is just wasting your time.


This is probably the right answer, but the most expensive one. I don't know how much coin Asus, GigaByte, or whoever saves by integrating these Realtek NICs, but I do appreciate my Lenovo business desktops and workstations I work with by day with their Intel networking hardware.

Re: gigabit ethernet throughput

Posted: Fri Nov 03, 2017 3:09 pm
by notfred
I'm seeing Intel GigE PCIe cards for ~$40 Canadian from Newegg, Amazon and Canada Computers. I don't call that expensive for never having to worry about Realtek again - fear the crab!

Re: gigabit ethernet throughput

Posted: Fri Nov 03, 2017 3:10 pm
by captaintrav
notfred wrote:
I'm seeing Intel GigE PCIe cards for ~$40 Canadian from Newegg, Amazon and Canada Computers. I don't call that expensive for never having to worry about Realtek again - fear the crab!

Point taken, the crab is free, and you get what you pay for.

Re: gigabit ethernet throughput

Posted: Fri Nov 03, 2017 9:44 pm
by Waco
Disable all of the CPU offloading in the driver on the Realtek card and you'll see a nice performance bump at the cost of a little CPU time.

Re: gigabit ethernet throughput

Posted: Sat Nov 04, 2017 6:32 am
by just brew it!
Waco wrote:
Disable all of the CPU offloading in the driver on the Realtek card and you'll see a nice performance bump at the cost of a little CPU time.

LOL... so Realtek is the NIC equivalent of the infamous S3 Virge GPU? Hardware-assisted deceleration FTL!

Re: gigabit ethernet throughput

Posted: Sat Nov 04, 2017 4:05 pm
by Waco
In my experience...yes. :lol:

Re: gigabit ethernet throughput

Posted: Sat Nov 04, 2017 4:32 pm
by titan
Besides the NIC and computers, your switch will be a limiting factor as well. Any decent switch will at least have 900 Mbps each way.

Re: gigabit ethernet throughput

Posted: Sat Nov 04, 2017 6:17 pm
by Waco
titan wrote:
Besides the NIC and computers, your switch will be a limiting factor as well. Any decent switch will at least have 900 Mbps each way.

Any decent small switch should be able to run full line-rate on every port in both directions. Even my crappy D-Link can do that!

Re: gigabit ethernet throughput

Posted: Sun Nov 05, 2017 3:33 am
by just brew it!
captaintrav wrote:
Aphasia wrote:
Any decently integrated NIC on the chipset should be able to get full speed. I usually get around 113-120MB/s (900-960mbps) worth, which pegs my NIC's at >90-96% Utilization during one way transfers. Also note that depending on what drives you have, if not an SSD 120MB/s is a fair bit of data that usually aren't reached on multi-file transfers, especially smaller files.

I agree, but this one seems to be a bit s#$!:
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03)

That said, at least it works, I had Realtek hardware in the past that wouldn't work reliably in Linux at all. I'm going to order up an Intel PCIe add-in card and test. I think I should be able to push 60-80MB/sec with the drives I have but haven't got there yet. Of course I'm just being picky since I have a large one-time amount of data to transfer, but what's the use being an 'enthusiast' if I can't be fussy? lol

FWIW I just ran iperf tests between two machines with that exact same Realtek NIC. I get a solid 941 Mb/sec in both directions, with ~10% CPU utilization.

And yes, the Realtek NIC drivers bundled with the Linux kernel used to suck quite badly. Circa 2010 you had to download the source tarball from Realtek's site and compile it yourself if you wanted any chance of having a reasonably performant and stable network connection. IME the stock drivers have been OK since around 2012 though.

Re: gigabit ethernet throughput

Posted: Sun Nov 05, 2017 8:13 pm
by Aphasia
I think all my integrated Nic's are actually Intel based, it doesnt even really show up on any CPU meter.

Re: gigabit ethernet throughput

Posted: Sun Nov 05, 2017 8:40 pm
by just brew it!
Aphasia wrote:
I think all my integrated Nic's are actually Intel based, it doesnt even really show up on any CPU meter.

Note that on Linux the % CPU usage shown by the "top" command is per-core (i.e. full utilization is 100 x number of cores). That 10% is 10% of one core; so on my 8-core FX-8350 it is only 1.25% of available CPU cycles. IOW not enough to matter.

Re: gigabit ethernet throughput

Posted: Sun Nov 05, 2017 8:41 pm
by notfred
Press "1" in top to get the per-cpu usages.

Re: gigabit ethernet throughput

Posted: Sun Nov 05, 2017 8:42 pm
by just brew it!
notfred wrote:
Press "1" in top to get the per-cpu usages.

Ahh, interesting. In the past I've always just used htop when I needed a more detailed view.

Re: gigabit ethernet throughput

Posted: Mon Nov 06, 2017 9:02 am
by Waco
nmon is pretty awesome now that someone ported it to Linux as well. I loved it on AIX and was overjoyed when it was ported to Linux.

Re: gigabit ethernet throughput

Posted: Mon Nov 06, 2017 9:06 am
by captaintrav
Waco wrote:
titan wrote:
Besides the NIC and computers, your switch will be a limiting factor as well. Any decent switch will at least have 900 Mbps each way.

Any decent small switch should be able to run full line-rate on every port in both directions. Even my crappy D-Link can do that!


Yes the switch doesn't seem to be a problem. Both an Intel NIC and the integrated NForce Ethernet (despite this not having the greatest rep either) show solid throughput in either direction.