TR Forums

I recently upgraded a client who had a windows 2003 small business server, 4 Gb of ram, 1 1000mb nic, running Amicus (a sql based database) and file sharing. I upgraded them to a Windows 2012 r2 server with a 6 core xeon, 32 gb of ram, 4 nic’s teamed and the lasted versions of the software. Now I’m getting complaints that it seems slower. Sometimes when they just go to open a word processing file, the Windows 7 workstation “hangs”. The Amicus software take 59 seconds to load. There are 16 workstations. I moved one of the workstations right next to the server so it was plugged in where the team was plugged in, without worrying about distance or a secondary switch. Made sure all cable was Cat 5e. I got the same speed results. Any easy way to narrow down where my problem lies? I did start the Amicus on the server and there it takes 25 seconds to load.

How long did it take to load before, with the old server?

Have you verified that the workstations are connecting at 1 Gb/sec (not 100 Mb/sec)?

Sounds like you upgraded the version of Amicus too? Maybe the new version is just slower.

Maybe something is messed up with the teaming. As an experiment, try turning that off and using just a single 1 Gbps link to the server.

Things to check under adapter settings, most of which depend on the particular adapter and driver:
Speed/Duplex?
Jumbo frames?
Send and Receive buffers?
Interrupt coalescing?
Hardware off-load for checksum, (de)fragmentation, etc.

The default Windows drivers do not always expose many of these settings. But if it's an Intel NIC, their driver had these and several other tweaks that may help.

I"m not sure the old server was much faster, but with the new server they were expecting it to be faster. The majority of the system are running 1000mb. I have a couple that are running 100. I have a similar server with a bigger network, I will do a speed of loading test on it for comparison.

The network cards are intel, so I'll check those settings. I've never done Jumbo frames, what is that actually good for?

Jumbo frames won't help if the switches don't support it. That would pretty rare this day and age, however. Jumbo frames allows for larger packets so there's less overhead in the protocol stack, which means more faster speed captain!

Sometimes you can have spotty performance if there are duplex mismatches, but gigabit Ethernet pretty much does away with that. You'd have to check your switches for that.

drsauced wrote:

Jumbo frames won't help if the switches don't support it. That would pretty rare this day and age, however. Jumbo frames allows for larger packets so there's less overhead in the protocol stack, which means more faster speed captain!

On higher end stuff sure jumbo frames are the norm, but other stuff is questionable.

@bigjohn888jb: How did you team the NICs? If the teaming requires LACP and the switch doesn't support it then that could be the problem.

Flatland_Spider wrote:

@bigjohn888jb: How did you team the NICs? If the teaming requires LACP and the switch doesn't support it then that could be the problem.

Yeah, I should've mentioned that in my previous post. For teaming to work, you generally need a compatible network switch. Furthermore, most (if not all) consumer switches don't have LACP support.

Weirdest error's I've seen are often down to either weird duplex error's or weirdly configured or mismatchning link-aggregated ports. Look at the aggregation mode's set on both the server and switch so that isn't a problem.

A close second to that can be links with kinked cables or barely working fibers, etc. So checking error rates on the switch ports is always a good thing. Another thing can be to look at how the application works, does it require reverse records available for the clients, etc. I've seen a few applications that needs it, otherwise they would have a 30 second timeout compared to a client with no reverse or faulty reverse record.

Re:Jumbo Frames
The norm is also that since you have offloading on the NIC's today, the overhead doesn't mather that much, and with a mixed enviroment with different NIC makers, you can run into compatibility issues with Jumboframes. So unless you are running 10G or have specialized loads like certain storage frames over ethernet, Jumbo frames usually doesn't matter that much for any form of desktop use. At time's it the difference between 98% and 98.5% or 99% utilization on a gig link, which in a non heterogenous enviroments usually isn't worth the trouble.

Same with link-aggregation really, most aggregation from default usually balances over links based on a hash of either mac or ip so it never helps with higher troughput on single clients, but it does gives you more concurrent bandwidth with multiple clients as long as your drive subsystem can handle it.

So the switch is a NETGEAR - JGS524 - 24-PORT GIGABIT ETHERNET SWITCH, which I'll check but probably does not support lacp. So unless I replace the switch, the teaming is probably slowing things down?

If I replace this with a switch that support teaming, will it be ok to have switches down stream (I've got a couple of runs that goes to an office with an 8 port netgear with 3 users attached) or do all the switches hanging off this main switch have to support the lacp.

If I must replace this, any recommendations on a good replacement that is not going to break the bank?

That switch does not support LACP. At best, enabling link aggregation on the server is likely not helping. At worst, it may be responsible for some of your performance issues.

The switch the server connects to needs to support LACP. Other switches downstream of it do not. But if you want to get any benefit from the link aggregation, you will need to evenly (as much as possible) spread the expected load across multiple ports on the LACP-capable switch.

I'll leave specific switch recommendations to those who have more experience with LACP switches.

For now, disable the link aggregation (use just one port on the server) and see if that helps.

As I understand it, an LACP-aware switch is not actually necessary to use NIC teaming in Server 2012. There is a "switch independent" mode of operation; I presume this just works as load balancing between two logically separate interfaces (but presumed to be on the same network). For those interested, MS has a fairly readable guide written on it here: http://www.microsoft.com/en-us/download ... x?id=30160

Flatland_Spider wrote:

but other stuff is questionable.

Usually, but not always in my experience. I've had situations where enabling checksum offloading and/or defragmentation to the NIC destroyed performance. Disabling it defintely loaded up the CPU, but performance increased dramatically overall with fewer unexplained glitches/stalls. (In that case, I trashed the NIC.)
And unless the server is low on memory, maxing out the send/receive buffers on Windows has always helped.

TwistedKestrel - that was a good guide o the MS part. What it boils down to is that suddenly there is several modes and ways to team NIC's where some of them are driver independent. And from what I can see, the secondary links only handle outbound traffic, since the primary NIC is the one associated with the MAC with regards to ARP and incomming traffic. Although that sounds fine in theory, in practice, I would'nt implement such a solution unless I was sure the switches and the clients could handle it properly. Although that depends on how the packets are construction of course.

As for the OP, if the teaming used was driver depenend, it's probably LACP and thus require switch support, so as already said, I would start with disabling it and see the effects of that. Or at the least, check that the teaming is correct vs. the switch with regards to the guide from microsoft with a non-LACP/LAG capable switch. With all settings in order it probably works fine though.

It's like setups where you have servers with 10-20 different secondary IP's on the same network for things like webhosting, unless you disable the ability for outbound traffic iniated by the host on all secondary ip's, you might get traffic iniated with the wrong source since you then have both provider order and the value of the addresses used where windows might choose the lowest ip for self iniated traffic where you expect it to be the primary one.

Thanks for all the input. As a first step, I disabled the team and disabled all the network interfaces but one. I will collect information from users this week to see if they notice any difference. I will then look at each nic settings and see if making some changes there will help. I have 6 cores and 32 gb of memory, so if that can help the nic, I don't think it will hurt anything else.

Unless you are maxing out the gigE, I'm betting you don't need the additional headroom.

Your issue is definitely that the switch doesn't support LACP. You can try enabling Static Link Aggregation or just Fault Tolerant links at the NIC level. The latter won't give you any performance benefit, though. And, as has been mentioned, you still have the option of using the built-in NIC teaming support in Server 2012 R2.

However, one thing I would do is enable Hyper-V and install a second OS on the server for file sharing. You can assign that VM it's own NIC so your workloads will be spread out. On Server 2012 R2 Standard you can run up to two VMs without additional licenses. Server 2012 R2 will do a check during the install to see if the host OS is Activated and if so won't prompt for activating the new VM.

If they are Intel NICs, be sure to enable SR-IOV when you setup the virtual switch(es), and then enable that for the VM. It'll give the VM direct access to the NIC instead of having to go through the hypervisor. For just file services you can get away with assigning just 2 vCPUs and 4GB of RAM. If you think you need more RAM (which you shouldn't for file services) you can assign a dynamic amount with the minimum being 4GB and determining an upper limit of something like 8GB, then let Windows/Hyper-V assign as required. I've got DHCP servers that drop down to just a couple hundred MB with dynamic memory assignment.

Another option may be adding an additional IP on one of the NICs and configuring the database to talk on that IP address.

There are a multitude of ways you can configure the link(s). Honestly though, if speed is that paramount you need to look at getting a good, managed, non-blocking switch for the server to connect to.

As for maxing out a GigE link, a GigE Windows 7 or Windows 8 client going to a GigE Server 2008 R2/2012/2012 R2 server can most definitely max out a single GigE link if transferring large files.

curtisb wrote:

As for maxing out a GigE link, a GigE Windows 7 or Windows 8 client going to a GigE Server 2008 R2/2012/2012 R2 server can most definitely max out a single GigE link if transferring large files.

Sure you can burst to that, but it is not likely that a small business is going to sustain it, unless they are doing something extremely heavy like video processing or rendering. Even for 16 workstations. A 5MB file takes .04 seconds to transfer. Even the largest file I work on, 44 MB, is 0.3 seconds worth of speed.

Usacomp2k3 wrote:

Unless you are maxing out the gigE, I'm betting you don't need the additional headroom.

Seconded. A single GigE link should suffice.

If you do want LACP, we've been using D-Link Web Smart switches with great success for the last few years. They support all the niceties that you would expect from a much more expensive HP or Cisco model, and I don't recall ever having one fail. Also quite easy to manage. Did I mention cheap?

We sell loads of these:
http://www.neweggbusiness.com/Product/P ... 06160741:s

homerdog wrote:

Seconded. A single GigE link should suffice.

If you do want LACP, we've been using D-Link Web Smart switches with great success for the last few years. They support all the niceties that you would expect from a much more expensive HP or Cisco model, and I don't recall ever having one fail. Also quite easy to manage. Did I mention cheap?

We sell loads of these:
http://www.neweggbusiness.com/Product/P ... 06160741:s

Wow! I agree that is an incredibly full-featured switch for the price. I did not realize that this level of capability had gotten this cheap.

Huh. I think I got one of those for my home network because it was a good deal for a gigabit switch at the time. Fancy stuff!

First question: how do you know this is a network problem, and not something else like a DNS resolution issue, an active directory problem, or something else?

The new breed of smartswitches, which is what they started as seem to have come down in price. I have an older GS108T from netgear which does the job, but saw that they are really low in price nowdays.

Buub wrote:

First question: how do you know this is a network problem, and not something else like a DNS resolution issue, an active directory problem, or something else?

Doesn't sound like a DNS or AD problem to me. Also he's trying to use LACP on a switch that doesn't support it. Loopdy Loop!

The easiest way to find out would be to drop down to a single link and test...

If you do suspect wanky DNS you could set the server as primary DNS and try (if it isn't already).

BTW those D-Link switches have great loopback detection (lol I would make a good D-Link sales rep )

homerdog wrote:

If you do suspect wanky DNS you could set the server as primary DNS and try (if it isn't already).

And just to clarify...in an AD environment your Domain Controllers should be the ONLY DNS servers configured for your workstations when they are inside the domain (I list the caveat for laptops that are often used outside of the domain). The DCs should be configured as DNS forwarders and do all lookups.

Not only can having non-DC DNS servers configured on clients cause issues, but it allows you to block DNS queries going out of your network from anything but your DCs...thus preventing users from configuring external DNS to bypass any other filtering you may be doing.

Only time I consider not using the DC for primary DNS is if the DC is in a remote location. Not just laptops, but maybe a remote site connected by an already overtaxed point to point T1 (ugh).

Seems now to be going off topic

Back to the original question. After removing the team, performance has improved. Today I maximized the transmit and receive buffers, so I'll see this week if that helps. Next I will see if I can get them to go for the new smart switch.

I purchased the d-link dgs-1210-28. Do I just plug it in and resetup the team on the server and it will recognize it, or do I need to do additional configuration on the switch? Under "Link Aggregation" I found "LACP" port settings, but no guidance if I need to change anything.

You need to specify which ports will be part of the channel on the switch and plug the server into those specific ports.

OK, I've got it all configured and working. All ready paid off by helping me isolate some problems with an 8 port switch that was acting erratically. I've been looking through the documentation, to learn what other things I can do with this. What I don't seem to be able to find is a way to monitor the individual ports in a way so I know what IP address is coming in on them. Short of looking at all the lights, going to the workstation and unplugging the nic and then back to the switch to see what light went out, is there an easier way?

Get a second person and some walkie talkies, maybe.

This is where accurate labeling really comes into it's own.