TR Forums

Have a clients building where they finally upgraded from a 2mbps down 768k up to a 100/100 squared business connection. Needless to say we want to have the new connection up and live.

I've got an old Netgear FVS318g in place from the previous tech. The major issue is that not only is the old router a 10/100 WAN port, it's throughput is a paltry 7-8mbps. So I went about getting a new temporary router until I could get on order an Enterprise grade wired router with dual WAN for failover. I figured like any reasonable person that I could simply swap this router out with a new one, set the static IP from the ISP and off we go, right? Nope. Apparently if I run the router by itself with just a single workstation connected directly to it, I get the full 100mbps squared connection. The second I plug the rest of the network (roughly 15 computers and a server) the connection degrades and nearly full drops the second I attempt a speedtest. I randomly can ping through but notice it giving me packet loss. I have a Static WAN which is setup correct and DHCP disabled on the router to allow the Server 2008 R2 to handle DHCP and DNS.

Really confused because these settings all work on the old router and the second I unplug one from the other, everything magically comes back alive to it's 8mbps self. I even tried disabling DHCP and DNS on the server to let the router handle it, thinking the two were conflicting. It shouldn't be the router either because I tried a lightly used (factory reset) Linksys EA6100 and got the same results. Thinking it was the router I actually tossed the Linksys in the trash and later pulled it out after getting similar results from the Archer C7.

I've been hardcore sleep deprived over the last week getting this seasonal place caught up from 100mbps switches hidden in closets, new builds, new fiber lines ran, ect. Anyone have a "Hey dummy, check this" type of suggestion? I included an ultra crude diagram of the network layout. We have no choice but to run a single line from their restaurant to the hotel and use an intermediary switch, of which I recently replaced a 100mbps switch. Everything LAN has been working great for the last 2-3 days. It is only when I try swapping in a new more capable router that I get issues. Right now as a temp router I'm using a brand spanking new TP-Link Archer C7 FWIW.

Here is the ultra quick diagram
http://imgur.com/a/etIi7

Massive packet loss can mean any number of things. Since you said it works with the old router... I have a dumb question - is the new router interface set up for Auto?

It's possible that one side or the other is set for Single-Duplex, and that will generate just this kind of problem.

Question: Is the "new/old" router the Hotel one or the Restaurant one?
Question: Have you verified no packet loss from the Hotel switch to the Restaurant switch?
Question: You've run DHCP from both the router and the server - a Duplicate IP addresses will cause all kinds of havoc. Have you checked that they used different chunks of the range when giving out IPs? Example, 192.168.0.0/24 network, Server DHCP has 192.168.0.10-99, Router DHCP has 192.168.0.100-199.

Vhalidictes wrote:

Massive packet loss can mean any number of things. Since you said it works with the old router... I have a dumb question - is the new router interface set up for Auto?

It's possible that one side or the other is set for Single-Duplex, and that will generate just this kind of problem.

Question: Is the "new/old" router the Hotel one or the Restaurant one?
Question: Have you verified no packet loss from the Hotel switch to the Restaurant switch?
Question: You've run DHCP from both the router and the server - a Duplicate IP addresses will cause all kinds of havoc. Have you checked that they used different chunks of the range when giving out IPs? Example, 192.168.0.0/24 network, Server DHCP has 192.168.0.10-99, Router DHCP has 192.168.0.100-199.

Hey Vhalidctes, I'll answer in order of the questions.

New router WAN interface set to static with proper info from ISP.

Single duplex shouldn't be possible as these are new gigabit unmanaged switches.

The routers are always located in the Hotel, that is where the connection used to come in from the old ISP.

No packet loss from switch to switch for LAN traffic and when old router in place. They run all day with their hotel software that also is their POS (PMS) in the restaurant. The program is sensitive to network outages and bad traffic on the LAN. If it was dropping packets, I'd image I wouldn't be able to leave On-Site.

I disbaled DHCP on the router to avoid the possibility of IP conflicts. This is how the old router is setup, with DHCP disabled. I tried after having the issue to disable the DHCP on the server and to instead enable it on the router as well as DNS. The connection works great if I plug in the new router by itself with the internet coming from the restaurant switch (where the connection resides) to the WAN port and the single workstation connected to the routers internal 4 port switch.

From looking at the DHCP scope on the server it is simply a single large block. 192.168.0.10 to 192.168.0.100.

I confirm a ipconfig /release and renew to make sure the server is assigning me a dynamic address within that range and it does.

Instead of plugging in all of the network at once, what happens if you reconnect it in segments? Can you isolate the bog-down to one particular router or block of machines?

Also, what device are you trying to use as the temporary router in place of the FVS318g?

Do all the computers experience packet loss? Or just the one you have connected? Does pinging from the router also display packet loss? Does pinging google have packet loss? Pinging the connected computer? Pinging one of the 15 on the network?

What do the firewall settings on the old router look like?

Is there packet loss between computers on the network with the new router? (E.g. traffic that should only go across the switches).

I'd check IP settings on all the computers to see if they are all getting them dynamically from the server or if any have a static configuration. But do that after you break down connections into segments like the previous poster suggested.

Is this the rough network diagram?:

Oh, and what network is associated with each interface on each router and what is its the routing table?

From the diagram it looks like you've got the wan of the router connected to the same switch as the lan? I must be missing something.

cheesyking wrote:

From the diagram it looks like you've got the wan of the router connected to the same switch as the lan? I must be missing something.

To be clear, I am asking if what I put was the setup. I didn't include the WAN. I'm assuming the Linksys router has multiple ports that can be used to router between networks or as a switch.

Drachasor wrote:

cheesyking wrote:

From the diagram it looks like you've got the wan of the router connected to the same switch as the lan? I must be missing something.

To be clear, I am asking if what I put was the setup. I didn't include the WAN. I'm assuming the Linksys router has multiple ports that can be used to router between networks or as a switch.

Sorry, was referring to the diagram Welch put up

cheesyking wrote:

Drachasor wrote:

cheesyking wrote:

From the diagram it looks like you've got the wan of the router connected to the same switch as the lan? I must be missing something.

To be clear, I am asking if what I put was the setup. I didn't include the WAN. I'm assuming the Linksys router has multiple ports that can be used to router between networks or as a switch.

Sorry, was referring to the diagram Welch put up

Oops, I missed that!

This is what happens when I respond to a thread while I have migraine.

To the OP: What is the second router (in the restaurant) used for?
Did you make sure the new/old router have the same private IP set on them?

So, duplex mismatch can happen with gigabit switches if the auto negotiate fails at the highest speed. Switches will default to the lowest speed, 10/half. There should be some lights indicating what they negotiated to. You can test by pinging the router from a client PC on each segment (or switch, in this case). If there is a duplex mismatch, you'll see drops in the ping, or overly long responses.

I can also see a case where, depending on how smart the switches are, the router just assumed 100/full, and the switch assumed 1000/full. I would also check the cables, gigabit switches generally don't care what gets plugged into them, but make sure you've got CAT5e or better straight-through cables in all connections. A crossover cable from a switch to the router might also be a culprit.

One thing you might do is run a pfSense box as the router/NAT box instead of the ones you have. It would be a lot of work cobbling together the hardware, but pfSense is also a lot more capable.

Sorry for the late response, I've been somewhat away due to work and being sick, I sound like a 40 year long smoker this week :|. I'll try to gather together answers to all of the questions. But really quick, let me give everyone the run down in order of equipment.

1. ISP's (Router)
2. 24 Port Gigabit Switch (unmanaged)
3. 8 Port TP-Link Gigabit Switch (unmanaged) - This is simply to continue a signal for a line that would have been over 350ft.
4. 8 Port TP-Link Gigabit Switch (unmanaged) - This connects only 3 "LAN" lines.
5. TP-Link C7 Archer (Router)

I did a little bit of further testing with the ISP and they seem to see ARP requests from the C7 Archer router, although the last 4 of the MAC show up 1 letter off (shows as a D instead of C), they said this is actually really common. In number 4 above I ONLY connected 1 of the switch ports to the C7's WAN port, removed the other 3 lines that lead to various switches in the hotel, effectively eliminating any sort of IP duplicate/DHCP issues. Ran a large packet ping to my website and on packet 48/50 the entire connection drops and refuses to reconnect unless I power cycle things and even then it takes a very long time to reconnect. This ping was done within the routers admin tools page, not a workstation. The workstation I accessed this over was the ONLY thing plugged into a single LAN port on the C7 router. Such a strange issue, also confirmed with the ISP that their unit is running Gigabit full-duplex via Auto negotiate to the C7.

It seems like I may have something broadcasting over the restaurant's side of the LAN. They only have 6 computers on that side and nothing fancy. I will be assigned 2 new static IPs from the ISP today so that I can test the connection without having to take down the entire network to test this issue out. I'm going to attempt to connect directly the ISPs router to our router to eliminate EVERYTHING else.

I also wonder if it is possible that with the long run from the restaurant to the hotel with that intermediary switch, if their router confirms auto neg for gigabit and then has a degraded connection and instead of stepping back down to 100mbps it just loses integrity and drops all together. This is a very strange issue.

So is the ISP router configured to do the restaurant traffic on a private lan on one public IP and pass through another public IP to your network? Are there specific ports on the ISP router for the two sets of traffic or is it all on just one port? The network just doesn't seem to make any sense to me.

The way I'd expect something like this to be done would be a single router (either the ISP one or something with more features connected directly to the isp router) managing both the restaurant network and your one on separate vlans. Then managed switches carrying both the networks (ideally over fibre so you don't need that extra switch) between the two buildings and splitting the traffic back out again at the other side.

I expect running fibre is out of the question but cheapo smart switches aren't that expensive (would support fibre if you ever got that far) and neither is a router that can handle that sort of vlan stuff. <fanboy Mikotik plug> A Mikrotik HEX for example is less than that Archer and can also do traffic shaping to prioritise your network's traffic over the hotel. If you do have multiple public IPs or connections and are paying extra for it you could even do away with that and have a single IP or connection. It can also do the dual wan stuff you want. </fanboy> (it's still not a daft idea to keep separate public IPs / connections but you don't need them).

Assuming you even need that traffic in the hotel, otherwise you'd just have the restaurant traffic on one router port connected to the switch in the restaurant and the cable to the hotel connected directly to another port on the router.

I have a feeling you've got some kind of loop or maybe a connection between the two networks you aren't expecting.

EDIT: some stuff and changed hotel to restaurant.

Good questions/suggestions Cheesy. I'd love for it to have been fiber, but that line is under buildings and cement from probably close to a decade ago, the line tests good though.

We don't need the restaurant and hotel to be separate public IPs for any reason. We will have (once they finally deliver) a 3 static IP block from the ISP. One of these will be for the admin network, the other is going to a wireless system setup by another company that blankets the "resort". The third is just in case we need one for later use with the RV park that is currently on it's own internet with another ISP.

The C7 Archer was just a temporary solution and I fully planned on getting an enterprise grade dual WAN router. I was considering some more Ubiquiti equipment but not sure just yet. At this point I just want to make sure I can get it to work with this equipment and can worry about the details of a new router afterwards. It does very much feel like a loop and any of those Cisco 200 series + business managed switches I've used in the past have loop back detection. I also noticed last night while testing that when bringing the old router back online that the DNS service, running on the server, was not functioning but showed it was up. A simple service stop/start brought the network back to life.

/sigh, of course this had to be held off until their opening week and I've got people breathing down my neck to make a 10 year old network function with all sorts of new stuff. Doesn't help that the local ISP ALWAYS screws me when it comes to their static IP assignment. Can't wait for one of theirs to tell me I never requested it, I keep phone call recordings automatically (legal in my state).

A shot in the dark here, but the FVS318G is old enough that it doesn't support many modern features like jumbo frames and VLANs. When you install a newer router, does the network attempt to use some newer features and choke on them?

Another possibility is that the newer units might have a UDP/TCP flood protection setting that's too low and gets triggered when a bunch of starved devices all try to reconnect to the Internet simultaneously.

If you had a managed switch you could setup a port mirror and actually see what's going on with wireshark or something.

I think we really need some details on what subnets and IPs are being used, maybe a full network diagram. The way you describe how things are laid out it sounds like ISP router has a single public IP which it nats to a private subnet. You then have a long run to the hotel where you have the new router doing a double nat to another private subnet. But then you talk about that router having a static IP assigned by the ISP which make it sound like you've got at least two networks going through a single unmanaged switch which doesn't sound right. The first diagram you posted even makes it look like you've got the wan and lan ports of the new router connected to the same switch which just makes no sense.

Also, if there's a POS system maybe that should be isolated from the rest of the network.

"the line tests good though" Is that tested with one of those simple continuity testers or because you've sent lots of traffic over it at 1Gbps without any losses? Can't remember if this has already been suggested but maybe that long run can't handle anything more than 100Mb reliably (could the old router even have its wan port set to 10Mb since it's all it needs with 8Mb throughput).

To add to what Cheesyking said... it really doesn't matter if the line tester works (although that's a good sign).

Network equipment, even SOHO stuff, will automagically set itself to a lower speed if it sees any data corruption. So it's the production equipment that sets tolerances, not the engineer.

I've "fixed" iffy long copper runs by swapping out network cards for different models that either can handle more noise or have looser tolerances.

Have an update for you guys.

When I was saying tested good I meant lots of traffic including large single files transferred back and forth without error or issue. Learned a long time ago that continuity in a single moment means nothing for a network, especially long runs when dealing with things like IP sec cameras.

So the issue was in fact a loopback. Little did I know that after the intermediary switch below a cabin there was another network switch in their boiler room (wish I was making this up) where the connection comes into next. From here the connection goes into a 16 port switch of which 2 were connected to the switch inside the building, causing a loopback.

I attached a single 8 port managed switch off of the ISP 0/0 eth port to allow 2-3 of our new static IPs to be able to function without interfering with each other. This is ONLY for WAN connections.

Once the end of season comes I think it may be time to remove a bunch of random unmanaged switches and centralized as much as possible to some quality managed ones to consolidate. We are able to get our full 100mbps down and oddly enough varying speeds for the upload ranging from 50mbps to 20mbps. The further away of the network the worst the upload gets.

I bet finding that switch was a massive relief.

I recommend beer, you deserve it.

cheesyking wrote:

I bet finding that switch was a massive relief.

I recommend beer, you deserve it.

Yes sir, the biggest of reliefs. I went from having their COO sounding like he was disappointed and calling me every few hours for a few days to this little gem of a quote this morning.

"Feels like you are truly a part of our team, so thank you, thank you and thank you again."

/Sigh, now onto the next problem

Hah, a loop. Glad you got that sorted out!

Is-or was-there any documentation on the network?

glad you found that one. network loops can be a pain..

like at this new job i just got, the previous admin purposely disabled STP/RSTP on ALL of the switches. like "no rstp" on the CLI.
still trying to figure out why...

wizardz wrote:

glad you found that one. network loops can be a pain..

like at this new job i just got, the previous admin purposely disabled STP/RSTP on ALL of the switches. like "no rstp" on the CLI.
still trying to figure out why...

A lot of network engineers hate STP, especially when they configure it improperly and then experience problems. There's also a subset of techs that don't understand Portfast and wonder why access ports take 15+ seconds to come up when enabled.

Generally I find RSTP to be set-and-forget, although I try to limit the size of L2 domains that I work with and wherever possible use switch stacks instead of a swarm of smaller switches.

I started to respond early on and then stopped myself because (1) I know there are far more knowledgeable network experts here, and (2) diagnosing problems with an inherited network configuration is a nightmare I don't ever want to be involved with again. But now, since the OP problem has been solved and we're doing tangents, I'll just relate a story relative to the latter:

Years ago I had a friend who founded a Point of Sale company that specialized in restaurant POS systems. It's large and surprisingly diverse market, and they were targeting the low end where their primary competition (at that time) was cash registers. Generally these were one- or two-terminal systems going into mom and pop restaurants that had never had a POS system before, so there was a lot of education involved. I sometimes helped out on the larger installation jobs, more as a favor than a job, and I learned a lot about restaurants as businesses and a lot more about the nightmares of putting delicate electronic equipment into an environmentally hostile environment where it was expected to run reliably as business-critical hardware despite user abuse and just about every random mishap you can imagine.

We quickly learned that the two absolute banes to our existence were (1) dirty power and (2) everything to do with networking. The former was mostly fixed when we started requiring the purchase of real power conditioning equipment -- not the cheap consumer "surge suppressors" but the kind of stuff that had enough metal coils inside it that you'd hurt your foot if you kicked it. Even better, we tried to insist that the PoS systems and their associated hardware got put on their own electrical circuit, but unless it was a brand-new, full build-out we're rarely get that. Usually you had to make do with whatever circuits were already in the space. We'd make sure the equipment didn't go on the same circuit as things like ice makers and dishwashers, but sooner or later we'd get a call that a terminal was rebooting "randomly" or nothing was working and they could smell melted plastic, and we'd inevitably discover that somebody had been moving plugs around and stuck an unprotected system or switch on a different circuit and was suffering from brown-outs or surges every time a fridge cycled on or off.

Networking (modulo the aforementioned icemaker-induced switch burn-outs) tended to be more stable in use, but could be a complete nightmare to set up in an already up-and-running restaurant. And the worst was if there was existing ethernet wiring in the space. We eventually came to so mistrust pre-existing networks that the company would always give a bid assuming a fresh wire run, because it was usually more expensive to try to use, and then troubleshoot, an existing layout. I fortunately missed a lot of the installs that drove that policy, but my favorite experience was in a very "dive-bar" sort of tavern, which surprisingly already had Cat5 cable running through it. We tried to use one particularly long run and found there was no signal going through whatsoever, Since we were going to have to use the same endpoints, and therefore presumably the same cable routing, we traced the cable out of the back of the house and into the bar. The guy I was working with, who had started from the other end, stopped and said "Well, there's your problem." He then used his penknife to pry a bullet out of the baseboard, where it had precisely severed the ethernet cable.

UberGerbil wrote:

my favorite experience was in a very "dive-bar" sort of tavern, which surprisingly already had Cat5 cable running through it. We tried to use one particularly long run and found there was no signal going through whatsoever, Since we were going to have to use the same endpoints, and therefore presumably the same cable routing, we traced the cable out of the back of the house and into the bar. The guy I was working with, who had started from the other end, stopped and said "Well, there's your problem." He then used his penknife to pry a bullet out of the baseboard, where it had precisely severed the ethernet cable.

"Looks like your Ethernet is shot."