Personal computing discussed

Moderators: Steel, notfred

 
DragonDaddyBear
Gerbil Elite
Posts: 652
Joined: Fri Jan 30, 2009 8:01 am

Re: Cisco ASA keeps killing my SSH connections

Fri Dec 01, 2017 9:48 am

Wow, that's an epic fail.

Maybe your issues stem from that and some kind of network confusion because there are two routes to your destination.
 
Glorious
Gold subscriber
Grand Admiral Gerbil
Posts: 10303
Joined: Tue Aug 27, 2002 6:35 pm

Re: Cisco ASA keeps killing my SSH connections

Fri Dec 01, 2017 12:15 pm

Darkmage wrote:
Y'all aren't going to believe this


Oh, I believe it alright haha
 
Vhalidictes
Gold subscriber
Gerbil Jedi
Posts: 1682
Joined: Fri Jan 07, 2005 2:32 pm
Location: Paragon City, RI

Re: Cisco ASA keeps killing my SSH connections

Fri Dec 01, 2017 1:15 pm

DragonDaddyBear wrote:
Wow, that's an epic fail.

Maybe your issues stem from that and some kind of network confusion because there are two routes to your destination.


Go Go Asymmetric Routing!

(This ranges from problematic to Bad, BTW. Hopefully inside to outside traffic isn't still using the ASA as a gateway.)
 
the
Gold subscriber
Gerbil Elite
Posts: 851
Joined: Tue Jun 29, 2010 2:26 am

Re: Cisco ASA keeps killing my SSH connections

Fri Dec 01, 2017 3:10 pm

This may an odd ball and I haven't dived too deep into this thread, but if the script is calling the keep alive function and the connection is dropping every 5 minutes or so, is that when then connection actually drops in that script? IE does calling that function force the VPN to drop due to a change in configuration?
Dual Opteron 6376, 96 GB DDR3, Asus KGPE-D16, GTX 970
Mac Pro Dual Xeon E5645, 48 GB DDR3, GTX 770
Core i7 3930K@4.2 Ghz, 32 GB DDR3, GA-X79-UP5-Wifi
Core i7 2600K@4.4 Ghz, 16 GB DDR3, GTX 970, GA-X68XP-UD4
 
DragonDaddyBear
Gerbil Elite
Posts: 652
Joined: Fri Jan 30, 2009 8:01 am

Re: Cisco ASA keeps killing my SSH connections

Fri Dec 01, 2017 6:54 pm

I've actuality seen where redundant connections cause a partial routing loop. The links balanced and when packets were looping out caused enough load on one link where the network would route a different way and get through.

Have you tried a few traceroutes?
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8012
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Sat Dec 02, 2017 9:19 am

the wrote:
This may an odd ball and I haven't dived too deep into this thread, but if the script is calling the keep alive function and the connection is dropping every 5 minutes or so, is that when then connection actually drops in that script? IE does calling that function force the VPN to drop due to a change in configuration?

Let me start from the beginning, since I see I've lost a couple people.

Customer has two full installations of the product. We'll call them Working and Broken. Product consists of a VM issuing commands to a cloud provider to spin up a cloud machine. Product attempts repeated SSH logins until it connects successfully, whereupon it knows the cloud machine is alive. At that point, product uploads a half-dozen script files that consist mainly of installation commands (OpenVPN, Java SDK, etc.). Product then runs the script remotely and streams the output back to Product where it is logged.

The Broken installation consists of a VM Ubuntu instance running through a pair of Cisco ASA firewalls configured in routed mode with failover support. During the run of the installation script, the ASA terminates all SSH connections to that cloud machine. This is a problem, as the Product no longer sees the completion of the installation script and does not know when to run the next script in the sequence.

The Working installation consists of a VM Ubuntu instance running through a single Cisco ASA firewall (slightly larger, more ports, same software version) configured in transparent mode upstairs in the same building. Using the same internet provider. Or at least... I thought so before Thursday, where I just learned that they have bypassed the ASA firewall entirely. Which may explain why Working actually works.

Customer has loaned me a spare ASA firewall, which I am using for testing.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
just brew it!
Gold subscriber
Administrator
Posts: 49736
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Cisco ASA keeps killing my SSH connections

Sat Dec 02, 2017 9:36 am

I'm betting having identical (or at least near-identical) hardware you can poke at is going to help a lot. It can be difficult to troubleshoot remotely, and even more so if you're trying to piece together what happened by looking at post-mortem log output.

A lot of the issues I deal with fall into the latter category, where all we have is log data (sometimes hundreds of MB of it from multiple related subsystems) to wade through to try and come up with a diagnosis.

And then there are the issues where the symptoms point at buggy HDD firmware, but the customer won't allow the drives to be shipped back to the HDD manufacturer for a post-mortem analysis due to privacy/security concerns. Pointing out that the data is encrypted by our software (and the HDD manufacturer does not have the decryption key) goes nowhere when dealing with an organization that says "that's against our rules, no exceptions".
Nostalgia isn't what it used to be.
 
Seeroftime
Gerbil In Training
Posts: 6
Joined: Sat Dec 02, 2017 11:21 am

Re: Cisco ASA keeps killing my SSH connections

Sat Dec 02, 2017 3:00 pm

Dark Mage, I have to ask, are these running new versions of the ASA IOS? The reason I ask is that on the newer releases of the ASA software, it automatically kills sessions for seemingly no reason at all. One of our vendors pointed that out to us when our Firewall got changed from a Palo Alto 3020 to an ASA 5516 this summer and our PMS suite would disconnect after 15 minutes, and it seems to ignore any keep alive messages sent by the system. The chief programmer for them mentioned that he's seen it with everyone that has an ASA running the new software. It also auto-terminates RDP sessions on us, though anything that isn't routed through that firewall is unaffected, as long as we remain active in the RDP sessions, they stay connected.
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8012
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Sun Dec 10, 2017 7:54 am

Seeroftime wrote:
Dark Mage, I have to ask, are these running new versions of the ASA IOS?
Define "new". I think the version is mentioned in here, somewhere towards the beginning. It's not the absolute latest, but it's pretty close. These firewalls are less than a year old.

Edited to add: Found it. Version 9.6 of the ASA software.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
Vhalidictes
Gold subscriber
Gerbil Jedi
Posts: 1682
Joined: Fri Jan 07, 2005 2:32 pm
Location: Paragon City, RI

Re: Cisco ASA keeps killing my SSH connections

Mon Dec 11, 2017 11:49 am

Darkmage wrote:
Seeroftime wrote:
Dark Mage, I have to ask, are these running new versions of the ASA IOS?
Define "new". I think the version is mentioned in here, somewhere towards the beginning. It's not the absolute latest, but it's pretty close. These firewalls are less than a year old.

Edited to add: Found it. Version 9.6 of the ASA software.


Depending on which version of the ASA you're using (Regular or X) 9.6.x is all you'll ever have because they are End-of-Support some time in 2018.
 
notfred
Maximum Gerbil
Posts: 4393
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: Cisco ASA keeps killing my SSH connections

Tue Dec 12, 2017 2:50 pm

I still have my money on a TCP window scaling issue as JBI and I suggested back in July.

Broken setup is on a redundant pair of ASA's. If the TCP flow switches from one to the other then the window scale factor gets lost and now the packets are being sent outside the TCP window as far as the ASA and its stateful firewall is concerned so it will TCP reset.

Working setup is on a single ASA so it saw the TCP 3 way handshake and always knows the window scale factor.
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8012
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Tue Dec 12, 2017 3:00 pm

Sigh. I confirmed it yesterday. Broken setup goes through a pair of ASAs set in failover mode. Working setup bypasses the ASA entirely. Crap.

Other duties have reared their head for the moment. I'm shelving this until the week after Christmas. I'll bang on it again at that point.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus

Who is online

Users browsing this forum: No registered users and 1 guest