TR Forums

Wow, that's an epic fail.

Maybe your issues stem from that and some kind of network confusion because there are two routes to your destination.

Darkmage wrote:

Y'all aren't going to believe this

Oh, I believe it alright haha

DragonDaddyBear wrote:

Wow, that's an epic fail.

Maybe your issues stem from that and some kind of network confusion because there are two routes to your destination.

Go Go Asymmetric Routing!

(This ranges from problematic to Bad, BTW. Hopefully inside to outside traffic isn't still using the ASA as a gateway.)

This may an odd ball and I haven't dived too deep into this thread, but if the script is calling the keep alive function and the connection is dropping every 5 minutes or so, is that when then connection actually drops in that script? IE does calling that function force the VPN to drop due to a change in configuration?

I've actuality seen where redundant connections cause a partial routing loop. The links balanced and when packets were looping out caused enough load on one link where the network would route a different way and get through.

Have you tried a few traceroutes?

the wrote:

This may an odd ball and I haven't dived too deep into this thread, but if the script is calling the keep alive function and the connection is dropping every 5 minutes or so, is that when then connection actually drops in that script? IE does calling that function force the VPN to drop due to a change in configuration?

Let me start from the beginning, since I see I've lost a couple people.

Customer has two full installations of the product. We'll call them Working and Broken. Product consists of a VM issuing commands to a cloud provider to spin up a cloud machine. Product attempts repeated SSH logins until it connects successfully, whereupon it knows the cloud machine is alive. At that point, product uploads a half-dozen script files that consist mainly of installation commands (OpenVPN, Java SDK, etc.). Product then runs the script remotely and streams the output back to Product where it is logged.

The Broken installation consists of a VM Ubuntu instance running through a pair of Cisco ASA firewalls configured in routed mode with failover support. During the run of the installation script, the ASA terminates all SSH connections to that cloud machine. This is a problem, as the Product no longer sees the completion of the installation script and does not know when to run the next script in the sequence.

The Working installation consists of a VM Ubuntu instance running through a single Cisco ASA firewall (slightly larger, more ports, same software version) configured in transparent mode upstairs in the same building. Using the same internet provider. Or at least... I thought so before Thursday, where I just learned that they have bypassed the ASA firewall entirely. Which may explain why Working actually works.

Customer has loaned me a spare ASA firewall, which I am using for testing.

I'm betting having identical (or at least near-identical) hardware you can poke at is going to help a lot. It can be difficult to troubleshoot remotely, and even more so if you're trying to piece together what happened by looking at post-mortem log output.

A lot of the issues I deal with fall into the latter category, where all we have is log data (sometimes hundreds of MB of it from multiple related subsystems) to wade through to try and come up with a diagnosis.

And then there are the issues where the symptoms point at buggy HDD firmware, but the customer won't allow the drives to be shipped back to the HDD manufacturer for a post-mortem analysis due to privacy/security concerns. Pointing out that the data is encrypted by our software (and the HDD manufacturer does not have the decryption key) goes nowhere when dealing with an organization that says "that's against our rules, no exceptions".

Dark Mage, I have to ask, are these running new versions of the ASA IOS? The reason I ask is that on the newer releases of the ASA software, it automatically kills sessions for seemingly no reason at all. One of our vendors pointed that out to us when our Firewall got changed from a Palo Alto 3020 to an ASA 5516 this summer and our PMS suite would disconnect after 15 minutes, and it seems to ignore any keep alive messages sent by the system. The chief programmer for them mentioned that he's seen it with everyone that has an ASA running the new software. It also auto-terminates RDP sessions on us, though anything that isn't routed through that firewall is unaffected, as long as we remain active in the RDP sessions, they stay connected.

Seeroftime wrote:

Dark Mage, I have to ask, are these running new versions of the ASA IOS?

Define "new". I think the version is mentioned in here, somewhere towards the beginning. It's not the absolute latest, but it's pretty close. These firewalls are less than a year old.

Edited to add: Found it. Version 9.6 of the ASA software.

Darkmage wrote:

Seeroftime wrote:

Dark Mage, I have to ask, are these running new versions of the ASA IOS?

Define "new". I think the version is mentioned in here, somewhere towards the beginning. It's not the absolute latest, but it's pretty close. These firewalls are less than a year old.

Edited to add: Found it. Version 9.6 of the ASA software.

Depending on which version of the ASA you're using (Regular or X) 9.6.x is all you'll ever have because they are End-of-Support some time in 2018.

I still have my money on a TCP window scaling issue as JBI and I suggested back in July.

Broken setup is on a redundant pair of ASA's. If the TCP flow switches from one to the other then the window scale factor gets lost and now the packets are being sent outside the TCP window as far as the ASA and its stateful firewall is concerned so it will TCP reset.

Working setup is on a single ASA so it saw the TCP 3 way handshake and always knows the window scale factor.

Sigh. I confirmed it yesterday. Broken setup goes through a pair of ASAs set in failover mode. Working setup bypasses the ASA entirely. Crap.

Other duties have reared their head for the moment. I'm shelving this until the week after Christmas. I'll bang on it again at that point.

Have you done any packet captures and looked at them in Wireshark? Have you checked to see if the TCP timestamps in the capture reach the ceiling of 2^32 at the point when the SSH connection dies? There is a bug in ASA 9.6.1 where the firewall incorrectly drops packets as PAWS failure. The bug is fixed in software 9.6.2(4) or 9.6.3.

If you have a Cisco account, the bug ID is CSCuq80704.