Personal computing discussed

Moderators: Steel, notfred

 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Cisco ASA keeps killing my SSH connections

Thu Jul 27, 2017 9:26 pm

I'm supporting a customer's rather unique implementation of our product. Suffice to say, they're not doing it the way we recommend and it's causing issues. I'm on the hook to get them up & running because we need the business.

Our engine creates a VM in a cloud provider and opens up an SSH connection through a Cisco ASA firewall to that VM. Once open, the engine runs a few scripts remotely to install a bunch of packages and configure the VM for our needs. We take the stdout from the install process and log it on the engine through the SSH connection. This normally works without a hitch, except for that #$?@#!! ASA.

The ASA has been terminating our SSH connections, usually at 5 minutes or so. I've tried just about everything I can Google up. I've added the KeepAliveInterval parameter to the OpenSSH call in the script. I've added KeepAliveInterval parameter to the /etc/ssh/ssh_config file. I've extended the timeouts for connections, half-closed connections and SSH management connections to the ASA. Nothing seems to have any effect. I upped the log level on OpenSSH client and couldn't find a single keep alive message in the log. It's as if the parameter is being ignored.

It's as if every Stack Exchange, Cisco Support Forum and blog post I can reach in English is lying to me. It's driving me up the wall.

Does anyone have any ideas?
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Thu Jul 27, 2017 9:33 pm

I should clarify. The connection isn't idle, really. It's streaming the stdout of the installation process. I can watch stuff scroll across my screen if I put a tail on the log file on the engine. But at five minutes, the log file just... stops. The ServerKeepAliveInterval and the ServerKeepAliveCountMax only seem to affect how long before the engine determines that the connection has been killed. Whatever keep alive it is supposed to be doing, it's not keeping it alive.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
Drachasor
Gerbil
Posts: 34
Joined: Sat Apr 22, 2006 3:35 pm

Re: Cisco ASA keeps killing my SSH connections

Fri Jul 28, 2017 12:24 am

I'm not convinced this is a problem with the ASA. Once you have a connection, if there is information constantly going back and forth, then it shouldn't be blocking that connection. Unless there is something changing an IP or port on the receiving end. You can examine the traffic coming into the ASA and determine if it is deciding to block your traffic or not though. You can also do a data capture on the ASA to capture the traffic and see if there isn't a clue in the packets being sent back and forth. Also, there should be no need for a keepalive if the connection is constantly having information go across -- SSH isn't going to think a connection is idle if there's currently traffic going through it, I don't think. Perhaps since you have the keep alive setup, then getting no response might kill the connection.

Are there firewalls on the server or client that could be blocking some of the traffic?

There are settings on the ASA to allow it to keep a connection open longer too. You could try upping the timeout for this connection to see if for some reason that's causing the problem. Though again, that should require no traffic on the connection for that time. It sounds like you did that though, and if it is having no effect, then that shouldn't be causing the problem. For fun you could try lowering the time to see if that causes a problem.

To be clear, your SSH connection passes through the ASA, it doesn't connect to the ASA, correct? Because that would be weird.

Seems like a client or server issue though. Seems like one of them is killing the connection or something is crashing or something else.
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Fri Jul 28, 2017 9:18 am

I'm convinced it's a problem with the ASA because when we bypass the ASA it works.

We're successfully SSH into the cloud VM, uploading a bunch of scripts and then running one of them remotely. It installs a bunch of stuff and we stream the stdout back to the engine and write it to log. If I do this at the office which uses a different brand of firewall, no problem. If I do it from my house, using the firewall in my router, no problem. It's only at the customer site, using these particular ASAs in high availability mode. Turning off one of the ASAs didn't affect anything either.

Modifying the connection timeout (currently set at 1 hour) didn't change anything. Changing the half-closed timeout didn't change anything. Scouring the internet for a solution shows several people setting the keep alive on the server should fix the problem. Other just extend the TCP connection timeout to something long, like 2 hours. But it's not working in my case as the connection dies right at five minutes.

To answer your question, we're connecting through the ASA not directly to it.

It's driving me up the wall.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
Glorious
Gold subscriber
Grand Admiral Gerbil
Posts: 10266
Joined: Tue Aug 27, 2002 6:35 pm

Re: Cisco ASA keeps killing my SSH connections

Fri Jul 28, 2017 10:39 am

Darkmage wrote:
To answer your question, we're connecting through the ASA not directly to it.

It's driving me up the wall.


Have you run tcpdump and looked for any fin packets? If you do see them, does the connection actually end on a rst? (i.e. look for flags F and R)

This sounds like half-closed connections getting you nailed by the half-closed timeout. Possibly by the TCP session time overall, but that seems sort of unlikely. Likewise with idle, but even less likely because it isn't actually idle as you...

In other words, you're totally on the right track but somehow the solutions you are trying aren't properly working for whatever reason.

We ran into something very much like this with Palo Alto, but I'm not really a networking guy (and organizationally I have no access to switches and firewalls). In effect it was closing (admittedly idle) connections BEFORE the idle timeout, so they didn't believe us when we complained. We had to prove it. So we did.

Turned out there is some dynamic adjustment that overrides that timeout when the session table has > X number of sessions or something like that, and that's how the thing works either by default or (more likely) the profile they cut and pasted from.

Of course they wouldn't precisely tell us and I can't look and find out.

The key here is that these things usually rst on idle but DON'T on half-closed. They just clear the session table.

Other than that, I don't know, this might be some hierarchy thing where you're changing the setting globally but it's overridden by an per port/application setting somewhere. Kind of analogous to my example, there was an non-obvious setting somewhere that overrode the obvious one leading to finger-pointing and teeth-gnashing.
Last edited by Glorious on Fri Jul 28, 2017 10:54 am, edited 1 time in total.
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Fri Jul 28, 2017 10:47 am

Well, some odd developments. It doesn't appear to be a timeout issue, despite it happening regularly at 5 minutes when our script is running.

I turned off the keepalives on the VM and manually opened an SSH session from the engine into it. I just sat there for ten minutes and the connection was just fine. So it's apparently not time-limited.

Perhaps it's size-limited? After a certain number of kilobytes it just dies? That seems... weird.

The guys who own the firewall tell me the firewall configuration is basically the stock one, with some high-availability stuff added. I dunno if that matters.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
just brew it!
Gold subscriber
Administrator
Posts: 49671
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Cisco ASA keeps killing my SSH connections

Fri Jul 28, 2017 10:56 am

Try disabling TCP window scaling.
Nostalgia isn't what it used to be.
 
Glorious
Gold subscriber
Grand Admiral Gerbil
Posts: 10266
Joined: Tue Aug 27, 2002 6:35 pm

Re: Cisco ASA keeps killing my SSH connections

Fri Jul 28, 2017 11:01 am

Darkmage wrote:
We take the stdout from the install process and log it on the engine through the SSH connection.


This is the part that makes me think half-closed, because that kind of thing is the reason why fin exists in TCP/IP: Denoting the end of an otherwise ambiguous data stream.

Hence, you wouldn't see it just by ssh and letting it sit, you'd only see it when you did your actual install process. And it would only be a problem with the ASA sitting inbetween and applying its half-closed policy.

EDIT: Like, the stream providing the logging ends, sending fin, but the connection is still up because you're still sending data/commands the other way and you're not done with that yet. ASA comes along, notices the first fin, and then after some timer, says sayonara to your session.

EDIT2: This is why I'd look at tcpdump, because even though you've already monkeyed with the half-closed timeout god only knows...
 
Vhalidictes
Gold subscriber
Gerbil Jedi
Posts: 1648
Joined: Fri Jan 07, 2005 2:32 pm
Location: Paragon City, RI

Re: Cisco ASA keeps killing my SSH connections

Fri Jul 28, 2017 11:37 am

Darkmage wrote:
Well, some odd developments. It doesn't appear to be a timeout issue, despite it happening regularly at 5 minutes when our script is running.

I turned off the keepalives on the VM and manually opened an SSH session from the engine into it. I just sat there for ten minutes and the connection was just fine. So it's apparently not time-limited.

Perhaps it's size-limited? After a certain number of kilobytes it just dies? That seems... weird.

The guys who own the firewall tell me the firewall configuration is basically the stock one, with some high-availability stuff added. I dunno if that matters.


Darkmage, I work with ASAs as a day-job.

The red flag to me was the size-limited question. If that's what you're seeing, I have a few questions:
1) Is this SSH connection going through a VPN?
2) What version of code is this ASA on? There's... a lot of what we call "caveats" that might affect SSH traffic, mostly on 8.x.x versions.

As some other people have said, being able to check the ASA traffic logs will probably help pinpoint what's going on.

EDIT: In theory couldn't the firewall owner open up a TAC case?
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Fri Jul 28, 2017 12:05 pm

Vhalidictes wrote:
Darkmage, I work with ASAs as a day-job.

The red flag to me was the size-limited question. If that's what you're seeing, I have a few questions:
Update: I'm not sure if it's size-related. I took the log file that our process is writing to, SCPd it up to the VM and then opened an SSH session manually. I then ran cat on the full file (which contained about three full logs from failed instances) five times in succession. The manual SSH connection stays up, no matter what I throw at it. So it doesn't appear to be size, per se. Nor time, as manual SSH connections can just sit there for well above the observed time limit.

Vhalidictes wrote:
1) Is this SSH connection going through a VPN?
No. We haven't set up our VPN stuff yet by this point.
Vhalidictes wrote:
2) What version of code is this ASA on? There's... a lot of what we call "caveats" that might affect SSH traffic, mostly on 8.x.x versions.
9.6

Vhalidictes wrote:
As some other people have said, being able to check the ASA traffic logs will probably help pinpoint what's going on.
Very well. I'll see if I can sit on the ASA as the process runs. Do you have an idiot's guide to displaying traffic logs on an ASA? I'm very new to these things.

Vhalidictes wrote:
EDIT: In theory couldn't the firewall owner open up a TAC case?
This is where we get into the politics of the situation. And like always, when politics are involved it's never a good thing. The short answer is no.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
Vhalidictes
Gold subscriber
Gerbil Jedi
Posts: 1648
Joined: Fri Jan 07, 2005 2:32 pm
Location: Paragon City, RI

Re: Cisco ASA keeps killing my SSH connections

Fri Jul 28, 2017 3:03 pm

Certainly. There's not a lot of really good guides and I don't know your (and the engineer's) skill levels, but this should be a good start for the basics:

http://vegaskid.net/2013/03/logging-opt ... cisco-asa/

Note: logging at a Debug level globally will kill most ASA models in minutes, assuming that your logging server can even keep up. If the normal logs aren't giving you enough info, you can debug the connection itself - you just need to be really specific. If you end up needing to, an old trick is to type in "undebug all" from the console and hit enter really fast if the firewalls starts to act up. YMMV.

These commands might also help:
"show conn <ip address>"
"show conn <ip address> state"
"show conn <ip address> detail"
 
notfred
Maximum Gerbil
Posts: 4390
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: Cisco ASA keeps killing my SSH connections

Fri Jul 28, 2017 5:24 pm

Have you tried JBI's suggestion of disabling TCP window scaling?

There used to be a lot of misconfigured firewalls out there that messed up the window scale option, I'm wondering if your ASA is one of those?
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Mon Jul 31, 2017 9:27 am

More updates: It appears my earlier claims of normal SSH connections being fine were premature. I tried to duplicate our installation process by killing the engine as soon as it connected to the VM and then SSHing into the VM directly. Once on the VM, I ran the first of our scripts to see if there was anything weird.

The script ran and completed without errors. After I pulled up the folder list to select the next script, the connection died on me. So what we have now is that the stream from our script goes just fine and I still have a connection that I can use to get responses from the VM. But something in that process triggers the disconnect... eventually.

Weird.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
asterger
Gerbil In Training
Posts: 1
Joined: Mon Jul 31, 2017 10:46 am

Re: Cisco ASA keeps killing my SSH connections

Mon Jul 31, 2017 10:56 am

If you use Putty for SSH:

Within Putty Configuration: Connection -> SSH -> Kex -> Max minutes before rekey 2 (default was 60). Since changing this have not had a single SSH hang-up.

-- asterger
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Fri Aug 04, 2017 9:12 am

Update: They gave me a spare ASA to play with and try to duplicate the issue in our test lab. We're trying to set it up today. I'll post back here when we're set up. Thanks for the advice guys.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
Drachasor
Gerbil
Posts: 34
Joined: Sat Apr 22, 2006 3:35 pm

Re: Cisco ASA keeps killing my SSH connections

Sat Aug 05, 2017 4:23 am

Darkmage wrote:
More updates: It appears my earlier claims of normal SSH connections being fine were premature. I tried to duplicate our installation process by killing the engine as soon as it connected to the VM and then SSHing into the VM directly. Once on the VM, I ran the first of our scripts to see if there was anything weird.

The script ran and completed without errors. After I pulled up the folder list to select the next script, the connection died on me. So what we have now is that the stream from our script goes just fine and I still have a connection that I can use to get responses from the VM. But something in that process triggers the disconnect... eventually.

Weird.


How long does this script take to run? If you log in manually and hit enter every 10 seconds or so for 10 minutes, does it still time out?
And to be 100% clear, on what device do you setup the ServerAlive settings?

Try adding a rule to allow traffic initiated in the other direction from port 22 to any port -- if traffic is allowed to be initiated either way, then that should at least give you some more information.
I'd add log statements to the end of related rules and before the "deny all" make rules that deny this specific traffic with logging enabled (and do this in both directions). That should give you some more info to work on as well.

For packet captures the GUI has a wizard which makes things easier: https://www.cisco.com/c/en/us/support/d ... sa-00.html
Even a Wireshark or similar capture on either end would be a lot better than nothing. You really need to try to capture all the traffic going between these two devices (on all ports).

Can you provide the sanitized rules you are using for this traffic? Also, have you checked to make sure it is not hitting another rule first? How are you implementing the timeout changes on the FW?
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Wed Aug 09, 2017 12:44 pm

Drachasor wrote:
How long does this script take to run?
Depending on the cloud provider, eight minutes or so. There are actually about six scripts involved, but it's not making it past the 2nd one at best.
Drachasor wrote:
If you log in manually and hit enter every 10 seconds or so for 10 minutes, does it still time out?
If the installation script is running, then yes. If I'm just SSHing into some random box, then no.
Drachasor wrote:
And to be 100% clear, on what device do you setup the ServerAlive settings?
On our engine, behind the firewall. This is the box that spins up the VM in the cloud provider, pushes the scripts to the VM, runs the scripts and logs the output of the script locally.

Drachasor wrote:
Try adding a rule to allow traffic initiated in the other direction from port 22 to any port -- if traffic is allowed to be initiated either way, then that should at least give you some more information.
That's a good idea. I can't leave that in production, but it may be informative.

Drachasor wrote:
I'd add log statements to the end of related rules and before the "deny all" make rules that deny this specific traffic with logging enabled (and do this in both directions). That should give you some more info to work on as well.
I'm not sure I can make this happen. Cisco's "deny all" seems to be baked into the device and you have to enable each individual protocol by telling the firewall to inspect each type of connection. It's weird.

Drachasor wrote:
For packet captures the GUI has a wizard which makes things easier: https://www.cisco.com/c/en/us/support/d ... sa-00.html
Even a Wireshark or similar capture on either end would be a lot better than nothing. You really need to try to capture all the traffic going between these two devices (on all ports).
The GUI requires a service contract with Cisco. Well played, Cisco. Well played indeed.

Drachasor wrote:
Can you provide the sanitized rules you are using for this traffic?
Yeah, probably.
Drachasor wrote:
How are you implementing the timeout changes on the FW?
Cisco timeout settings:
timeout conn 1:00:00 half-closed 0:30:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
Kougar
Silver subscriber
Minister of Gerbil Affairs
Posts: 2024
Joined: Tue Dec 02, 2008 2:12 am
Location: Texas

Re: Cisco ASA keeps killing my SSH connections

Sun Aug 13, 2017 4:04 am

Darkmage wrote:
Drachasor wrote:
I'd add log statements to the end of related rules and before the "deny all" make rules that deny this specific traffic with logging enabled (and do this in both directions). That should give you some more info to work on as well.
I'm not sure I can make this happen. Cisco's "deny all" seems to be baked into the device and you have to enable each individual protocol by telling the firewall to inspect each type of connection. It's weird.


Yes, Cisco ASA's have an unwritten implicit DENY ALL statement at the end of every ACL. So exceptions need to be added and be very specific. My understanding is that it also means anything caught by this unwritten statement doesn't get logged, you need to create an explicit deny statement in the ACL before it will log packets blocked by that specific explicit deny parameter. http://blog.ine.com/2010/01/02/ccna-the ... -deny-all/

How did the second ASA go, guessing it exhibits the same problem?
 
Coldfirex
Graphmaster Gerbil
Posts: 1118
Joined: Wed Dec 26, 2001 7:00 pm
Location: College Station, TX
Contact:

Re: Cisco ASA keeps killing my SSH connections

Sun Aug 13, 2017 7:19 am

My guess its the threat detection and its shunning that connection for a period of time. You can either shut off that feature, or whitelist the connection IP in threat detection.
Your bargaining posture is highly dubious.
 
Coldfirex
Graphmaster Gerbil
Posts: 1118
Joined: Wed Dec 26, 2001 7:00 pm
Location: College Station, TX
Contact:

Re: Cisco ASA keeps killing my SSH connections

Mon Aug 14, 2017 9:03 am

"show threat-detection shun" to confirm this is the issue.
Your bargaining posture is highly dubious.
 
Drachasor
Gerbil
Posts: 34
Joined: Sat Apr 22, 2006 3:35 pm

Re: Cisco ASA keeps killing my SSH connections

Tue Aug 15, 2017 4:58 am

Kougar wrote:
Darkmage wrote:
Drachasor wrote:
I'd add log statements to the end of related rules and before the "deny all" make rules that deny this specific traffic with logging enabled (and do this in both directions). That should give you some more info to work on as well.
I'm not sure I can make this happen. Cisco's "deny all" seems to be baked into the device and you have to enable each individual protocol by telling the firewall to inspect each type of connection. It's weird.


Yes, Cisco ASA's have an unwritten implicit DENY ALL statement at the end of every ACL. So exceptions need to be added and be very specific. My understanding is that it also means anything caught by this unwritten statement doesn't get logged, you need to create an explicit deny statement in the ACL before it will log packets blocked by that specific explicit deny parameter. http://blog.ine.com/2010/01/02/ccna-the ... -deny-all/

How did the second ASA go, guessing it exhibits the same problem?


I've worked with Cisco ASAs before. This is correct. You can put in any sort of rule before the deny-all though, including allowing all traffic between two devices. Just leave out any port-related commands.

Anyhow, this might be a problem with the script. Is it possible the script is creating a ridiculous burst of traffic or multiple connections?
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Wed Aug 16, 2017 7:53 pm

Drachasor wrote:
Anyhow, this might be a problem with the script. Is it possible the script is creating a ridiculous burst of traffic or multiple connections?
Ridiculous burst of traffic may be the issue. We're streaming the console for multiple install scripts, a couple maven projects, that sort of thing. It's not quite streaming video, but it is several minutes of lots of text.

Sorry, I haven't been playing with this ASA the last week. Family emergency and a new release to get out the door.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Wed Oct 04, 2017 12:12 pm

Just FYI if anyone is still curious. I'm back on this problem again, starting today. I'll let you know what I find.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
just brew it!
Gold subscriber
Administrator
Posts: 49671
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Cisco ASA keeps killing my SSH connections

Wed Oct 04, 2017 1:09 pm

Took that long to clear off your interrupt stack, eh? When I saw the topic reply notification my first thought was, "Ahh, he finally figured it out, let's see what the resolution was." :lol:

I can relate. I was just in a half-hour meeting this morning, the purpose of which was for all of us to basically go "So what's the status of this issue we all were working on 2 months ago?" The answer: "Same as it was 2 months ago, when all 4 of us got dragged off of this project to deal with other, higher-priority stuff."
Nostalgia isn't what it used to be.
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Thu Oct 05, 2017 3:23 pm

Customer has provided me with a configuration that works on a Cisco ASA and I have the one that does not. The key difference seems to be routed vs transparent mode. I'm planning on verifying that I still have the problem after I rejiggered the network a bit, then configuring in transparent mode and seeing if that solves the problem.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
TheRazorsEdge
Gerbil First Class
Posts: 146
Joined: Tue Apr 03, 2007 1:10 pm

Re: Cisco ASA keeps killing my SSH connections

Thu Oct 05, 2017 7:18 pm

Darkmage wrote:
Customer has provided me with a configuration that works on a Cisco ASA and I have the one that does not. The key difference seems to be routed vs transparent mode. I'm planning on verifying that I still have the problem after I rejiggered the network a bit, then configuring in transparent mode and seeing if that solves the problem.


If that doesn't pan out, you may want to check the Protection Tools or IDS settings.

The TCPKeepAlive messages are not sent over the SSH tunnel, so they can be dropped/filtered by other modules on the ASA. They are basically empty packets vulnerable to spoofing---no timestamps, identification, or crypto.

Alternatively, try using the ServerAliveInterval option when starting the SSH session, as its keep-alive messages are sent out encrypted over the SSH tunnel---and therefore will get through the firewall as long as the connection was permitted.
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Tue Oct 17, 2017 2:13 pm

Update: Transparent mode is not the issue. I (finally) got it working in transparent mode and it still kills the connection after five minutes. I'm now comparing the working configuration to the non-working configuration line-by-line in order to see if I can find the magic line of configuration that translates to "work, dammit".
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Wed Nov 29, 2017 5:08 pm

Ugh. Got pulled off this again due to a new version release of the engine. Back at it.

Comparing the broken vs the working ASA configurations, I notice something... suspicious, I guess. The working configuration has the following:
class-map CONNS
 match any
...
 policy-map CONNS
 class CONNS
  set connection timeout embryonic 0:00:10 half-closed 0:05:00 idle 0:00:10 reset
...
service-policy CONNS interface [named interface]

My reading of the Cisco ASA configuration documentation makes me think that this is a two-parter. The first couple lines defines a name for certain types of connections (in this case, anything). The three lines in the middle defines a list of enforced policies only for the named connection. In this case, it's defining half-open timeouts at 10 seconds, half-closed timeouts at 5 minutes and ten seconds of idle. No idea what "reset" indicates. And the last line is directing the ASA to apply the policy map to the physical interface that connects to our equipment. Is that close to accurate?

I base these conclusions upon this part of the Cisco documentation. I'm really only making informed guesses here. Somewhere around here there must be a "ASA configuration file for Dummies" document.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus
 
Vhalidictes
Gold subscriber
Gerbil Jedi
Posts: 1648
Joined: Fri Jan 07, 2005 2:32 pm
Location: Paragon City, RI

Re: Cisco ASA keeps killing my SSH connections

Wed Nov 29, 2017 5:39 pm

Darkmage, I may be shooting in the dark here, but IIRC that configuration is optional.

You should be able to remove it. Alternatively you can set the half-open timer to insanely long, but keep in mind that ASAs have a hard session limit which could theoretically be hit.
 
Darkmage
Gold subscriber
Lord High Gerbil
Topic Author
Posts: 8001
Joined: Sat Mar 13, 2004 9:44 am
Location: Hell, Virginia

Re: Cisco ASA keeps killing my SSH connections

Fri Dec 01, 2017 9:41 am

Y'all aren't going to believe this. I was talking with the sysadmin at the customer site (this is the dude who gave me a copy of the working configuration). He pulled up the network diagram and suddenly remembered that they had bypassed the ASA firewall with an additional Ethernet cable. So there's actually no guarantee that the "working" configuration actually works.
If there is one thing a remote-controlled, silent and unseeable surveillance/killing machine needs, it’s more whimsy. -- Marcus

Who is online

Users browsing this forum: No registered users and 2 guests