Personal computing discussed
Moderators: renee, Steel, notfred
Darkmage wrote:To answer your question, we're connecting through the ASA not directly to it.
It's driving me up the wall.
Darkmage wrote:We take the stdout from the install process and log it on the engine through the SSH connection.
Darkmage wrote:Well, some odd developments. It doesn't appear to be a timeout issue, despite it happening regularly at 5 minutes when our script is running.
I turned off the keepalives on the VM and manually opened an SSH session from the engine into it. I just sat there for ten minutes and the connection was just fine. So it's apparently not time-limited.
Perhaps it's size-limited? After a certain number of kilobytes it just dies? That seems... weird.
The guys who own the firewall tell me the firewall configuration is basically the stock one, with some high-availability stuff added. I dunno if that matters.
Vhalidictes wrote:Update: I'm not sure if it's size-related. I took the log file that our process is writing to, SCPd it up to the VM and then opened an SSH session manually. I then ran cat on the full file (which contained about three full logs from failed instances) five times in succession. The manual SSH connection stays up, no matter what I throw at it. So it doesn't appear to be size, per se. Nor time, as manual SSH connections can just sit there for well above the observed time limit.Darkmage, I work with ASAs as a day-job.
The red flag to me was the size-limited question. If that's what you're seeing, I have a few questions:
Vhalidictes wrote:No. We haven't set up our VPN stuff yet by this point.1) Is this SSH connection going through a VPN?
Vhalidictes wrote:9.62) What version of code is this ASA on? There's... a lot of what we call "caveats" that might affect SSH traffic, mostly on 8.x.x versions.
Vhalidictes wrote:Very well. I'll see if I can sit on the ASA as the process runs. Do you have an idiot's guide to displaying traffic logs on an ASA? I'm very new to these things.As some other people have said, being able to check the ASA traffic logs will probably help pinpoint what's going on.
Vhalidictes wrote:This is where we get into the politics of the situation. And like always, when politics are involved it's never a good thing. The short answer is no.EDIT: In theory couldn't the firewall owner open up a TAC case?
Darkmage wrote:More updates: It appears my earlier claims of normal SSH connections being fine were premature. I tried to duplicate our installation process by killing the engine as soon as it connected to the VM and then SSHing into the VM directly. Once on the VM, I ran the first of our scripts to see if there was anything weird.
The script ran and completed without errors. After I pulled up the folder list to select the next script, the connection died on me. So what we have now is that the stream from our script goes just fine and I still have a connection that I can use to get responses from the VM. But something in that process triggers the disconnect... eventually.
Weird.
Drachasor wrote:Depending on the cloud provider, eight minutes or so. There are actually about six scripts involved, but it's not making it past the 2nd one at best.How long does this script take to run?
Drachasor wrote:If the installation script is running, then yes. If I'm just SSHing into some random box, then no.If you log in manually and hit enter every 10 seconds or so for 10 minutes, does it still time out?
Drachasor wrote:On our engine, behind the firewall. This is the box that spins up the VM in the cloud provider, pushes the scripts to the VM, runs the scripts and logs the output of the script locally.And to be 100% clear, on what device do you setup the ServerAlive settings?
Drachasor wrote:That's a good idea. I can't leave that in production, but it may be informative.Try adding a rule to allow traffic initiated in the other direction from port 22 to any port -- if traffic is allowed to be initiated either way, then that should at least give you some more information.
Drachasor wrote:I'm not sure I can make this happen. Cisco's "deny all" seems to be baked into the device and you have to enable each individual protocol by telling the firewall to inspect each type of connection. It's weird.I'd add log statements to the end of related rules and before the "deny all" make rules that deny this specific traffic with logging enabled (and do this in both directions). That should give you some more info to work on as well.
Drachasor wrote:The GUI requires a service contract with Cisco. Well played, Cisco. Well played indeed.For packet captures the GUI has a wizard which makes things easier: https://www.cisco.com/c/en/us/support/d ... sa-00.html
Even a Wireshark or similar capture on either end would be a lot better than nothing. You really need to try to capture all the traffic going between these two devices (on all ports).
Drachasor wrote:Yeah, probably.Can you provide the sanitized rules you are using for this traffic?
Drachasor wrote:Cisco timeout settings:How are you implementing the timeout changes on the FW?
timeout conn 1:00:00 half-closed 0:30:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
Darkmage wrote:Drachasor wrote:I'm not sure I can make this happen. Cisco's "deny all" seems to be baked into the device and you have to enable each individual protocol by telling the firewall to inspect each type of connection. It's weird.I'd add log statements to the end of related rules and before the "deny all" make rules that deny this specific traffic with logging enabled (and do this in both directions). That should give you some more info to work on as well.
Kougar wrote:Darkmage wrote:Drachasor wrote:I'm not sure I can make this happen. Cisco's "deny all" seems to be baked into the device and you have to enable each individual protocol by telling the firewall to inspect each type of connection. It's weird.I'd add log statements to the end of related rules and before the "deny all" make rules that deny this specific traffic with logging enabled (and do this in both directions). That should give you some more info to work on as well.
Yes, Cisco ASA's have an unwritten implicit DENY ALL statement at the end of every ACL. So exceptions need to be added and be very specific. My understanding is that it also means anything caught by this unwritten statement doesn't get logged, you need to create an explicit deny statement in the ACL before it will log packets blocked by that specific explicit deny parameter. http://blog.ine.com/2010/01/02/ccna-the ... -deny-all/
How did the second ASA go, guessing it exhibits the same problem?
Drachasor wrote:Ridiculous burst of traffic may be the issue. We're streaming the console for multiple install scripts, a couple maven projects, that sort of thing. It's not quite streaming video, but it is several minutes of lots of text.Anyhow, this might be a problem with the script. Is it possible the script is creating a ridiculous burst of traffic or multiple connections?
Darkmage wrote:Customer has provided me with a configuration that works on a Cisco ASA and I have the one that does not. The key difference seems to be routed vs transparent mode. I'm planning on verifying that I still have the problem after I rejiggered the network a bit, then configuring in transparent mode and seeing if that solves the problem.
class-map CONNS
match any
...
policy-map CONNS
class CONNS
set connection timeout embryonic 0:00:10 half-closed 0:05:00 idle 0:00:10 reset
...
service-policy CONNS interface [named interface]