FAQ-SSH Login to the Firewall Failed Due to Inconsistent Forward and Return Paths in Hot Standby Deployment

Publication Date:  2015-07-02 Views:  319 Downloads:  0
Issue Description
Network Topology:



Symptom:

The USG5500 was hardened and the login method was changed from Telnet to SSH. After the change, the PC was unable to access the physical IP address of the outside interface of firewall FW2.
Handling Process
1. The field personnel logged in to the backup firewall (FW2) through the console port and examined the SSH configuration, which was correct. However, quick session backup was not enabled.

The important SSH configuration information was as follows:


stelnet  server enable
ssh user sshuser authentication-type password
ssh user sshuser service-type stelnet
#
aaa
  local-user sshuser password cipher S<$]-`ZJ
  local-user sshuser service-type ssh
  local-user sshuser level 3

#
user-interface vty 0 4
  acl 2000 inbound                        
  authentication-mode aaa
  idle-timeout 20 0
  protocol inbound ssh
#


2. For SSH packets from the PC to FW2, the gateway of the PC was the virtual IP address of the inside interface (G0/0/2) of the FW2 (the standby firewall). Due to hot standby configuration, the TCP packets from the PC were sent to FW1 (the active firewall). Because the destination address was the physical address (192.168.22.122) of the outside interface (G0/0/1). Therefore, FW1 forwarded the packets to FW2. The return packets from the standby firewall (FW2) were sent to the PC out of the inside interface (G0/0/2) of FW2. Therefore, the forward (SYN or ACK) and return (SYN-ACK) packets between the PC and FW2 took different paths.

      PC -> G0/0/2 FW1(Master) G0/0/1 -> G0/0/1 FW2(Slave)
      FW2(Slave) G0/0/2 ---> PC


3. The sessions on the active and standby and the traffic statistics on the active firewall were displayed. The result indicated that the standby firewall had statistics in both directions, but the active firewall had statistics only in one direction and the active firewall discarded packets due to stateful inspection. Therefore, it is concluded that the forward and return paths are inconsistent.

Sessions on the standby firewall:

HRP_S<USG5500-2> display firewall session table verbose source inside 192.168.22.151
22:03:07  2013/05/29
Current total sessions : 5
  tcp  VPN: public -> public
  Zone: trust -> local  TTL: 00:30:00  Left: 00:30:00 
  Interface: I0  Nexthop: 127.0.0.1  MAC: 00-00-00-00-00-00
  <-- packets:713 bytes:29023   --> packets:1030 bytes:62551
  192.168.22.122:23<--192.168.22.151:4277--------Telnet protocol 
 
  tcp  VPN: public -> public
  Zone: trust -> local  TTL: 00:00:05  Left:  timeout 
  Interface: I0  Nexthop: 127.0.0.1  MAC: 00-00-00-00-00-00
  <-- packets:1 bytes:48   --> packets:1 bytes:44 //The standby firewall have statistics in both directions
  192.168.22.122:22<--192.168.22.151:4339-------ssh protocol

 
Sessions on the active firewall:

HRP_M<USG5500-1> display firewall session table verbose source inside 192.168.22.151
22:06:48  2013/05/29
Current total sessions : 5
  tcp  VPN: public -> public
  Zone: trust -> local  TTL: 00:30:00  Left: 00:29:55 
  Interface: G0/0/1  Nexthop: 192.168.22.122  MAC: 00-22-a1-06-b3-cb
  <-- packets:734 bytes:29864   --> packets:0 bytes:0
  192.168.22.122:23<--192.168.22.151:4277-------Telnet protocol
 
  tcp  VPN: public -> public
  Zone: trust -> local  TTL: 00:00:05  Left:  timeout 
  Interface: G0/0/1  Nexthop: 192.168.22.122  MAC: 00-22-a1-06-b3-cb
  <-- packets:1 bytes:48   --> packets:0 bytes:0 //The active firewall has statistics in only one direction
  192.168.22.122:22<--192.168.22.151:4354----ssh protocol


Traffic statistics on the active firewall:

HRP_M[USG5500-1-diagnose] display firewall statistic acl 
22:50:39  2013/05/29

Current Show sessions count: 5
   
Protocol(TCP) SourceIp(192.168.22.151) DestinationIp(192.168.22.122)   
SourcePort(4446) DestinationPort(22) VpnIndex(public)   
           Receive           Forward           Discard   
Obverse : 2          pkt(s) 1          pkt(s) 2          pkt(s)   
Reverse : 0          pkt(s) 0          pkt(s) 0          pkt(s)
   
Discard detail information:
  DP_FW_Send                   :exit 13:     1
  DP_FW_FastSend               :exit 0:     1  //Packet loss caused by stateful inspection


4. After quick session backup was enabled using the hrp mirror session enable command, the problem was resolved.

5. However, one question remains unanswered: for the same forwarding process, why did Telnet succeed but SSH fail?

6. To answer the question, a test environment was set up in a lab to analyze the packet forwarding process.

The SYN packet is sent from the PC to the active firewall (FW1), and a SYN session is established. The session expiration time is 5s and the session state on FW1 is SYN. Then, the SYN packet is forwarded to the standby firewall (FW2) and a SYN session is set up on the FW2. Then, FW2 sends a SYN-ACK packet to the PC out of interface G0/0/2 and the session state on the FW2 changes to SYN+ACK.

If quick session backup was disabled, the sessions are synchronized using the aging thread. The SYN+ACK session state covers the SYN session state, but the SYN session state does not cover the SYN+ACK session state.

After receiving the SYN+ACK packet from FW2, the PC returns an ACK packet, which is first forwarded to the active firewall (FW1). Because stateful inspection is enabled and quick session backup is disabled, the session state on FW2 has not been synchronized to FW1 by the aging thread when the ACK packet arrives on FW1. The session state on FW1 remains SYN and discards the ACK packet.

In Telnet login, the PC sends a Telnet data packet after sending an ACK packet. The Telnet data packet is equivalent to the ACK packet and can be retransmitted. If the SYN+ACK session state on FW2 is not synchronized to FW1, the retransmitted Telnet data packets are discarded. However, once the SYN+ACK session state on FW2 is synchronized to FW1 by the aging thread, the retransmitted Telnet data packet passes the stateful inspection and the three-way handshake on FW1 is completed. Subsequent Telnet packets can be transmitted.




In SSH login, the PC does not send any data packet after sending the ACK packet, and the ACK packet is not retransmitted. When the ACK packet arrives on FW1, the packet is discarded due to stateful inspection. The SYN session on FW1 will expire in 5s. As a result, SSH login fails. 

Root Cause
1. SSH login to the outside interface of the standby firewall (FW2) failed because the forward and return paths were inconsistent and the ACK packet was not retransmitted by the SSH client. As a result, the three-way handshake was not completed and the SYN session expired.

2. Telnet login succeeded when the forward and return paths were inconsistent because the Telnet client sent a Telnet data (Push+ACK) after sending the ACK packet and the Telnet data packet was retransmitted. When the SYN+ACK session state on the standby firewall (FW2) was synchronized to the active firewall (FW1) by the aging thread, the retransmitted Telnet data packet sent to FWs completed the three-way handshake and the TCP connection was established.

3. In the test, Telnet login does not always succeed and SSH login does not always fail. It depends on the implementation of the client. The result varies with the client, such as SecureCRT, XShell, and PuTTY. Therefore, the problem must be analyzed based on the actual environment.
Solution
1. The best solution is not to manage the standby firewall via the active firewall. If you need to manage the standby firewall from a public network, log in to the outside interface of the firewall. If you need to manage the standby firewall from a private network, log in to the inside interface of the firewall.

2. If you must manage the standby firewall via the active firewall, enable quick session backup.


[USG] hrp mirror session enable

END