No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

USG 6650 Block Asymmetric Traffic from Branch To DC

Publication Date:  2017-09-15 Views:  69 Downloads:  0
Issue Description

Fault Symptom :

Branch Users Cannot Access Services in one of DC when 2 DCs are online/active.
when only one DC active, branch can access the services in both DC normally.

Version Information :

Software version : USG6650 V500R001C30SPC100
Patch version : V500R001SPH006

Network Overview :

Customer has 2 DCs which running Active-Active Gateway Function. these 2 DCs has its own Firewall and Router to advertise Servers Segment to the Branch.
these 2 DCs must able be to serve access from the branch wherever the access arrived at which DCs.

Network Topology :

  

configuration script :

USG6650

ospf 10 router-id 10.15.254.1
 default-route-advertise
 import-route direct
 area 0.0.0.0
  network 10.15.253.8 0.0.0.3
  network 10.15.253.16 0.0.0.3
  network 10.15.254.1 0.0.0.0
#
ip route-static 0.0.0.0 0.0.0.0 x.x.x.x track ip-link  description Internet
ip route-static x.x.x.x 255.255.255.0 Vlanif3683
ip route-static x.x.x.x 255.255.255.255 NULL0
ip route-static x.x.x.x 255.255.255.255 NULL0
ip route-static vpn-instance default 0.0.0.0 0.0.0.0 10.15.254.254 description Management

Operation Scenario :

Firewall USG advertise Network traffic from Core Switch to the Router and vice versa.
router get the branch network information through BGP routing and import to its internal routing.
Firewall continues advertisement of Branch Network information to Internal DC. this scenario applies to both DCs. 

 

Alarm Information

No Alarm

Handling Process

1. Checking the routing table of Internal DC devices, Firewall and Router to the Branch Network whether each DCs traffic to the branch is going to each Uplink Devices or Not

2. Do ping from the Branch to Servers reside in DCs

<GRS_CR_DC_01>ping 10.15.2.50
PING 10.15.2.1: 56  data bytes, press CTRL_C to break   
Request timed out.
  
Request timed out.
 
Request timed out.
   
Request timed out.
  
Request timed out.
 
--- 10.15.2.50 ping statistics ---
   
5 packet(s) transmitted
   
0 packet(s) received
   
100.0% packet loss
   

it found that the Branch cannot ping the server.

3. Do Tracert from Branch to Servers reside in DCs


traceroute to  10.15.2.50(10.15.2.50), max hops: 30 ,packet length: 40,press CTRL _C to break
1 10.4.2.26 8 ms  6 ms  5 ms
2 10.4.4.4 6 ms  6 ms  6 ms
3 172.17.88.81 9 ms  7 ms  7 ms
4 172.17.52.9 120 ms  21 ms  19 ms 
5 172.17.52.10 20 ms  19 ms  21 ms
6 *  *  *
7 *  *  *
8 *  *  *

it found that the branch cannot reach the servers and found that the trace is going to DC1. in fact the server resides in DC2.

4. Checking routing table of Branch Router. all DCs Server Segment already received by the Branch Router

5. Checking the Firewall Security Policy and try to disable the TCP and ICMP detection check

undo firewall session link-state check

6. Test Ping and Traceroute Again

<GRS_CR_DC_01>ping 10.15.2.50 
PING 10.15.2.50: 56  data bytes, press CTRL_C to break
   
Reply from 10.15.2.50: bytes=56 Sequence=1 ttl=248 time=14 ms
   
Reply from 10.15.2.50: bytes=56 Sequence=2 ttl=248 time=16 ms
   
Reply from 10.15.2.50: bytes=56 Sequence=3 ttl=248 time=13 ms
   
Reply from 10.15.2.50: bytes=56 Sequence=4 ttl=248 time=13 ms
   
Reply from 10.15.2.50: bytes=56 Sequence=5 ttl=248 time=13 ms
 
--- 10.15.2.50 ping statistics ---
   
5 packet(s) transmitted
   
5 packet(s) received
   
0.00% packet loss
   
round-trip min/avg/max = 13/13/16 ms

<GRS_CR_DC_01>tracert 10.15.2.50
traceroute to  10.15.2.1(10.15.2.1), max hops: 30 ,packet length: 40,press CTRL _C to break
1 10.4.2.26 8 ms  6 ms  5 ms
2 10.4.4.4 6 ms  6 ms  6 ms
3 172.17.88.81 9 ms  7 ms  7 ms
4 172.17.52.9 120 ms  21 ms  19 ms 
5 172.17.52.10 20 ms  19 ms  21 ms
10.15.253.9 26 ms  21 ms  40 ms
7 10.15.2.1 20ms 21ms 20ms

<GRS_CR_DC_01>

 

 

 



 



 

Root Cause

it found that the root cause is about asymmetric routing traffic happened during branch and DC communication. by default firewall will check the traffic packet and not allow for asymmetric traffic routing.
by disabling the TCP and ICMP detection check. the firewall will not check the traffic path. whether the traffic is symmetric or asymmetric. 

Solution

the solution for this case is by disabling the TCP and ICMP detection check which allow asymmetric routing traffic between Branch and DCs.
disabling TCP and ICMP detection can be done by command :

undo firewall session link-state check

Suggestions

Advice :

when you running VXLAN Active-Active Gateway where having separate Firewalls and Routers in each DC, it's recommended to disable the link-state check detection since by default the detection check is active.
and by default the firewall will not allow asymmetric traffic passed through. or you can put Load Balancer in front of router so load balancer will check the request from the Branch before traffic from the branch pass through the datacenter.

 

Conclusion

when you have 2 different DCs keeping and advertise same segments to the branches, the branch network normally will choose only one path to the DC. it will be a problem when the server or application is not same between DC1 and DC2.
the traffic data may transmit in asymmetric way. when there is only router in front of each DCs, there will be no problem, the router will pass through the asymmectric traffic but in will be a different treatment when there is firewall, by default firewall will discard or not allow traffic that pass through in asymmetric way. so to let firewall pass through the asymmetric traffic, you need to disable the link-state detection check. there is also another way to solve this problem is by using load-balancer in front of the routers so traffic from branch will touch load balancer first and load balancer will determine where the traffic need to be sent. 

END