No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

Troubleshooting Of AP Offline For ACU2 (V200R006C10SPC100)

Publication Date:  2016-04-21 Views:  130 Downloads:  0
Issue Description

APs lost connection with ACU2, and CPU usage become high.

Handling Process
1. Check the AP offline reason.
When the AP lost connect with AC, we check the S7703-1 log and find all the AP go offline the same time with the same reason  “Echo interval timeout”.  “echo interval timeout” means the AP and AC cannot communicate normal. The packet would lost between AP and AC in the network. So we check the switch log as below:
2016-4-17 19:21:33+04:00 S7703-1 %%01WLAN/3/AP_NORMAL_TO_FAULT(l)[11686240]:AP changed from normal to fault. (MAC=[70.d9.xx.xx.dc.00 (hex)], ApID=22, Sysname=FLOOR5_587_ROOM, Reason=Echo interval timeout)
2016-4-17 19:21:33+04:00 S7703-1 %%01WLAN/3/AP_NORMAL_TO_FAULT(l)[11686241]:AP changed from normal to fault. (MAC=[70.d9.xx.xx.09.40 (hex)], ApID=588, Sysname=FLOOR2_217_ROOM, Reason=Echo interval timeout)
2. Check the S7703-2 and ACU2.
Check the S7703-1, we find the MAC address flapping record.
Line 18159: Apr 17 2016 23:19:59 S7703-1 L2IFPPI/4/MFLPVLANALARM:OID 1.3.6.1.4.1.2011.5.25.160.3.7 MAC move detected, VLANID = 124, MacAddress = 0000-XXXX-017c, Original-Port = Eth-Trunk0, Flapping port = Eth-Trunk10. Please check the network accessed to flapping port.
Check the S7703-2, we find the MAC address flapping record too.
Apr 17 2016 23:21:34 S7703-2 L2IFPPI/4/MFLPVLANALARM:OID 1.3.6.1.4.1.2011.5.25.160.3.7 MAC move detected, VLANID = 124, MacAddress = 0000-XXXX-017c, Original-Port = Eth-Trunk10, Flapping port = Eth-Trunk0. Please check the network accessed to flapping port.
Check the vlan 124 and confirm that is capwap source vlan. And the port Eth-Trunk0 connect to ACU2 and Eth-Trunk10  connect to core switch. The MAC move detected frequently would cause the network not stable. It would cause the communication  lost between AP and AC. we check the backup ACU2 logs for why the MAC address learned from the Eth-Trunck10 just as below. And check the AP offline log and confirm the link is unstable just as below:
capwap source ip-address 10.32.X.1
ip pool 124
gateway-list 10.32.X.1
network 10.32.X.0 mask 255.255.254.0
#

interface Vlanif124
ip address 10.32.X.11 255.255.255.0
vrrp vrid 124 virtual-ip 10.32.X.1
admin-vrrp vrid 124
vrrp vrid 124 priority 150
vrrp vrid 124 preempt-mode timer delay 120
        dhcp select global
Note: The time of S7703-1 is 4 hours later than switch and S7703-2.
3. Check the the log of S7703-2 and ACU2.
The S7703-2 change to master due to the reason “protocol timer expired”, that means the backup ACU2 cannot receive VRRP announcement.  If the ACU2 change to master, it would send the packet with virtual MAC  [0000-XXXX-017c] , then the flapping happened in the switch . Unfortunately, the VRRP state change several times and it would cause the network unstable.
For the VRRP announcement cannot received by backup ACU2, we check all the logs and confirm the network is unstable during the problem:
Apr 17 2016 23:18:39+08:00 S7703-2 VRRP/2/VRRPCHANGETOMASTER:OID 1.3.6.1.2.1.68.0.1 The status of VRRP changed to master. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, ChangeReason=protocol timer expired)
Apr 17 2016 23:18:41+08:00 S7703-2 VRRP/2/VRRPMASTERDOWN:OID 1.3.6.1.4.1.2011.5.25.127.2.30.1 The state of VRRP changed from master to other state. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, CurrentState=2, ChangeReason=priority calculation)
2016-4-17 23:18:39+08:00 S7703-2 %%01VRRP/4/STATEWARNINGEXTEND(l)[92196]:Virtual Router state BACKUP changed to MASTER, because of protocol timer expired. (Interface=Vlanif124, VrId=124, InetType=IPv4)
Apr 17 2016 23:19:17+08:00 S7703-2 VRRP/2/VRRPCHANGETOMASTER:OID 1.3.6.1.2.1.68.0.1 The status of VRRP changed to master. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, ChangeReason=protocol timer expired)
Apr 17 2016 23:19:41+08:00 S7703-2 VRRP/2/VRRPMASTERDOWN:OID 1.3.6.1.4.1.2011.5.25.127.2.30.1 The state of VRRP changed from master to other state. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, CurrentState=2, ChangeReason=priority calculation)
Apr 17 2016 23:19:44+08:00 S7703-2 VRRP/2/VRRPCHANGETOMASTER:OID 1.3.6.1.2.1.68.0.1 The status of VRRP changed to master. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, ChangeReason=protocol timer expired)
Apr 17 2016 23:20:03+08:00 S7703-2 VRRP/2/VRRPCHANGETOMASTER:OID 1.3.6.1.2.1.68.0.1 The status of VRRP changed to master. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, ChangeReason=protocol timer expired)
Apr 17 2016 23:20:19+08:00 S7703-2 VRRP/2/VRRPMASTERDOWN:OID 1.3.6.1.4.1.2011.5.25.127.2.30.1 The state of VRRP changed from master to other state. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, CurrentState=2, ChangeReason=priority calculation)
Apr 17 2016 23:20:33+08:00 S7703-2 VRRP/2/VRRPCHANGETOMASTER:OID 1.3.6.1.2.1.68.0.1 The status of VRRP changed to master. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, ChangeReason=protocol timer expired)
Apr 17 2016 23:20:40+08:00 S7703-2 VRRP/2/VRRPMASTERDOWN:OID 1.3.6.1.4.1.2011.5.25.127.2.30.1 The state of VRRP changed from master to other state. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, CurrentState=2, ChangeReason=priority calculation)
Apr 17 2016 23:20:45+08:00 S7703-2 VRRP/2/VRRPCHANGETOMASTER:OID 1.3.6.1.2.1.68.0.1 The status of VRRP changed to master. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, ChangeReason=protocol timer expired)
Apr 17 2016 23:21:03+08:00 S7703-2 VRRP/2/VRRPMASTERDOWN:OID 1.3.6.1.4.1.2011.5.25.127.2.30.1 The state of VRRP changed from master to other state. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, CurrentState=2, ChangeReason=priority calculation)
Apr 17 2016 23:21:09+08:00 S7703-2 VRRP/2/VRRPCHANGETOMASTER:OID 1.3.6.1.2.1.68.0.1 The status of VRRP changed to master. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, ChangeReason=protocol timer expired)
Apr 17 2016 23:21:09+08:00 S7703-2 VRRP/2/VRRPMASTERDOWN:OID 1.3.6.1.4.1.2011.5.25.127.2.30.1 The state of VRRP changed from master to other state. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, CurrentState=2, ChangeReason=priority calculation)
Apr 17 2016 23:21:33+08:00 S7703-2 VRRP/2/VRRPMASTERDOWN:OID 1.3.6.1.4.1.2011.5.25.127.2.30.1 The state of VRRP changed from master to other state. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, CurrentState=2, ChangeReason=priority calculation)
Apr 17 2016 23:25:21+08:00 S7703-2 VRRP/2/VRRPCHANGETOMASTER:OID 1.3.6.1.2.1.68.0.1 The status of VRRP changed to master. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, ChangeReason=protocol timer expired)
Apr 17 2016 23:25:21+08:00 S7703-2 VRRP/2/VRRPMASTERDOWN:OID 1.3.6.1.4.1.2011.5.25.127.2.30.1 The state of VRRP changed from master to other state. (VrrpIfIndex=11, VrId=124, IfIndex=11, IPAddress=10.32.X.12, NodeName=S7703-2, IfName=Vlanif124, CurrentState=2, ChangeReason=priority calculation)

4. Check the network connection
Check all the logs and confirm the network is unstable during the problem happened.
Check the AP log and confirm the link between AP and AC is unstable. 60% packet loss.
%2016-Apr-17 19:20:36.000.1 FLOORM_FINANCEOFFICE 01PING/6/PING_RESULT(D)[475366]:327691 User used ping 10.32.X.1: 5 packet(s) transmitted, 2 packet(s) received, 60.00% packet loss.
5. Confirm the VRRP announcement is not dropped by backup ACU2
<SS7703-2>display cpu-defend statistics wired
-----------------------------------------------------------------------
Packet Type               Pass Packets        Drop Packets
-----------------------------------------------------------------------
8021X                                0                   0
… …
portal                               0                   0
radius                               0                   0
rarp-reply                           0                   0
rarp-request                         0                   0
snmp                           3560985                8635
ssh-client                           0                   0
ssh-server                           6                   0
sshv6-client                         0                   0
sshv6-server                         0                   0
tcp                             168392                   0
telnet-client                       12                   0
telnet-server                        0                   0
telnetv6-server                      0                   0
ttl-expired                      38722              237127
unknown-multicast                25343                   0
unknown-packet                 8221815                  58
vrrp                           5224234                   0
wapi                                 0                   0
-----------------------------------------------------------------------
6. Check the HSB backup tunnel state. The HSB tunnel is unstable.
2016-4-17 19:18:40+04:00 S7703-1 %%01HSB/5/TUNNLE_CHANGE(l)[11686203]:Hot-standby-service 0 status change from CONNECT to DISCONNECT (LocalIP: 10.32.X.11 PeerIP: 10.32.X.12 LocalPort: 10241 PeerPort: 10241).
2016-4-17 19:18:41+04:00 S7703-1 %%01HSB/5/TUNNLE_CHANGE(l)[11686204]:Hot-standby-service 0 status change from DISCONNECT to CONNECT (LocalIP: 10.32.X.11 PeerIP: 10.32.X.12 LocalPort: 10241 PeerPort: 10241).
2016-4-17 19:18:46+04:00 S7703-1 %%01HSB/5/TUNNLE_CHANGE(l)[11686206]:Hot-standby-service 0 status change from CONNECT to DISCONNECT (LocalIP: 10.32.X.11 PeerIP: 10.32.X.12 LocalPort: 10241 PeerPort: 10241).
2016-4-17 19:18:53+04:00 S7703-1 %%01HSB/5/TUNNLE_CHANGE(l)[11686207]:Hot-standby-service 0 status change from DISCONNECT to CONNECT (LocalIP: 10.32.X.11 PeerIP: 10.32.X.12 LocalPort: 10241 PeerPort: 10241).
2016-4-17 19:18:59+04:00 S7703-1 %%01HSB/5/TUNNLE_CHANGE(l)[11686210]:Hot-standby-service 0 status change from DISCONNECT to CONNECT (LocalIP: 10.32.X.11 PeerIP: 10.32.X.12 LocalPort: 10241 PeerPort: 10241).
2016-4-17 19:19:10+04:00 S7703-1 %%01HSB/5/TUNNLE_CHANGE(l)[11686213]:Hot-standby-service 0 status change from CONNECT to DISCONNECT (LocalIP: 10.32.X.11 PeerIP: 10.32.X.12 LocalPort: 10241 PeerPort: 10241).
… …
… …
2016-4-17 22:58:15+04:00 S7703-1 %%01HSB/5/TUNNLE_CHANGE(l)[17890]:Hot-standby-service 0 status change from CONNECT to DISCONNECT (LocalIP: 10.32.X.11 PeerIP: 10.32.X.12 LocalPort: 10241 PeerPort: 10241).
2016-4-17 22:58:16+04:00 S7703-1 %%01HSB/5/TUNNLE_CHANGE(l)[17891]:Hot-standby-service 0 status change from DISCONNECT to CONNECT (LocalIP: 10.32.X.11 PeerIP: 10.32.X.12 LocalPort: 10241 PeerPort: 10241).
2016-4-17 22:58:27+04:00 S7703-1 %%01HSB/5/TUNNLE_CHANGE(l)[17893]:Hot-standby-service 0 status change from CONNECT to DISCONNECT (LocalIP: 10.32.X.11 PeerIP: 10.32.X.12 LocalPort: 10241 PeerPort: 10241).
2016-4-17 22:58:29+04:00 S7703-1 %%01HSB/5/TUNNLE_CHANGE(l)[17895]:Hot-standby-service 0 status change from DISCONNECT to CONNECT (LocalIP: 10.32.X.11 PeerIP: 10.32.X.12 LocalPort: 10241 PeerPort: 10241).
7. Check the MAC address flapping time range.  The last mac flap record is around [Apr 18 2016 04:22:19]
Apr 17 2016 23:20:19 S7703-2 L2IFPPI/4/MAC_FLAPPING_ALARM:OID 1.3.6.1.4.1.2011.5.25.42.2.1.7.12 The MAC address has flap value. (L2IfPort=0, entPhysicalIndex=0, BaseTrapSeverity=4, BaseTrapProbableCause=549, BaseTrapEventType=1, MacAddr=0000-XXXX-017c, VLANID=124, FormerIfDescName=Eth-Trunk10, CurrentIfDescName=Eth-Trunk0, DeviceName=S7703-2)
Apr 17 2016 23:20:57 S7703-2 L2IFPPI/4/MAC_FLAPPING_ALARM:OID 1.3.6.1.4.1.2011.5.25.42.2.1.7.12 The MAC address has flap value. (L2IfPort=0, entPhysicalIndex=0, BaseTrapSeverity=4, BaseTrapProbableCause=549, BaseTrapEventType=1, MacAddr=0000-XXXX-017c, VLANID=124, FormerIfDescName=Eth-Trunk10, CurrentIfDescName=Eth-Trunk0, DeviceName=S7703-2)
Apr 17 2016 23:21:20 S7703-2 L2IFPPI/4/MAC_FLAPPING_ALARM:OID 1.3.6.1.4.1.2011.5.25.42.2.1.7.12 The MAC address has flap value. (L2IfPort=0, entPhysicalIndex=0, BaseTrapSeverity=4, BaseTrapProbableCause=549, BaseTrapEventType=1, MacAddr=0000-XXXX-017c, VLANID=124, FormerIfDescName=Eth-Trunk0, CurrentIfDescName=Eth-Trunk10, DeviceName=S7703-2)
Apr 17 2016 23:27:01 S7703-2 L2IFPPI/4/MAC_FLAPPING_ALARM:OID 1.3.6.1.4.1.2011.5.25.42.2.1.7.12 The MAC address has flap value. (L2IfPort=0, entPhysicalIndex=0, BaseTrapSeverity=4, BaseTrapProbableCause=549, BaseTrapEventType=1, MacAddr=0000-XXXX-017c, VLANID=124, FormerIfDescName=Eth-Trunk10, CurrentIfDescName=Eth-Trunk0, DeviceName=S7703-2)
Apr 17 2016 23:27:15 S7703-2 L2IFPPI/4/MAC_FLAPPING_ALARM:OID 1.3.6.1.4.1.2011.5.25.42.2.1.7.12 The MAC address has flap value. (L2IfPort=0, entPhysicalIndex=0, BaseTrapSeverity=4, BaseTrapProbableCause=549, BaseTrapEventType=1, MacAddr=0000-XXXX-017c, VLANID=124, FormerIfDescName=Eth-Trunk10, CurrentIfDescName=Eth-Trunk0, DeviceName=S7703-2)
Apr 17 2016 23:27:45 S7703-2 L2IFPPI/4/MAC_FLAPPING_ALARM:OID 1.3.6.1.4.1.2011.5.25.42.2.1.7.12 The MAC address has flap value. (L2IfPort=0, entPhysicalIndex=0, BaseTrapSeverity=4, BaseTrapProbableCause=549, BaseTrapEventType=1, MacAddr=0000-XXXX-017c, VLANID=124, FormerIfDescName=Eth-Trunk10, CurrentIfDescName=Eth-Trunk0, DeviceName=S7703-2)
Apr 17 2016 23:32:59 S7703-2 L2IFPPI/4/MAC_FLAPPING_ALARM:OID 1.3.6.1.4.1.2011.5.25.42.2.1.7.12 The MAC address has flap value. (L2IfPort=0, entPhysicalIndex=0, BaseTrapSeverity=4, BaseTrapProbableCause=549, BaseTrapEventType=1, MacAddr=0000-XXXX-017c, VLANID=124, FormerIfDescName=Eth-Trunk10, CurrentIfDescName=Eth-Trunk0, DeviceName=S7703-2)
… …
… …
Apr 18 2016 04:11:54 S7703-2 L2IFPPI/4/MAC_FLAPPING_ALARM:OID 1.3.6.1.4.1.2011.5.25.42.2.1.7.12 The MAC address has flap value. (L2IfPort=0, entPhysicalIndex=0, BaseTrapSeverity=4, BaseTrapProbableCause=549, BaseTrapEventType=1, MacAddr=0000-XXXX-017c, VLANID=124, FormerIfDescName=Eth-Trunk10, CurrentIfDescName=Eth-Trunk0, DeviceName=S7703-2)
Apr 18 2016 04:22:19 S7703-2 L2IFPPI/4/MAC_FLAPPING_ALARM:OID 1.3.6.1.4.1.2011.5.25.42.2.1.7.12 The MAC address has flap value. (L2IfPort=0, entPhysicalIndex=0, BaseTrapSeverity=4, BaseTrapProbableCause=549, BaseTrapEventType=1, MacAddr=0000-XXXX-017c, VLANID=124, FormerIfDescName=Eth-Trunk10, CurrentIfDescName=Eth-Trunk0, DeviceName=S7703-2)
8. Check the AP state and confirm the AP still unstable after the second manual reboot around 21:30.
Apr 17 2016 21:53:44+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.07.80 (hex)], APType=AP2030DN, APName=FLOORB_PARKING_AP4, APID=90)
Apr 17 2016 21:53:49+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.03.a0 (hex)], APType=AP2030DN, APName=FLOORG_JZSMAINENTERANCE_AP1, APID=232)
Apr 17 2016 21:53:50+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.90.e0 (hex)], APType=AP4030DN, APName=FLOOR2_COR(281)_AP48, APID=1105)
Apr 17 2016 21:53:53+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.0d.80 (hex)], APType=AP2030DN, APName=VILLA35_AP3, APID=26)
Apr 17 2016 21:53:54+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.5b.80 (hex)], APType=AP4030DN, APName=VILLA32_AP2, APID=942)
Apr 17 2016 21:53:54+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.e6.60 (hex)], APType=AP2030DN, APName=FLOOR4_449_ROOM, APID=553)
Apr 17 2016 21:53:58+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.95.40 (hex)], APType=AP2030DN, APName=FLOOR5_541_ROOM, APID=105)
Apr 17 2016 21:53:58+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.d9.e0 (hex)], APType=AP2030DN, APName=VILLA24_AP6, APID=272)
Apr 17 2016 21:54:01+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.04.20 (hex)], APType=AP2030DN, APName=VILLA29_AP3, APID=205)
Apr 17 2016 21:54:02+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.f8.60 (hex)], APType=AP4030DN, APName=FLOORG_COR(BKOFF)_AP2, APID=1060)
Apr 17 2016 21:54:31+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.69.a0 (hex)], APType=AP2030DN, APName=FLOOR3_342_ROOM, APID=78)
Apr 17 2016 21:55:47+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.b6.e0 (hex)], APType=AP2030DN, APName=FLOOR1_186_ROOM, APID=627)
Apr 17 2016 22:46:13+04:00 S7703-1 WLAN/4/AP_NORMAL_NOTIFY:OID 1.3.6.1.4.1.2011.6.139.13.1.1.2 The communication between the AP and the AC recovers to normal . (APMAC=[70.d9.XX.XX.0c.a0 (hex)], APType=AP2030DN, APName=FLOOR1_174_ROOM, APID=5)
9. CPU spike to 98%
Check the Log and confirm the task which cause CPU 98% are WMT_SRV(61.5%), IFPD(15.2%), CWP_FWD(4.8%). All task relate to AP online. 
2016-4-17 19:50:45+04:00 S7703-1 %%01MON/4/CPU_USAGE_HIGH(l)[1256]:The CPU is overloaded, and the top three thread CPU occupancy are WMT_SRV(61.5%), IFPD(15.2%), CWP_FWD(4.8%) . (CpuUsage=98%, Threshold=80%)
Apr 17 2016 19:53:37+04:00 S7703-1 ENTITYTRAP/4/CPUUSAGERESUME:OID 1.3.6.1.4.1.2011.5.25.219.2.14.2 CPU utilization resumed from exceeding the pre-alarm threshold.(Index=9, HwEntityPhysicalIndex=9, PhysicalName="SRU Board 0", EntityThresholdType=1, EntityThresholdWarning=80, EntityThresholdCurrent=25, EntityTrapFaultID=144896)
When the CPU spike to 98%, we can confirm that all AP change standy to normal. So the CPU high is cause by the AP state switch this time. AP state switch, there are much of data should be backup and create mount of interface for AP management, so it would cost a lot of cpu source and bring the CPU to high, this is normal state for this situation.
10. Configuration optimization:
a. HSB tunnel keep-alive period is too much sensitive.
hsb-service 0
service-ip-port local-ip 10.32.X.11 peer-ip 10.32.X.12 local-data-port 10241 peer-data-port 10241
service-keep-alive detect retransmit 2 interval 1  //Recommend change to the default 5x3
b. The HSB tunnel for backup date is the same VLAN with AP management VLAN.
Recommend create an new VLAN for backup data only of HSB tunnel.
c. For the cpu high cause by snmp which happened another time, we recommend change the CPU car,
cpu-defend policy mypolicy
packet-type  snmp rate-limit  96 wired
d. Other parameters. preempt-mode timer delay is much short.
interface Vlanif124
ip address 10.32.X.11 255.255.255.0
vrrp vrid 124 virtual-ip 10.32.X.1
admin-vrrp vrid 124
vrrp vrid 124 priority 150
vrrp vrid 124 preempt-mode timer delay 120  // It would cause backup switch to master slower. We recommend change to 1800s due to many vaps need to backup data.
dhcp select global
Root Cause
Accord to the log timestamp, the AP go offline is caused by network unstable. The details just as below:
1. The HSB tunnel disconnects and connect happened first. The HSB tunnel keep alive time is 2x1s. Tunnel disconnect would caused by keep alive lost due to network unstable.
2. The backup ACU2 change to master statues several times due to VRRP announcement lost.
3. When the backup ACU2 change to master, there are two master ACU2 in the network.
4. The two master ACU2 would cause MAC address flapping in the switch.
5. The MAC address flapping happened frequently in the switch would lead to the network unstable and AP would “echo timeout” due to the unstable network.
Suggestions
During the Troubleshooting, using different way to reduce the arrange of  possible root cause. That will be helpful for the work.

END