Before rebooting the machine we observed that the USG had all the led interfaces blinking simultaneously.
After the reboot the services were again unavailable and the device wasn’t operate correctly; two network interfaces lost their configuration and had to be configured manually.
Step 1 - Analyze the alarm records and state of the device
There were no alarm records and the state of the device was normal. It seems the hardware worked normally.
After checking the logs of the firewall, the system logs were normal. The operation logs showed that the customer was modifying the parameters of IPSEC.
Step 2 - Analyze the logs
At 17:06:55, the user admin, whose IP address was 192.168.1xx.y , deleted the rule of ACL 3002. At 17:07:19, the user admin, whose IP address was 192.168.1xx.y, added the rule for ACL 3002. After that, there was no operation until the morning of 1 June.
Checking the configuration before the reboot, we found the ACL 3002 was referenced to the IPSEC policy, which was referenced to the interface 1/0/0. After the operation above (rule permit ip), all the traffic, whose output interface was 1/0/0, would be put into IPSEC tunnel. And then the business was blocked.
Step 3 - Analyze logs about reboot.
The ACL, defining the data flow to be protected by IPSEC tunnel, was modified to match all the traffic. It caused the block of the business.
The ACL 3002 has already been modified as the following:
Rule 5 permit ip source 172.16.x.x 0.0.255.255 destination 10.1.w.w 0.0.0.255
Rule 10 permit ip source 172.20.x.x 0.0.255.255 destination 10.1.w.w 0.0.0.255