Scenario-based Camera Fault Analysis

Scenario-based camera fault analysis aggregates and analyzes offline camera faults by site based on intelligent algorithms, helping O&M personnel quickly detect and locate faults.

Scenario-based camera fault analysis is applicable to the following faults:

• Currently offline: Performs centralized O&M on currently offline cameras by site.
• Frequently offline: Intelligently identifies frequently offline cameras, such as intermittent disconnections. The offline frequency can be customized.
• Offline for a long time: Intelligently identifies zombie cameras and cameras with long faulty time. The offline detection duration can be customized.
• Simultaneously offline: Intelligently identifies cameras that go offline at the same time on a site due to network faults.
• Periodically offline: Intelligently identifies cameras that are powered off periodically.

Figure 2-64 Scenario-based Camera Fault Analysis

You can view the issue analysis result and affected cameras in the issue details offered by scenario-based camera fault analysis. The currently offline scenario as an example. Figure 2-65 shows the issue details page.

Figure 2-65 Issue details of currently offline cameras

Camera Fault Demarcation and Locating

eSight camera fault demarcation and locating diagnoses camera faults, such as offline cameras, camera quality problems, and record loss. This function has the following highlights:

• Automatically restores E2E camera links based on the LLDP, MAC forwarding table, and ARP table.
• Automatically demarcates and locates faults based on the device status, port status, device alarms, and KPIs on links.
• Displays device fault causes in a visualized manner and provides rectification suggestions.

eSight supports the following diagnosis types:

• Offline camera diagnosis: Diagnoses the cause why a camera is offline.
• Video quality diagnosis: Diagnoses the causes of artifacts and frame freezing.
• Live video exception diagnosis: Diagnoses the cause of a live video exception. This type is applicable only to industry video scenarios.
• Record loss diagnosis: Diagnoses the cause of record loss in a specified period. This type is applicable only to industry video scenarios.

Figure 2-66 Camera fault demarcation and locating

Feature Dependencies

This service depends on the video device management service. Before using this feature, install the video device management service.

Application Limitations

• To ensure the normal running of the intelligent video analysis service, the CPU and memory configuration requirements must be met. The CPU and memory required for the intelligent video analysis service vary depending on the number of managed cameras. When 500,000 cameras are managed, about 20% of the CPU and 6 GB memory are required.
• The number of offline cameras, number of frequently offline cameras, and number of cameras offline for a long period must not exceed the numbers specified in Table 2-79. If one or more offline rate indicators exceed the thresholds, the performance of the intelligent analysis service will be severely affected, and the camera analysis result may be inaccurate.
  Table 2-79 Maximum number of offline cameras supported by intelligent analysis

  Number of Equivalent NEs	Maximum Number of Managed Cameras	Number of Currently Offline Cameras (30%)	Number of Frequently Offline Cameras (1%)	Number of Cameras Offline for a Long Period (1%)	Number of Simultaneously Offline Cameras (1%)	Number of Periodically Offline Cameras (1%)
  0–5000	125,000	37,500	1250	1250	1250	1250
  5001–20,000	500,000	150,000	5000	5000	5000	5000

License Support

A license is required for this feature. Table 2-80 describes the license item.

Compatible Product Versions

For details, see the eSight Video Surveillance Management Device List.

Procedure

1. Choose Maintenance Center > IVS > Diagnosis and Analysis.
2. Click in the upper right corner of the page. The fault analysis setting page is displayed.

3. In the navigation pane, choose Single-Fault Analysis Settings.
4. Click the Algorithm Settings tab page.
5. Select camera diagnosis algorithms.
   • Security: The default camera fault demarcation and locating process is used. Select this scenario in common scenarios.
   • Industry video: The customized camera fault demarcation and locating process is used to quickly locate faults. Select this scenario in industry video scenarios. Enable or disable algorithms as required.

6. (Optional) If Run the ping command to check the camera network reachability. is enabled, click to set the camera network segment.
   1. Click Add, set the IP address and the number of mask bits, and click Confirm.
      Table 2-81 Parameters for adding a network segment

      Parameter	Description	Settings
      IP	Reference IP address of a camera network segment. IP and Mask Bit together determine a network segment.	Example: 10.10.10.10
      Mask Bit	The subnet mask is used to identify whether two IP addresses belong to the same subnet. Its value is a 32-bit binary value. The bits on the left are network bits, represented by the binary digit 1. The bits on the right are host bits, represented by the binary digit 0. Mask Bit indicates the number of network bits. Common masks are as follows: 8-bit mask 11111111.00000000.00000000.00000000 (255.0.0.0) 16-bit mask 11111111.11111111.00000000.00000000 (255.255.0.0) 24-bit mask 11111111.11111111.11111111.00000000 (255.255.255.0) 30-bit mask 11111111.11111111.11111111.11111100 (255.255.255.252)	Value: an integer ranging from 8 to 30

   2. If you need to add multiple camera network segments, repeat 6.a.

7. (Optional) If Check whether fault alarms are generated for the private line to which the camera belongs. is enabled, click to set the sharing platform interface HTTPS parameters.
   1. If HTTPS certificate authentication is enabled on eSight, you have obtained the CA certificate file of the connected device and imported it to the SouthBoundNodeService Certificates list in System Management > Certificate Management > Service Certificate Management on eSight. For details, see Certificate Management.
   2. Set the sharing platform interface HTTPS parameters.
      Table 2-82 Sharing platform interface HTTPS parameters

      Parameter	Description	Settings
      IP Address	Service IP address of the sharing platform	Set this parameter to the IP address of the sharing platform.
      Port	Port number of the sharing platform	Set this parameter to the port number of the sharing platform.
      User Name	User name used by eSight to invoke the sharing platform interface	Set this parameter to the user name provided by the sharing platform.
      Password	Password used by eSight to invoke the sharing platform interface	Set this parameter to the password provided by the sharing platform.

   3. Click Connection test. After the connection test is successful, click Confirm.

Prerequisites

• Network devices have been added to eSight.
• You are authorized to manage network devices and set egress devices.

Scenarios

Procedure

1. Choose Maintenance Center > IVS > Diagnosis and Analysis.
2. Click in the upper right corner of the page. The fault analysis setting page is displayed.

3. In the navigation pane, choose Single-Fault Analysis Settings.
4. Click the Egress Device Settings tab page.
5. Click Add Device.
6. In the Add Devices dialog box, select a device type from By Device Type.

7. In the Available Resource list, select the devices to be added.
8. Click Confirm.

   To add multiple types of devices at the same time, repeat steps 6 through 8.

9. Click OK.
   • To delete an added egress device, click in the Operation column. To delete egress devices in batches, click Delete Device in the upper part of the page.
   • If an added egress device is deleted from the network device list, the corresponding device is automatically deleted from the egress device list.

Procedure

1. Choose Maintenance Center > IVS > Diagnosis and Analysis.
2. Click in the upper right corner of the page. The fault analysis setting page is displayed.

3. In the navigation pane, choose Single-Fault Analysis Settings.
4. On the Parameter Settings tab page, set network device threshold check rules and click Confirm.

Procedure

1. Choose Maintenance Center > IVS > Diagnosis and Analysis.
2. Click in the upper right corner of the page. The fault analysis setting page is displayed.

3. In the navigation pane, choose Automatic Alarm Analysis Settings.
4. Turn on the feature switch and set Filter criteria and Suppression policy.

   Frequent triggering of automatic alarm analysis tasks affects the performance of the intelligent analysis service. You are advised to set Filter criteria and Suppression policy properly.

5. Click Confirm.

Restoring Layer-2 Links Between Cameras and Gateways (LSW/AR Access Scenario)

Table 2-87 describes the rules for restoring layer-2 links between cameras and gateways in the LSW/AR access scenario.

"Not managed" indicates that the device model is supported by eSight but is not managed by eSight. "Not supported" indicates that the device model is not supported by eSight.

Restoring Layer-2 Links Between Cameras and Gateways (PON Access Scenario)

Table 2-88 describes the rules for restoring layer-2 links between cameras and gateways in the PON access scenario.

Restoring Layer-2 Links from IVSs to Gateways

Table 2-89 describes the rules for restoring layer-2 links from IVSs to gateways.

Restoring Links from Camera Gateways to IVS Gateways

Table 2-90 describes the rules for restoring links from camera gateways to IVS gateways.

Prerequisites

• IVSs, cameras, network devices, and PON devices have been connected to eSight.
• Automatic terminal discovery is enabled on eSight. If this function is disabled, fault diagnosis cannot be performed.
• For network devices:
  • LLDP has been enabled on network devices. Network devices for which LLDP is disabled cannot be displayed in the diagnosis topology, and may lead to inaccurate display of other network devices in the topology.
  • The read and write community names have been configured in SNMP for the gateway devices on the cameras and the IVS for layer 3 link restoration.
  • Layer 3 gateway devices have been divided into three layers of interfaces, and IP addresses have been configured for these interfaces or these interfaces have been bound to VLANIF interfaces for layer 3 link restoration.
• For PON devices:
  • The MAC address synchronization function in /opt/oss/envs/Product-PONMgmtService/<Timestamp>/etc/access.poll/macPoll.xml has been enabled so that the OLT can learn the MAC address of the camera.
  • PON device information has been automatically or manually synchronized to eSight.
• For the CloudVCN, you need to set an external service IP address. For details, see 3 in "Video Device Management."

Diagnosis Scope

Diagnosis between cameras, network devices, PON devices, and IVSs is supported.

Restrictions

• NAT cannot be performed between cameras and the IVS.
• A camera must have been online and periodically restored or manually restored in the topology while online. Otherwise, offline faults cannot be diagnosed.
• Multiple intermediate links may be displayed together. This depends on the aging time and restoration time of the MAC forwarding table or ARP table. In active/standby scenarios, after the active/standby switchover, the ARP or MAC forwarding table on the active device does not age. In this case, the topology displays both the active and standby devices.
• Topology restoration for non-Huawei network devices is not supported.
• Microwave device diagnosis is not supported.
• If the access device of a camera or the IVS is not a Huawei device or cannot be managed by eSight, the device cannot be identified during E2E diagnosis.
  • If the upper-layer device is a layer 2 forwarding device, the upper-layer device is displayed in the topology restoration result as the access device.
  • If the upper-layer device is a layer 3 forwarding device, topology restoration is not supported.
• A maximum of 20 cameras can be diagnosed at the same time. KPI diagnosis supports only five pages at the same time. If quality diagnosis is performed for more than five pages, KPI data and trend charts are not displayed.
• For all-in-one servers, the VCN/CloudVCN requires that the IPC and VCN gateway can be managed. Otherwise, the E2E diagnosis function is unavailable.
• After offline diagnosis or quality diagnosis starts, the performance data and trend chart display data within only one hour. After one hour, the collection task for collecting device performance data is closed and no longer displayed. Offline diagnosis or quality diagnosis can be performed again to collect and display performance data.
• The topology of a single camera can have a maximum of 20 links. If there are more than 20 links, topology restoration fails.
• The camera diagnosis results are filtered based on user permissions. Users can only view the diagnosis results of authorized devices.
• Only S series, CE series, and NE series network devices support layer 3 link restoration. AR series network devices do not support this function.

Procedure

1. Access the fault diagnosis page.
   • Choose Maintenance Center > IVS > Device Resource from the main menu. In the navigation pane, choose SDC. In the camera list, click in the Operation column corresponding to a camera.
   • Choose Maintenance Center > IVS > Diagnosis and Analysis. In the Issue List, click the issue name. The issue details page is displayed. In the Affected Cameras area, click next to a camera.
   • Choose Maintenance Center > IVS > SDC Quality Map from the main menu. Search for a camera and click in the camera details on the left.

2. Offline Camera Diagnosis or video quality diagnosis is automatically performed based on the camera status.
3. View the diagnosis result on the fault diagnosis page.
   Table 2-91 Description of the fault diagnosis page

   Area	Description
   Basic Information	Displays camera names, IP addresses, regions, locations, and other information.
   Network topology	Diagnosis type Offline camera diagnosis: Diagnoses the cause why a camera is offline. Video quality diagnosis: Diagnoses the causes of artifacts and frame freezing. Topology In the topology, a faulty device is displayed in red, a device that is suspected to be faulty is displayed in yellow, and a normal device is displayed in light blue. Right after the topology is restored, all devices are light blue, which means no faulty devices by default. KPI display Sets KPIs to be monitored during topology restoration. The following KPIs are supported: optical power, bandwidth usage, and packet loss rate. The collection period and update period are both 10 seconds. Legend Move the cursor to to display the legend describing KPI data.
   Video quality view	Displays camera quality statuses.
   Diagnosis Overview	In the Diagnosis Overview area, view the diagnosis result. If a device in the camera network topology is red, you can click the device to view the fault summary of the device.
   Alarm detail	Display device fault details.
   Trend	Displays the online duration trend and intactness duration trend of cameras in the last seven days.

   • If the camera information changes, you can click the corresponding diagnosis type to perform diagnosis again.
   • If the fault occurs because the CPU or memory usage of the camera is high, you are advised to restart the camera to rectify the fault. In the topology view, click Restart, view the risk warning, select I understand the risks and want to continue, and click OK.

Related Operation

The intelligent video analysis service can report camera link relationships to the link management service manually or automatically.

• Automatic polling reporting: After the intelligent video analysis service is started, cameras are polled every two days. Link relationships of all cameras are reported.
• Manual reporting: Click in the Operation column of the camera list to report the link relationships of the camera.

Prerequisites

• The IVS platform and cameras have been connected to eSight.

• The Industry video diagnosis algorithm is selected. For details, see Setting Camera Diagnosis Algorithms.

Diagnosis Scope

Diagnosis between cameras, private line networks, IVSs, and client networks is supported.

eSight can demarcate and locate camera faults of industry videos. When a customer reports a camera fault in the fault reporting system, eSight diagnoses the fault and pushes the diagnosis result to the comprehensive fault platform.

Restrictions

• For all-in-one servers, the VCN/CloudVCN requires that the IPC and VCN gateway can be managed. Otherwise, the E2E diagnosis function is unavailable.
• The camera diagnosis results are filtered based on user permissions. Users can only view the diagnosis results of authorized devices.

Procedure

1. Access the fault diagnosis page.
   • Choose Maintenance Center > IVS > Device Resource from the main menu. In the navigation pane, choose SDC. In the camera list, click in the Operation column corresponding to a camera.
   • Choose Maintenance Center > IVS > Diagnosis and Analysis. In the Issue List, click the issue name. The issue details page is displayed. In the Affected Cameras area, click next to a camera.
   • Choose Maintenance Center > IVS > SDC Quality Map from the main menu. Search for a camera and click in the camera details on the left.

2. Offline Camera Diagnosis or video quality diagnosis is automatically performed based on the camera status.
3. View the diagnosis result on the fault diagnosis page.
   Table 2-92 Description of the fault diagnosis page

   Area	Description
   Basic Information	Displays camera names, IP addresses, regions, locations, and other information.
   Network topology	Diagnosis type Offline camera diagnosis: Diagnoses the cause why a camera is offline. Video quality diagnosis: Diagnoses the causes of artifacts and frame freezing. Live video exception diagnosis: Diagnoses the cause of a live video exception. Record loss diagnosis: Diagnoses the cause of record loss in a specified period. Record loss diagnosis is performed only to analyze faults that occurred in the last seven days. You need to click History in the upper right corner of the page to display the diagnosis result. Before the diagnosis, you need to set the diagnosis time period. In the topology, a faulty device is displayed in red, a device that is suspected to be faulty is displayed in yellow, and a normal device is displayed in light blue. Right after the topology is restored, all devices are light blue, which means no faulty devices by default.
   Video quality view	Displays camera quality statuses.
   Diagnosis Overview	Click View Diagnosis Details. The diagnosis details include the diagnosis details and result. If a device in the camera network topology is red, you can click the device to view the fault summary of the device.
   Alarm detail	Display device fault details.
   Trend	Displays the online duration trend and intactness duration trend of cameras in the last seven days.

   • If the camera information changes, you can click the corresponding diagnosis type to perform diagnosis again.
   • If the fault occurs because the CPU or memory usage of the camera is high, you are advised to restart the camera to rectify the fault. In the topology view, click Restart, view the risk warning, select I understand the risks and want to continue, and click OK.

Question

Which devices support fault diagnosis?

Answer

Table 2-93 lists the devices that support fault diagnosis.

Question

How do I configure camera fault demarcation and locating for industry videos?

Procedure

1. Log in to the eSight O&M plane.
2. Enable the southbound HTTPSServer service so that the operations platform can invoke video interfaces using port 32240.
   1. Choose System Management > System Settings > Configuration Item from the main menu.
   2. In the navigation pane, choose HTTP Configuration.
   3. On the HTTPS Protocol tab page, set the IVSOperationPlatformHTTPSServer status to Enable. Wait for at least 1 minute for the operation to take effect.

3. Set parameters for synchronizing camera groups from the operations platform and set the HTTPS service and southbound HTTPS service authentication parameters of the operations platform. For details, see Synchronizing Camera Groups from the Operations Platform.
4. Choose Maintenance Center > IVS > Diagnosis and Analysis.
5. Click in the upper right corner of the page and click the Parameter Settings tab page.
6. Select the Industry video diagnosis algorithm. For details, see Setting Camera Diagnosis Algorithms.
7. Configure interconnection parameters for fault demarcation and locating on the operations platform. For details, see the operations platform documentation.
   • URL: https://IP address used by eSight to provide services for external systems:32240
   • Access path: /rest/southapi/ivs/faultInfo
   • User name for authentication: The value must be the same as that configured for southbound HTTPS service authentication on the operations platform.
   • Password for authentication: The value must be the same as that configured for southbound HTTPS service authentication on the operations platform.

8. Verify the result of camera fault demarcation and locating.
   1. When detecting a camera fault, the customer reports the fault on the operations platform.
   2. Choose Monitoring Center > Alarm > Current Alarm on eSight and check whether the camera fault alarm (308521595) is generated.
      • If yes, go to the next step. The camera fault alarm is reported to the comprehensive fault platform.
      • If no, reconfigure the fault demarcation and locating function.
   3. Go to the fault diagnosis page and view fault demarcation and locating details.
      1. Choose Maintenance Center > IVS > Device Resource from the main menu.
      2. In the navigation pane, choose SDC.
      3. Click next to the camera that reports the fault and click View Diagnosis Details in the Diagnosis Overview area.

Question

How do I configure alarm-triggered fault demarcation and locating for industry videos?

Answer

After this feature is configured, eSight automatically triggers camera offline diagnosis when a camera or DVR reports the following alarms and the feature conditions are met, and reports the diagnosis result through the northbound interface in additional alarm information.

The configuration procedure is as follows:

1. Configure camera fault demarcation and locating for industry videos. For details, see How Do I Configure Camera Fault Demarcation and Locating (Industry Video).
2. Configure whether to report camera alarms over northbound interfaces. For details, see How Do I Set Whether Cameras Report Alarms over Northbound Interfaces (for Industry Videos)?.
3. Configure northbound alarm reporting for key cameras.
   1. Choose Maintenance Center > IVS > Device Resource from the main menu.
   2. In the navigation pane, choose SDC.
   3. Set to Quick Search and set search criteria to filter cameras.
   4. Click in the upper right corner and choose Set Northbound Alarm Reporting.
   5. Select Set Northbound Alarm Reporting and Filter by condition, and then click Apply.

4. (Optional) Set the importance level of key cameras.
   1. Click in the upper right corner and choose Set Importance.
   2. Select the corresponding importance level and Filter by condition, and then click Apply.

5. Enable the automatic alarm analysis feature.
   1. Choose Maintenance Center > IVS > Diagnosis and Analysis.
   2. Click in the upper right corner of the page. The fault analysis setting page is displayed.
   3. In the navigation pane, choose Automatic Alarm Analysis Settings.
   4. Turn on the feature switch and set Filter criteria and Suppression policy.

6. Verify the configuration.
   1. Change the status of a key camera to Offline.
   2. Wait for 1 to 2 minutes and check whether the camera alarms are reported through the northbound interface and whether the alarm additional information contains the camera fault diagnosis result.