No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

HUAWEI CLOUD Stack 6.5.0 Alarm and Event Reference 04

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
SDR Alarm Handling Reference

SDR Alarm Handling Reference

ALM-2000301 SDR File Generation Fails

Description

Service Detail Record (SDR) generates bills at the first quarter of every hour. If the data source cannot be accessed or an exception occurs, bills fail to generate. If no bill is generated, the service starts the retry mechanism. This alarm is generated when the number of retry attempts is equal to the configured threshold.

Attribute

Alarm ID

Alarm Severity

Auto Clear

2000301

Critical

No

Parameters

Parameter

Description

Location Info

Service for which the alarm is generated

Name of the service for which the alarm is generated

Other Information

Service name

Name of the service for which the alarm is generated

MicroService

Name of the component that sends an alarm

RecordID

Task ID

ResourceType

Resource type of generated bills, for example, volume

StartTime

Start time of a period when bills are generated

EndTime

End time of a period when bills are generated

FailCause

Cause for generating an alarm, for example, Query data from ceilometer error

Job ID

ID of the task that fails to generate bills

Impact on the System

The billing system fails to collect bills during the failure period, affecting billing.

Possible Cause

  • Data sources are abnormal. There are two types of data sources: One is provided by ceilometer in the cascading and cascaded FusionSphere OpenStack. The other is non-ceilometer, which is provided by a service.
  • The upload channel is abnormal, that is, the ManageOne SFTP server is abnormal.

Procedure

  1. Log in to ManageOne Maintenance Portal using a browser.

    • URL: https://Address for accessing the homepage of ManageOne Maintenance Portal:31943, for example, https://oc.type.com:31943
    • Default username: admin; default password: Huawei12#$

  2. On the menu bar in the upper part of the page, choose Alarms > Current Alarms.
  3. In the alarm list, locate the alarm to be handled and click on the left of the alarm.
  4. Click the Original Alarms tab page and check whether the alarm cause is Query data from ceilometer error from the Other Information column.

    NOTE:

    You must obtain Other Information from the Original Alarms tab page. If there are multiple alarms, perform the following steps for each alarm.

    • If yes, go to 5 to check whether the PUB-SRV01 node is connected to Ceilometer.
    • If no, go to 14.

  5. Log in to a FusionSphere OpenStack controller node.

    • Region Type I scenario (cascading system):

      Use PuTTY to log in to the controller node in the cascading FusionSphere OpenStack system using the Cascading-ExternalOM-Reverse-Proxy IP address.

      The default account is fsp. The default password is Huawei@CLOUD8.

    • Region Type I scenario (cascaded system):

      Use PuTTY to log in to the controller node in the cascaded FusionSphere OpenStack system using the Cascaded-ExternalOM-Reverse-Proxy IP address.

      The default account is fsp. The default password is Huawei@CLOUD8.

    • Region Type II or Region Type III scenario (non-cascading system):

      Use PuTTY to log in to the controller node in the FusionSphere OpenStack system using the ExternalOM-Reverse-Proxy IP address.

      The default account is fsp. The default password is Huawei@CLOUD8.

  6. Run the following command and enter the password Huawei@CLOUD8! of the root user to switch to the root user:

    su - root

  7. Run the following command to disable user logout upon system timeout:

    TMOUT=0

  8. Import environment variables.

    source set_env

  9. To enable the built-in DC administrator's Keystone V3 authentication, enter 1, press Enter, and enter the password of account OS_USERNAME as instructed.

    Default password: FusionSphere123

  10. Run the following command to obtain the domain name of Ceilometer and record it (for example, metering.az0.dc0.cloudservice.com:443).

    openstack endpoint list --service ceilometer --interface public

    Information similar to the following is displayed.

    Figure 16-4 Obtaining the Ceilometer domain name

  11. Log in to the PUB-SRV01 node using PuTTY.

    The default username is meteradmin. The default password is Huawei@123.

    For details about how to obtain the default account and password, go to HUAWEI CLOUD Stack 6.5.0 Security Management Guide and click Download to obtain HUAWEI CLOUD Stack 6.5.1 Account List.

  12. Run the following command to check whether Ceilometer is normal:

    curl metering.az0.dc0.cloudservice.com:443

    metering.az0.dc0.cloudservice.com:443 indicates the domain name obtained in 10.

    If the command output contains "Empty reply from server", Ceilometer is normal.

    • If yes, contact technical support for assistance.
    • If no, go to 13.

  13. Check the alarm reported by Ceilometer. Clear the alarm by performing steps provided in section "Resource Pool" > "FusionSphere OpenStack Alarm Reference" > "ALM-73203 Component Fault" in HUAWEI CLOUD Stack 6.5.0 Alarm&Event Reference. Check whether the alarm is cleared.

    • If yes, go to 27.
    • If no, go to 14.

  14. Obtain the failed task ID (recordId) from the Other Information column.
  15. Use PuTTY to log in to the PUB-SRV01 and PUB-SRV02 node, respectively. Perform 16 to 22.

    The default username is meteradmin. The default password is Huawei@123.

    For details about how to obtain the default account and password, go to HUAWEI CLOUD Stack 6.5.0 Security Management Guide and click Download to obtain HUAWEI CLOUD Stack 6.5.1 Account List.

  16. Run the following command to disable user logout upon system timeout:

    TMOUT=0

  17. Run the following command to check whether the SDR process is running properly:

    ps -ef |grep meteradmin

    • If yes, go to 20.
    • If no, go to 18.

  18. Run the following command to start the SDR process:

    sh /home/meteradmin/meterticket-*/bin/startup.sh

  19. Run the following command to check whether the SDR process is running properly:

    ps -ef |grep meteradmin

    • If yes, go to 20.
    • If no, contact technical support for assistance.

  20. Run the following command to check whether port 9443 is listened to:

    netstat -lnp | grep 9443

    • If yes, go to 21.
    • If no, contact technical support for assistance.

  21. Run the following command to switch to the /var/log/meterticket-agent directory:

    cd /var/log/meterticket-agent

  22. Run the following command to query the error logs of recordId and restore the environment where bills are generated:

    grep recordId *

    recordId indicates the alarm ID obtained in 14.
    • If the command output contains "com.jcraft.jsch.SftpException: Failure", contact technical support for assistance.
    • If the command output contains "com.jcraft.jsch.SftpException: Permission Deny", perform 23 and 24.
    • If the command output contains "com.jcraft.jsch.JSchException: timeout: socket is not established", contact technical support for assistance.
    • If the command output contains "connection refused", go to 25.
    • If the command output contains other information, go to 33.

  23. On the PUB-SRV01 or PUB-SRV02 node, run the following command to switch to the floating IP address of the SFTP server.

    sftp meteradmin@SFTP node(ManageOne-Service03 or ManageOne-Service04)floating IP address

    The default username is meteradmin. The default password is ManageOne12#$. To obtain the SFTP node floating IP address, search for ManageOne-Tenant-Float-IP on the 2.1 Tool-generated IP Parameters sheet in the exported xxx_export_all_CN.xlsm file.

    For details about how to obtain the default account and password, go to HUAWEI CLOUD Stack 6.5.0 Security Management Guide and click Download to obtain HUAWEI CLOUD Stack 6.5.1 Account List.

  24. Run the following commands to check the /opt/meterfiles/uploads directory and its subdirectories on the SFTP server:

    cd /opt/meterfiles

    ll

    Check whether the meteradmin user can perform operations on the directories.
    • If yes, contact technical support for assistance.
    • If no, run the following command to change the directory permission.

      chown -R meteradmin:users uploads

      After changing the username to meteradmin, go to 27.

  25. Contact technical support to check whether the SFTP function of ManageOne is normal.
  26. After the environment is restored, run the following command on the PUB-SRV01 or PUB-SRV02 node to check whether the data source is restored:

    curl -k https://$agent_ip:9443/meterticket/agent/snapshot

    $agent_ip indicates the node where SDR Agent is located, that is, the PUB-SRV01 or PUB-SRV02 node.

    If information similar to the following is displayed but the content does not contain "failed" or "abnormal", the environment is normal.

    {
    "sftpConnection": "sftp connect success",
    "dataSource": {
    "https://metering.az2.dc2.ironic.com:443/v2/meters/-->volume": "normal",
    "https://metering.az2.dc2.ironic.com:443/v2/meters/-->shutoff_instance": "normal",
    "https://metering.az0.dc0.domainname.com:443/v2/meters/-->vpn.connection_cascade": "normal",
    }

  1. Use PuTTY to log in to the PUB-SRV01 and PUB-SRV02 node, respectively.

    The default username is meteradmin. The default password is Huawei@123.

    For details about how to obtain the default account and password, go to HUAWEI CLOUD Stack 6.5.0 Security Management Guide and click Download to obtain HUAWEI CLOUD Stack 6.5.1 Account List.

  2. Run the following command to manually generate a bill and check whether it is successfully generated:

    curl -H "Content-Type:application/json" -X POST -d '{"record_id":"$RECORDID"}' -v -k https://$CONTROLLERIP:9443/meterticket/controller/manualGenerate

    RECORDID indicates the task ID for generating the bill, that is, recordId obtained in 14. CONTROLLERIP indicates the PUB-SRV03 node IP address.

    • If yes, the command output contains "200 OK". Go to 29.
    • If no, the command output contains other information. Contact technical support for assistance.

  3. On the PUB-SRV01 or PUB-SRV02 node, run the following command to switch to the floating IP address of the SFTP server.

    sftp meteradmin@SFTP node(ManageOne-Service03 or ManageOne-Service04)floating IP address

    The default username is meteradmin. The default password is ManageOne12#$. To obtain the SFTP node floating IP address, search for ManageOne-Tenant-Float-IP on the 2.1 Tool-generated IP Parameters sheet in the exported xxx_export_all_CN.xlsm file.

    For details about how to obtain the default account and password, go to HUAWEI CLOUD Stack 6.5.0 Security Management Guide and click Download to obtain HUAWEI CLOUD Stack 6.5.1 Account List.

  4. Run the following commands to switch to the following directory to view the bill:

    cd /opt/meterfiles/uploads

    cd resource

    resource indicates the resource type in Other Information.

  5. Run the following command to check the current time and record it:

    date

  6. Check whether the latest bill is generated. If the difference between the time when the bill is generated and the current time is less than three minutes, SDR is restored.

    The command output is as follows. Information in the red box is the time when the bill is generated.

    • If yes, locate the alarm record in the alarm list on ManageOne Maintenance Portal, click in the Operation column, manually clear the alarm, and check whether it is cleared.
      • If yes, no further action is required.
      • If no, contact technical support for assistance.
    • If no, an internal data error may occur. In this case, go to 33.

  7. Contact technical support for assistance.

Reference

If no charging requirement is proposed, mask this alarm by performing steps provided in Setting Masking Rules.

ALM-2000317 SDR is abnormal

Description

This alarm is generated when the SDR health check is abnormal.

Attribute

Alarm ID

Alarm Severity

Auto Clear

2000317

Critical

Yes

Parameters

Parameter

Description

Location Info

Resource name

Name of the device for which the alarm is generated

Resource type

MONITOR

Monitor type

Service monitoring

Host IP address

IP address of the VM for which the alarm is generated

Details

Data in recent periods

Threshold

Threshold for generating an alarm

Impact on the System

SDR files may fail to be generated.

Possible Cause

The SDR process is not started or the database connection is abnormal.

Procedure

  1. Log in to the PUB-DB01 node using PuTTY.

    The default username is gaussdb. The default password is Huawei@123.

  2. Run the following command to disable user logout upon system timeout:

    TMOUT=0

  3. Run the following command and enter the password of user root to switch to user root:

    sudo su - root

    Run the following command to query the database status:

    service had query

    • If the database is running properly, go to 4.
    • If the database is running improperly, contact technical support for assistance.

  4. Log in to ManageOne Maintenance Portal using a browser.

    • URL: https://Address for accessing the homepage of ManageOne Maintenance Portal:31943, for example, https://oc.type.com:31943
    • Default username: admin; default password: Huawei12#$

  5. On the menu bar in the upper part of the page, choose Alarms > Current Alarms.
  6. In the alarm list, locate the alarm to be handled and click on the left of the alarm.
  7. Choose Location Info, obtain the host IP address, that is, the IP address of the node where the alarm is generated.
  8. Use PuTTY to log in to the node for which the alarm is generated. Ensure that the IP address of the node obtained in 7 is used to establish the connection.

    The default username is meteradmin. The default password is Huawei@123.

  9. Run the following command to disable user logout upon system timeout:

    TMOUT=0

  10. Run the following command to query the SDR process:

    ps -ef |grep meteradmin

    • If yes, go to 11 and then 13.
    • If no, go to 12.

  11. Run the following command to restart the SDR process:

    sh /home/meteradmin/meterticket-*/bin/stop.sh

    sh /home/meteradmin/meterticket-*/bin/startup.sh

  12. Run the following command to start the SDR process:

    sh /home/meteradmin/meterticket-*/bin/startup.sh

  13. Run the following command to check whether the SDR process is running properly:

    ps -ef |grep meteradmin

    • If yes, go to 14.
    • If no, contact technical support for assistance.

  14. Wait for 10 minutes and check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, contact technical support for assistance.

ALM-2000327 SDR's certificate alarm

Description

This alarm is generated when the certificate of the SDR node expires.

Attribute

Alarm ID

Alarm Severity

Auto Clear

2000327

Critical

Yes

Parameters

Parameter

Description

Location Info

Resource name

Name of the device for which the alarm is generated

Resource type

MONITOR

Monitor type

Service monitoring

Host IP address

IP address of the VM for which the alarm is generated

Details

Data in recent periods

Threshold

Threshold for generating an alarm

Impact on the System

API invocation may be adversely affected.

Possible Cause

The certificate of the SDR node has expired or will expire within 30 days.

Procedure

  1. Log in to ManageOne Maintenance Portal using a browser.

    • URL: https://Address for accessing the homepage of ManageOne Maintenance Portal:31943, for example, https://oc.type.com:31943
    • Default username: admin; default password: Huawei12#$

  2. On the menu bar in the upper part of the page, choose Alarms > Current Alarms.
  3. In the alarm list, locate the alarm to be handled and click on the left of the alarm.
  4. Choose Location Info, obtain the host IP address, that is, the IP address of the node where the alarm is generated.
  5. Determine the node for which the alarm is generated based on the node IP address obtained in 4.

    • If the node is a controller node, perform 6 to 17.
    • If the node is an agent node, perform 18 to 29.

Replacing the Certificate on the Controller Node

  1. Use a file transfer tool, such as WinSCP, to upload the obtained certificate to the /home/meteradmin directory on the controller node.
  2. Use PuTTY to log in to the controller node as user meteradmin.

    NOTE:

    The default password of user root is Cloud12#$, that of user meteradmin is Huawei@123, and that of the server.crt certificate is Huadan@szx666.

  3. Run the following command to disable user logout upon system timeout:

    TMOUT=0

  4. Run the following command to back up original certificate store file server.keystore:

    mv /home/meteradmin/meterticket-controller/resources/keystore/server.keystore /home/meteradmin/meterticket-controller/resources/keystore/server.keystore.bak

  5. Run the following command to generate new certificate store file server.keystore:

    keytool -genkey -alias Certificate store alias -keypass Alias password -keyalg Algorithm -keysize Key length -validity Validity (days) -keystore Directory and name of the generated certificate-storepass Keystore password-dname "C=two-word country code, ST=state or province name, L=city or region name, O=organization name, OU=department name, CN=issuer name"

    For example, run the following command to generate server.keystore:

    keytool -genkey -alias sdr_jetty -keypass Huadan@szx666 -keyalg RSA -keysize 2048 -validity 3650 -keystore server.keystore -storepass Huadan@szx666 -dname "C=CN, ST=Shaanxi, L=Xi'an, O=Huixin, OU=IT, CN=PRIVATE_CLOUD"

  6. Run the following command to import the root certificate:

    keytool -import -v -trustcacerts -alias ca_root -file CA certificate name -keystore server.keystore

    For example, run the following command to import the ca.crt certificate:

    keytool -import -v -trustcacerts -alias ca_root -file ca.crt -keystore server.keystore

    The following information is displayed:

    Enter keystore password:
    NOTE:

    If the command output indicates that the keytool command does not exist, run the following command to import environment variables:

    source /etc/profile

  7. Enter the password of the certificate store and press Enter.
  8. Enter yes and press the Enter key.

    If the following information is displayed, the CA certificate is imported:

    Certificate was added to keystore  
      [Storing server.keystore]

  9. Run the following command to import the signed certificate:

    keytool -import -v -trustcacerts -alias ca_root -file Certificate name -keystore server.keystore

    For example, run the following command to import the server.crt certificate:

    keytool -import -v -trustcacerts -alias ca_server -file server.crt -keystore server.keystore

    The following information is displayed:

    Enter keystore password:

  1. Enter the password of the certificate store and press Enter.
  2. Enter yes and press Enter.
  3. If the alarm persists after the certificate is replaced, contact technical support for assistance.

Replacing the Certificate on the Agent Node

  1. Use a file transfer tool, such as WinSCP, to upload the obtained certificate to the /home/meteradmin directory on the agent node.
  2. Use PuTTY to log in to the agent node as the meteradmin user.

    NOTE:

    The default password of user root is Cloud12#$, that of user meteradmin is Huawei@123, and that of the server.crt certificate is Huadan@szx666.

  3. Run the following command to disable user logout upon system timeout:

    TMOUT=0

  4. Run the following command to back up original certificate store file server.keystore:

    mv /home/meteradmin/meterticket-agent/resources/keystore/server.keystore /home/meteradmin/meterticket-agent/resources/keystore/server.keystore.bak

  5. Run the following command to generate new certificate store file server.keystore:

    keytool -genkey -alias Certificate store alias -keypass Alias password -keyalg Algorithm -keysize Key length -validity Validity (days) -keystore Directory and name of the generated certificate-storepass Keystore password-dname "C=two-word country code, ST=state or province name, L=city or region name, O=organization name, OU=department name, CN=issuer name"

    For example, run the following command to generate server.keystore:

    keytool -genkey -alias sdr_jetty -keypass Huadan@szx666 -keyalg RSA -keysize 2048 -validity 3650 -keystore server.keystore -storepass Huadan@szx666 -dname "C=CN, ST=Shaanxi, L=Xi'an, O=Huixin, OU=IT, CN=PRIVATE_CLOUD"

  6. Run the following command to import the root certificate:

    keytool -import -v -trustcacerts -alias ca_root -file CA certificate name -keystore server.keystore

    For example, run the following command to import the ca.crt certificate:

    keytool -import -v -trustcacerts -alias ca_root -file ca.crt -keystore server.keystore

    The following information is displayed:

    Enter keystore password:
    NOTE:

    If the command output indicates that the keytool command does not exist, run the following command to import environment variables:

    source /etc/profile

  7. Enter the password of the certificate store and press Enter.
  8. Enter yes and press the Enter key.

    If the following information is displayed, the CA certificate is imported:

    Certificate was added to keystore  
      [Storing server.keystore]

  9. Run the following command to import the signed certificate:

    keytool -import -v -trustcacerts -alias ca_root -file Certificate name -keystore server.keystore

    For example, run the following command to import the server.crt certificate:

    keytool -import -v -trustcacerts -alias ca_server -file server.crt -keystore server.keystore

    The following information is displayed:

    Enter keystore password:

  1. Enter the password of the certificate store and press Enter.
  2. Enter yes and press the Enter key.
  3. If the alarm persists after the certificate is replaced, contact technical support for assistance.
Translation
Download
Updated: 2019-08-30

Document ID: EDOC1100062365

Views: 45769

Downloads: 33

Average rating:
This Document Applies to these Products
Related Version
Related Documents
Share
Previous Next