No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

HUAWEI CLOUD Stack 6.5.0 Alarm and Event Reference 04

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
ALM-16006 LVS Concurrency Overload

ALM-16006 LVS Concurrency Overload

Description

This alarm is generated when the number of concurrent requests received by an LVS node per second exceeds 1 million.

Attribute

Alarm ID

Alarm Severity

Alarm Type

16006

Major

Communications alarm

Parameters (for Alarm Locating)

Table 17-13 Parameters

Name

Description

Example

namespace

Namespace to which the ELB service belongs.

elb

ServiceName

Name of the service that generates the alarm.

ELB

InstanceName

Name of the microservice that generates the alarm.

elb-svc

LVSAddr

List of LVS node IP addresses.

10.8.154.144,10.8.154.145

Impact on the System

The data plane may respond slowly.

System Actions

None.

Possible Causes

The number of concurrent requests received by an LVS node per second exceeds 1 million.

Solution

Perform the following steps:

  1. Log in to the FusionStage tenant management zone console.

    The console address is https://Access address of the console home page, for example, https://console.demo.com.

  2. Choose Console > Application > FusionStage from the main menu.
  3. Choose Application Operations > Application Operations
  4. Choose Alarm Center > Alarm List to query the alarm information about ALM-16006 LVS Concurrency Overload.

    Additional information: concurrency overload hosts

  5. Use PuTTY to log in to the LVS node.

    The default username is paas. The default password is QAZ2wsx@123!.

  6. Run the following command to query the distribution policy:

    sudo ipvsadm -Ln

    Information similar to the following is displayed:

    IP Virtual Server version 1.2.1 (size=4194304)
    Prot LocalAddress:Port Scheduler Established(Sec.) Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    TCP  10.8.154.17:7793 consh                       persistent 864000
      -> 10.8.154.146:1051           FullNat 6      0          0
      -> 10.8.154.147:1051           FullNat 6      0          0
    • If the distribution policy is round robin (the content in red is rr), go to 7.
    • If the distribution policy is not round robin (for example, the content in red is consh), the possible cause is that too many connections are initiated during a certain period on the client. Go to 9.

  7. Use PuTTY to log in to the ELB management node.

    The default username is paas. The default password is QAZ2wsx@123!.

  8. Check whether the connection to the LVS nodes not displayed in the additional information is abnormal, which results in overload of the LVS node displayed in the additional information.

    1. Run the following command to obtain the token:

      token=`curl -i -k -H Accept:application/json -H 'Content-Type:application/json;charset=utf8' -X POST -d '{"auth": {"identity":{"methods": ["password"],"password":{"user": {"name":"${name}","password":"${password}","domain": {"name":"${domain_name}"}}}},"scope": {"project":{"name": "${project}"}}}}' https://${HOST}:28330/v3/auth/tokens 2>/dev/null | grep X-Subject-Token|awk -F ':' '{print $2}' | tr -d "015" | sed s/[[:space:]]//g`

      Obtain and replace the following parameter values in the command with the actual ones:
      • ${name}: Indicates the username.
      • ${domain_name}: Indicates the tenant name.
      • ${project}: Indicates the project name.
      • ${password}: Indicates the login password.
      • ${HOST}: Indicates the address of the Identity and Access Management (IAM) service. For details, see section "Obtaining the IP Address of the IAM Server" in FusionStage ELB 6.5.0.SPC100 Installation Guide - HUAWEI CLOUD Stack.
    2. Check command: curl -k -H "X-Auth-Token: ${token}" https://10.247.204.219:8443/v1.0/elbaas/version
      NOTE:

      10.247.204.219 indicates the VIP of the management node.

    3. If the version of the LVS node in the command output is unknown, the connection is abnormal.
      • If the connection is abnormal, contact technical support to rectify the connection fault.
      • If the communication is normal, go to 9.

  9. Use PuTTY to log in to the LVS node with throughput overload.

    The default username is paas. The default password is QAZ2wsx@123!.

  10. Run the following command and enter the password of the root user to switch to the root user:

    su - root

    Default password: QAZ2wsx@123!

  11. Run the following command to view the traffic information:

    watch -n 1 '/sbin/ipvsadm -ln --stats'

    The command output is as follows. The value in the Conns column indicates the number of concurrent connections of the LVS node. Check the hosts on which concurrent connections per second increase sharply. After you have identified the host with overloaded traffic, you are advised to control the traffic, for example, reduce the number of application visits.

    Every 1.0s: ipvsadm -ln --stats
    IP Virtual Server version 1.2.1 (size=4194304)
    Prot LocalAddress:Port  Conns InPkts  OutPkts  InBytes  OutBytes  -> RemoteAddress:Port
    TCP 192.168.0.1:8083    0     0       0        0        0
    -> 10.96.6.33:1055       0     0       0        0        0
    TCP 192.168.0.2:8083    0     0       0        0        0
    -> 10.96.6.33:1051       0     0       0        0        0
    TCP 192.168.0.3:8082    0     0       0        0        0
    TCP 192.168.10.30:8080   3250  19500   13000    1400750  2793075   -> 10.96.6.33:1052       0     0       0        0        0

Alarm Clearance

After the fault is rectified, the system automatically clears the alarm.

Related Information

None.

Translation
Download
Updated: 2019-08-30

Document ID: EDOC1100062365

Views: 34637

Downloads: 31

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next