Notes and Constraints

• If auto.create.groups.enable is set to true, the consumer group status is EMPTY, and no offset has been submitted, the system automatically deletes the consumer group 10 minutes later.
• If auto.create.groups.enable is set to false, the system does not automatically delete consumer groups. You can manually delete them.
• If a consumer group has never committed an offset, the group will be deleted after the Kafka instance restarts.

Procedure

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view the instance details.
4. In the navigation pane, choose Consumer Groups.
5. Click Create Consumer Group.
6. Set consumer group parameters by referring to Table 2-32 and click OK.
   Table 2-32 Consumer group parameters

   Parameter	Description
   Consumer Group Name	Enter 3 to 64 characters, starting with a letter or underscore (_). Use only letters, digits, periods (.), hyphens (-), and underscores (_). If a consumer group name starts with a special character, for example, a number sign (#), monitoring data cannot be displayed.
   Description	Enter 0 to 200 characters.

   View the new consumer group in the consumer group list.

Viewing the Consumer Group List (Console)

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view its details.
4. In the navigation pane, choose the Consumer Groups tab.

   The consumer group name, status, Coordinator (ID), and description are displayed. Coordinator (ID) indicates the broker where the coordinator component is located. The consumer group status can be:

   • DEAD: The consumer group has no member or metadata.
   • EMPTY: The consumer group has metadata but has no member.
   • PREPARING_REBALANCE: The consumer group is to be rebalanced.
   • COMPLETING_REBALANCE: All members have joined the consumer group.
   • STABLE: Members in the consumer group can consume messages normally.

5. (Optional) To query a consumer group, enter a consumer group name or status, Coordinator (ID),description, or keyword, then press Enter.
6. (Optional) To refresh the consumer group list, click in the upper right corner.

Viewing the Consumer Group List (Kafka CLI)

• For a Kafka instance with SASL disabled, run the following command in the /bin directory of the Kafka client:

  ./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --list

  Parameter description: connection-address indicates the Kafka instance address, which can be obtained in the Connection area on the Basic Information page on the Kafka console.

  Example:

  [root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9092,192.xx.xx.212:9092,192.xx.xx.147:9092 --list
  test
  __consumer-group-dial-test
  [root@ecs-kafka bin]#

• For a Kafka instance with SASL enabled, do as follows:
  1. (Optional) If the username and password, and the SSL certificate has been configured, skip this step and go to 2. Otherwise, do as follows:

     Create the ssl-user-config.properties file in the /config directory of the Kafka client. Add the username and password, and the SSL certificate configuration by referring to Step 3.

  2. Run the following command in the /bin directory of the Kafka client:

     ./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --list --command-config ../config/ssl-user-config.properties 

     Parameter description: connection-address indicates the Kafka instance address, which can be obtained in the Connection area on the Basic Information page on the Kafka console.

     Example:

     [root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9093,192.168.xx.xx:9093,192.168.xx.xx:9093 --list --command-config ../config/ssl-user-config.properties
     test
     __consumer-group-dial-test
     [root@ecs-kafka bin]#

Prerequisites

The consumer list and connection address can be viewed only when consumers in a consumer group are connected to the Kafka instance (that is, the consumer group is in the STABLE state).

Viewing the Consumer List (Console)

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click a Kafka instance name to go to the instance details page.
4. In the navigation pane, choose Consumer Groups.
5. Click the name of the desired consumer group.
6. On the Consumers tab page, view the consumer list.

   In the consumer list, you can view the consumer ID, consumer address, and client ID.

7. (Optional) To query a specific consumer, enter the consumer ID in the search box and press Enter.

Viewing the Consumer List (Kafka CLI)

• For a Kafka instance with SASL disabled, run the following command in the /bin directory of the Kafka client:

  ./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --group ${group-name} --members --describe

  Parameter description:

  • connection-address: can be obtained from the Connection area on the Basic Information page on the Kafka console.
  • group-name: consumer group name.

  Example:

  [root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9092,192.168.xx.xx:9092,192.168.xx.xx:9092 --group test --members --describe
  
  GROUP           CONSUMER-ID                                           HOST            CLIENT-ID        #PARTITIONS     
  test            console-consumer-571a64fe-b0c4-47ce-833d-9e0da5a88d14 /192.168.0.215  console-consumer 3               
  [root@ecs-kafka bin]#

• For a Kafka instance with SASL enabled, do as follows:
  1. (Optional) If the username and password, and the SSL certificate has been configured, skip this step and go to 2. Otherwise, do as follows:

     Create the ssl-user-config.properties file in the /config directory of the Kafka client. Add the username and password, and the SSL certificate configuration by referring to Step 3.

  2. Run the following command in the /bin directory of the Kafka client:

     ./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --group ${group-name} --members --describe --command-config ../config/ssl-user-config.properties 

     Parameter description:

     • connection-address: can be obtained from the Connection area on the Basic Information page on the Kafka console.
     • group-name: consumer group name.

     Example:

     [root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9093,192.168.xx.xx:9093,192.168.xx.xx:9093 --group test --members --describe --command-config ../config/ssl-user-config.properties
     
     GROUP           CONSUMER-ID                                           HOST            CLIENT-ID        #PARTITIONS     
     test            console-consumer-566d0c82-07d3-4d87-9a6e-f57a9bc9fc69 /192.168.0.215  console-consumer 3               
     [root@ecs-kafka bin]#

Viewing Consumer Connection Addresses (Console)

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view the instance details.
4. In the navigation pane, choose Consumer Groups.
5. Click the desired consumer group.
6. On the Consumers tab page, view the consumer addresses.
   Figure 2-26 Consumer list
   

Prerequisites

The consumer offset cannot be reset on the fly. You must first stop retrieval of the desired consumer group.

After a client is stopped, the server considers the client offline only after the time period specified in ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG (1000 ms by default).

Viewing Consumer Offsets (Console)

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view the instance details.
4. In the navigation pane, choose Consumer Groups.
5. Click the name of the desired consumer group.
6. On the Consumer Offset tab page, view the list of topics that the consumer group has subscribed to, total number of messages accumulated in the topic, consumption progress in each partition of the topic (number of accumulated messages, offset, latest offset, consumer ID, consumer address, and client ID).
   Figure 2-27 Consumer offsets
   

7. (Optional) To query the consumer offsets of a specific topic, enter the topic name in the search box and press Enter.

Viewing Consumer Offsets (Kafka CLI)

• For a Kafka instance with SASL disabled, run the following command in the /bin directory of the Kafka client:

  ./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --offsets --describe --all-groups

  Parameter description: connection-address indicates the Kafka instance address, which can be obtained in the Connection area on the Basic Information page on the Kafka console.

  Example:

  [root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9092,192.168.xx.xx:9092,192.168.xx.xx:9092 --offsets --describe --all-groups
  
  Consumer group '__consumer-group-dial-test' has no active members.
  
  GROUP                      TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
  __consumer-group-dial-test __dms_dial_test 0          350             350             0               -               -               -
  __consumer-group-dial-test __dms_dial_test 1          350             350             0               -               -               -
  __consumer-group-dial-test __dms_dial_test 2          350             350             0               -               -               -
  
  Consumer group 'test' has no active members.
  
  GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
  test            topic-01        0          5               5               0               -               -               -
  test            topic-01        1          3               3               0               -               -               -
  test            topic-01        2          10              10              0               -               -               -
  [root@ecs-kafka bin]#

• For a Kafka instance with SASL enabled, do as follows:
  1. (Optional) If the username and password, and the SSL certificate has been configured, skip this step and go to 2. Otherwise, do as follows:

     Create the ssl-user-config.properties file in the /config directory of the Kafka client. Add the username and password, and the SSL certificate configuration by referring to Step 3.

  2. Run the following command in the /bin directory of the Kafka client:

     ./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --offsets --describe --all-groups --command-config ../config/ssl-user-config.properties 

     Parameter description: connection-address indicates the Kafka instance address, which can be obtained in the Connection area on the Basic Information page on the Kafka console.

     Example:

     [root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9093,192.168.xx.xx:9093,192.168.xx.xx:9093 --offsets --describe --all-groups --command-config ../config/ssl-user-config.properties
     
     Consumer group '__consumer-group-dial-test' has no active members.
     
     GROUP                      TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
     __consumer-group-dial-test __dms_dial_test 0          347             347             0               -               -               -
     __consumer-group-dial-test __dms_dial_test 1          347             347             0               -               -               -
     __consumer-group-dial-test __dms_dial_test 2          347             347             0               -               -               -
     
     Consumer group 'test' has no active members.
     
     GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
     test            topic-01        0          5               5               0               -               -               -
     test            topic-01        1          3               3               0               -               -               -
     test            topic-01        2          10              10              0               -               -               -
     [root@ecs-kafka bin]#

Resetting Consumer Offsets

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view the instance details.
4. In the navigation pane, choose the Consumer Groups tab.
5. Click the name of the desired consumer group.
6. On the Consumer Offset tab page, you can perform the following operations:
   • To reset the consumer offset of all partitions of a single topic, click Reset Consumer Offset in the row containing the desired topic.
   • To reset the consumer offset of a single partition of a single topic, click Reset Consumer Offset in the row containing the desired partition.
   • To reset the consumer offset of all partitions in all topics, click Reset Offset above the list.

7. In the displayed Reset Consumer Offset dialog box, set the parameters by referring to Table 2-33.
   Table 2-33 Parameters for resetting the consumer offset

   Parameter	Description
   Reset By	You can reset an offset by: Time: Reset the offset to the specified time. Offset: Reset the offset to the specified position. If you reset offsets in batches, they can only be reset to the specified time.
   Time	Set this parameter if Reset By is set to Time. Select a time point. After the reset is complete, retrieval starts from this time point. Earliest: earliest offset Custom: a custom time point Latest: latest offset
   Offset	Set this parameter if Reset By is set to Offset. Enter an offset, which is greater than or equal to 0. After the reset is complete, retrieval starts from this offset.

8. Click OK.
9. Click Yes in the confirmation dialog box. The consumer offset is reset.

Constraints

• Rebalancing logs are stored for seven days by default. To retain them longer, see "Log Management" > "Log Groups" in Log Tank Service (LTS) 2.2.0 Usage Guide (for Huawei Cloud Stack 8.5.0) > User Guide
• Enabling rebalancing logging will create a log group, log stream, and dashboard in LTS.

Prerequisites

• Ensure that you have permissions to create log groups and log streams in LTS.
• Rebalancing logging can be enabled or disabled only when the Kafka instance is in the Running state.

Enabling Rebalancing Logging

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view the instance details.
4. In the navigation pane, choose Rebalancing Logs.
5. Click Enable Logging. If the message "Rebalancing logging enabled" is displayed in the upper right corner of the page, the rebalancing log function is enabled successfully.

   Enabling rebalancing logging will create a log group and log stream in LTS.

Viewing Rebalancing Logs

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view the instance details.
4. In the navigation pane, choose Rebalancing Logs.
5. On the Dashboard tab page, view the number of consumer group rebalancing times and reasons. On the Logs tab page, view rebalancing logs.

   To search for logs, see "Log Search and Analysis" > "Searching Logs" > "Accessing the Log Search Page" in Log Tank Service (LTS) 2.2.0 Usage Guide (for Huawei Cloud Stack 8.5.0) > User Guide.

   An example rebalancing log:

   {
       "level":"INFO",
       "timestamp":"2023-03-23 17:23:22,906",
       "message":{
           "leaderId":"consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635",
           "generationId":"1",
           "reason":"Assignment received from leader for group KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0 for generation 1",
           "groupId":"KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0",
           "coordinatorId":"0",
           "type":"END_REBALANCE",
           "group":"GroupMetadata(groupId=KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0, generation=1, protocolType=Some(consumer), currentState=CompletingRebalance, members=Map(consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635 -> MemberMetadata(memberId=consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635, clientId=consumer-1, clientHost=/172.31.2.168, sessionTimeoutMs=10000, rebalanceTimeoutMs=300000, supportedProtocols=List(range), )))"
       }
   }

   Table 2-34 describes the parameters.

   Table 2-34 Rebalancing parameters

   Parameter	Description
   level	Level of the rebalancing logs. The only value is INFO.
   timestamp	Time of rebalancing.
   leaderId	Leader consumer ID.
   generationId	Generation ID of the consumer group. Generation is the number of times that a consumer group performs rebalancing. It is incremented by 1 each time a rebalancing is complete.
   reason	Reason for triggering rebalancing.
   groupId	Consumer group ID.
   coordinatorId	Broker where the Coordinator component is.
   type	Operation that triggered rebalancing. Values: JOIN_GROUP: A new consumer is added to a consumer group. OVER_CAPACITY: The group limit is exceeded. UPDATE_MEMBER: The consumer metadata is updated. PROTOCOL_CHANGE: The protocol is changed. HEARTBEAT_EXPIRED: The consumer heartbeat timed out. SYNC_GROUP: The reassignment plan is synchronized. END_REBALANCE: Rebalancing ended. LEAVE_GROUP: A consumer left a consumer group.
   group	Information about consumers in the consumer group.

Disabling Rebalancing Logging

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view the instance details.
4. In the navigation pane, choose Rebalancing Logs.
5. In the upper right corner of the page, click Disable Logging. In the dialog box that is displayed, click OK.

   This only disables the rebalancing logging function. The log groups and log streams on LTS are retained. If you no longer need the logs, delete the log groups and log streams on LTS.

Procedure

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view details.
4. In the navigation pane, choose Consumer Groups.
5. In the row containing the consumer group to be edited, click Edit.
6. Modify the description and click OK.

   After the modification is successful, you can view the new description in the Description column.

Procedure

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view its details.
4. In the navigation pane, choose Consumer Groups.
5. Export consumer groups in either of the following ways:
   • Select the desired consumer groups and choose Export > Export selected data to an XLSX file to export specified consumer groups.
   • Choose Export > Export all data to an XLSX file to export all consumer groups.

Constraints

• If auto.create.groups.enable is set to true, the consumer group status is EMPTY, and no offset has been submitted, the system automatically deletes the consumer group 10 minutes later.
• If auto.create.groups.enable is set to false, the system does not automatically delete consumer groups. You can manually delete them.
• If a consumer group has never committed an offset, the group will be deleted after the Kafka instance restarts.

Prerequisites

The status of the consumer group to be deleted is EMPTY.

Deleting a Consumer Group on the Console

1. Use a browser to log in to ManageOne as a VDC administrator, a VDC operator, or a user with the DMS FullAccess permission.

   URL of the unified portal: https://ManageOne Unified Portal URL, for example, https://console.demo.com/moserviceaccesswebsite/unifyportal#/home On the home page, choose Self-service Cloud Service Center.

2. Click in the upper left corner of the page, select a region, and choose Application > Distributed Message Service.

   Log in to the DMS console.

3. Click the desired Kafka instance to view the instance details.
4. In the navigation pane, choose the Consumer Groups tab.
5. Delete consumer groups using either of the following methods:
   • Select one or more consumer groups and click Delete Consumer Group above the consumer group list.
   • In the row containing the consumer group you want to delete, click Delete.

   A consumer group can be deleted only when its status is EMPTY.

   Consumer group statuses include:

   • DEAD: The consumer group has no member or metadata.
   • EMPTY: The consumer group has metadata but has no member.
   • PREPARING_REBALANCE: The consumer group is to be rebalanced.
   • COMPLETING_REBALANCE: All members have joined the consumer group.
   • STABLE: Members in the consumer group can consume messages normally.

6. In the displayed Delete Consumer Group dialog box, click OK.

Using the CLI to Delete a Consumer Group

The following uses Linux as an example.

• For a Kafka instance with SASL disabled, run the following command in the /bin directory of the Kafka client:

  ./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --delete --group ${consumer-group-name}

  Parameter description:

  • connection-address: can be obtained from the Connection area on the Basic Information page on the Kafka console.
  • consumer-group-name: consumer group name.

  Example:

  [root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9092,192.168.xx.xx:9092,192.168.xx.xx:9092 --delete --group group-01
  Deletion of requested consumer groups ('group-01') was successful.
  [root@ecs-kafka bin]#

• For a Kafka instance with SASL enabled, do as follows:
  1. (Optional) If the SSL certificate has been configured, skip this step and go to 2. Otherwise, do as follows:

     Create the ssl-user-config.properties file in the /config directory of the Kafka client and add the SSL certificate configurations by referring to Step 3.

  2. In the /bin directory of the Kafka client, run the following command:

     ./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --delete --group ${consumer-group-name} --command-config ../config/ssl-user-config.properties

     Parameter description:

     • connection-address: can be obtained from the Connection area on the Basic Information page on the Kafka console.
     • consumer-group-name: consumer group name.

     Example:

     [root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9093,192.168.xx.xx:9093,192.168.xx.xx:9093 --delete --group group-02 --command-config ../config/ssl-user-config.properties
     Deletion of requested consumer groups ('group-02') was successful.
     [root@ecs-kafka bin]#