No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionInsight HD 6.5.0 Administrator Guide 02

Rate and give feedback :
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Configuring Racks for Hosts

Configuring Racks for Hosts

Scenarios

All hosts in a large cluster are usually deployed on multiple racks. Hosts on different racks communicate with each other through switches. The network bandwidth between different hosts on the same rack is much greater than that on different racks. In this case, plan the network topology based on the following requirements:

  • To improve the communication speed, it is recommended that data be exchanged between hosts on the same rack.
  • To improve the fault tolerance capability, distribute processes or data of distributed services on different hosts of multiple racks as dispersedly as possible.
Hadoop uses a file directory structure to represent hosts. Figure 6-4 displays a cluster with a two-layer network structure. In this example, you are advised to name the rack for Node 1 as /Switch1/Rack1 and that for Node 4 as /Switch1/Rack2.
Figure 6-4 Two-layer network structure

The HDFS cannot automatically determine the network topology of each DataNode in the cluster. You need to set the rack name to identify the rack where the host is located so that the NameNode can draw the network topology of the required DataNodes and back up data of the DataNodes to different racks. Similarly, YARN needs to obtain rack information and allocate tasks to different NodeManagers as required.

If the cluster network topology changes, you need to reallocate racks for hosts on FusionInsight Manager so that related services can be automatically adjusted.

Impact on the System

If the name of the host rack is changed, policy for storing HDFS replicas, Yarn task assignment, and storage location of Kafka partitions will be affected. After the modification, restart the HDFS, Yarn, and Kafka for the configuration to take effect.

Improper rack configuration will unbalance loads (including CPU, memory, disk, and network) among nodes in the cluster, which decreases the cluster reliability and stability. Therefore, before allocating racks, take all aspects into consideration and properly set racks.

Rack Allocation Policies

NOTE:

Physical rack: Indicates the real rack where the host resides.

Logical rack: Indicates the rack name of the host on FusionInsight Manager.

Policy 1: Each logical rack has nearly the same number of hosts.

Policy 2: The name of the logical rack of the host must comply with that of the physical rack to which the host belongs.

Policy 3: If there are only few hosts on a physical rack, combine this physical rack and other physical racks with few hosts into a logical rack, which complies with policy 1. Hosts in two equipment rooms cannot be placed in one logical rack. Otherwise, performance problems may be caused.

Policy 4: If there are lots of hosts on a physical rack, divide these hosts into multiple logical racks, which complies with policy 1. Hosts with great differences should not be placed in the same logical rack. Otherwise, the cluster reliability will be decreased.

Policy 5: You are advised to set default or other values for logical racks on the first layer, and the values in the same cluster must be consistent.

Policy 6: The number of hosts in each rack cannot be less than 3.

Policy 7: A cluster can contain at most 50 logical racks. If there are too many logical racks in a cluster, the maintenance is difficult.

Best Practice Example

For example, there are 100 hosts in a cluster, 40 located in equipment room A and 60 located in equipment room B. In room A, there are 11 hosts on physical rack Ra1 and 29 hosts on physical rack Ra2. In room B, there are six hosts on physical rack Rb1, 33 hosts on physical rack Rb2, 18 hosts on physical rack Rb3, and three hosts on physical rack Rb4.

According to the rack allocation policy, each logical rack contains nearly the same number (for example, 20) of hosts. The allocation details are as follows:

  • Logical rack /default/racka1: contains 11 hosts on physical rack Ra1 and nine hosts on physical rack Ra2
  • Logical rack /default/racka2: contains the remaining 20 hosts (except the nine hosts of logical rack /default/racka1) on physical rack Ra2
  • Logical rack /default/rackb1: contains six hosts on physical rack Rb1 and 13 hosts on physical rack Rb2
  • Logical rack /default/rackb2: contains the remaining 20 hosts (except the 13 hosts of logical rack /default/rackb1) on physical rack Rb2
  • Logical rack /default/rackb3: contains 18 hosts on physical rack Rb3 and three hosts on physical rack Rb4

Rack allocation example:

Procedure

  1. Log in to FusionInsight Manager.
  2. Click Host.
  3. Select the check box of the target host.
  4. Select Set Rack from the More drop-down list.

    • Set rack names in hierarchy based on the actual network topology. Separate racks from different layers using slashes (/).
    • Rack naming rules are as follows: /level1/level2/... The number of levels must be at least 1, and the name cannot be empty. A rack can contain letters, digits, and underscores (_) and cannot exceed 200 characters.

      For example, /default/rack0.

Download
Updated: 2019-05-17

Document ID: EDOC1100074522

Views: 6103

Downloads: 12

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next