No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionInsight HD 6.5.0 Administrator Guide 02

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Configuration Principles

Configuration Principles

How to Obtain the Best Cluster Performance

  • Principle 1: CPU core allocation principle
    • Data node: It is recommended that two to four cores be reserved for other processes (database and HBase) and other cores be allocated to YARN.
    • Control node: It is recommended that six to eight cores be reserved because there are many running processes.
  • Principle 2: memory allocation

    Except the memory allocated to the OS and other services, allocate as much as possible memory to YARN.

  • Principle 3: virtual CPU cores allocation

    Set the number of virtual CPU cores to 1.5 to 2 times the number of logical CPU cores. If the upper layer computing applications have low computing capability requirements, you can set the parameter to 2 times the number of logical CPU cores.

  • Principle 4: disk I/O throughput

    Mount more disks to improve the disk I/O throughput.

Factors Affecting Performance

  • Factor 1: file server disk I/O

    Generally, the disk write and read speed is at 100 Mbit/s level. For example, the theoretical sequential read speed of the second generation SATA disks is 300 Mbit/s, in single-disk reading only. It is impossible to achieve 1 Gbit/s import speed. In single-disk reading, adding the number of maps may not be able to improve the import speed. As the number of map increases, the sequential file read on a disk becomes random file read, which deteriorates the read performance. If the poorest random read may become 800 Kbit/s. Therefore, to increase disk I/O read efficiency of a file server, you can use a professional file server, such as the NAS system. In addition, you can use more simple methods, such as building Raid 0 or Raid 5.

  • Factor 2: file server network bandwidth

    The large the network bandwidth of a single file server, the better the performance. 10000 Mbit/s or greater is recommended.

  • Factor 3: cluster node hardware configuration

    If the cluster node hardware configuration is high, for example, a node has many CPU cores and large memory, you can increase the number of concurrent map or reduce operations. If the hardware configuration of a single node is difficult to improve, you can increase the number of nodes in the cluster.

  • Factor 4: SFTP parameter configuration

    aes128-cbc is preferred if compression and encryption algorithm are not used and umac-64@openssh.com is preferred if the integrity check algorithm is used.

  • Factor 5: cluster parameter configuration
  • Factor 6: Linux file read-ahead value

    Run the following Linux command to set the disk file read-ahead value to 16384:

    echo 16384 > /sys/block/sda/queue/read_ahead_kb

    NOTE:

    sda indicates the disk name.

Download
Updated: 2019-05-17

Document ID: EDOC1100074522

Views: 6130

Downloads: 12

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next