No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionInsight HD 6.5.0 Product Description 02

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Hardware and Running Environment Requirements

Hardware and Running Environment Requirements

Hardware Requirements

HUAWEI FusionInsight HD supports universal x86 servers or Huawei TaiShan ARM server. Enterprises can select servers as required.

  • Nodes installed in the same batch must use servers of the same architecture, x86 or ARM. During capacity expansion, servers of other architectures can be added.
  • The service end and client can be installed on servers of different architectures, x86 or ARM.

Ensure that each server meets the requirements listed in Table 3-3.

Table 3-3 Minimum requirements for hardware

Hardware

Configuration

CPU

  • X86:
    • Minimum configuration: dual-socket four-core Intel processor.
    • Recommended configuration: dual-socket eight-core Intel processor.
    • Recommended configuration for clusters with over 2000 nodes: dual-socket ten-core Intel processor for management nodes and control nodes.
  • Huawei TaiShan ARM server: dual-socket 32-core 1616 processor or more.

Bit-mode

64 bits

Memory

  • ≥ 64 GB. To meet the requirements of actual service running, the memory size must be planned based on actual service deployment. For details, see 3.3-Deployment Principles.
  • Clusters with more than 100 nodes: 256 to 512 GB memory is recommended for a single management node or control node.

NIC

  • Two GE electrical ports are used on the management plane and the ports are bonded.
  • Two 10GE optical ports are used on the service plane and the ports are bonded.
  • Two GE electrical ports are used on the external management network and the ports are bonded.
    NOTE:
    • The methods of naming NICs on the x86 platform and ARM platform are different if the NICs are not bonded. For example, the NIC on the x86 platform is named eth1 while on the ARM platform is named enahisic2i1. This document uses NICs on the x86 platform as an example. For NICs on the ARM platform, replace the names as required.
    • If the systemd patch is installed in the OS, use the default NIC name, for example, enahisic2i1. Do not change the default NIC name. Otherwise, the FusionInsight service is unavailable.
    • For details about the bond configuration type, see Network Planning > Cluster Networking >

      Host Port Configuration in FusionInsight HD 6.5.0 Hardware Deployment and Networking Guide.

RAID configuration

The non-OS disks of the management node and control node are used to store FusionInsight HD metadata, and the non-OS disks of the data node are used to store FusionInsight HD Hadoop data. To ensure reliability, the recommended RAID configurations on each node are as follows:

  • Management node: Both the OS disk and non-OS disk use an exclusive RAID group of RAID 1.
  • Control node: Both the OS disk and each non-OS disk use an exclusive RAID group of RAID 1.
  • Data node: The OS disk uses an exclusive RAID group of RAID 1; while RAID 0 or no RAID is configured for the non-OS disk.
NOTE:
  • If no RAID is configured for non-OS disks in the scenario where data disks are mounted to a RAID card, you need to configure the JBOD mode for the disks.
  • You are advised to configure RAID 5 for Kafka and Elasticsearch and Solr data disks.
  • If you use old disks for the installation, configure a RAID and format the disks before the installation.

Number of disks

The disk quantity indicates the number of disks that can be detected by the OS after RAID groups are configured. For example, if a node has six disks and three RAID 1 groups are configured, the disk quantity of the node is 3.

If not all services are installed, you can reduce the number of metadata disks or data disks required by the services that are not installed. For details about the number of metadata disks or data disks required by each service, see the Preparing OS.

  • All nodes are deployed separately:
    • Management node: Each node has three disks (one OS disk and two metadata disk, that is, 2+4 physical disks).
    • Control node: Each node has six disks (one OS disk and five metadata disks, that is, 2+10 physical disks).
    • Data node:

      Each node has two physical disks that form a RAID1 group, and the disks are used as OS disks. The number of data disks is configured based on the capacity plan in each scenario.

  • Management nodes and control nodes are deployed together, and data nodes are deployed separately:
    • Management node and control node: Each node has eight disks (one OS disk and seven metadata disks, that is, 2+14 physical disks).
    • Data node: Each node has two physical disks that form a RAID1 group, and the disks are used as OS disks. The number of data disks is configured based on the capacity plan in each scenario.
  • All nodes are deployed together: The number of disks is configured based on the capacity plan in each scenario.
  • Disk quantity planning: Plan the disk quantity based on the total service volume.

    Example:

    The total amount of raw data is 1 PB, 10 TB of the raw data is hot data, and hot data is to be stored on SSDs. The size of a SATA disk is 4 TB, and that of an SSD is 500 GB. Data is stored in HDFS with three copies, and the tiered storage policy is ONE_SSD. The minimum number of required SSDs: 10 TB/500 GB x 3 = 60, the minimum number of required SATA disks: 1 PB/4 TB x 3 = 750.

NOTE:
  • It is recommended that all OS disks and metadata disks employ SAS disks and data disks employ SATA disks. If SSDs are configured, you are advised to use them for control nodes and management nodes.
  • If the Flume role of the Flume service is deployed on an independent node and uses the file channel, one OS disk and at least one data disk (not smaller than 300 GB, RAID 1) are required.
  • If the Supervisor role of Storm is deployed on a separated node, one OS disk and one Storm data disk (not smaller than 50 GB, RAID 0) are required.
  • If the Solr service is installed and the number of Solr service nodes exceeds 50, it is recommended that ZooKeeper metadata disks employ SSD disks.
  • Install the two data nodes of SolrServerAdmin. Add a data disk to each of the two data nodes.
  • If Solr index data is stored in the HDFS, no local disk space is occupied.
  • Each Redis instance corresponds to a disk partition. (The number of Redis instances is the number of CPU cores minus 2.) It is recommended that the number of Redis partitions allocated on each disk be equal to or less than five to ensure cluster performance.
  • Data nodes can be deployed on pure SSDs or on both SSDs and hard disks. The disk type is selected based on the HDFS data storage policy.

Disk space

For details about how to configure RAID, see How to Configure the Capacity of RAID1 in the Software Installation. If the disk capacity is 600 GB, the disk can be rebuilt within 4 hours after a disk fault occurs. You can flexibly configure the disk capacity based on the actual requirements.

  • Management node: The space of the OS disk is greater than or equal to 600 GB, and the space of each non-OS disk is greater than or equal to 600 GB.
  • Control node: The space of the OS disk is greater than or equal to 600 GB, and the space of each non-OS disk is greater than or equal to 600 GB.
  • Data node: The space of the OS disk is greater than or equal to 600 GB, and the space of each non-OS disk is greater than or equal to 600 GB.
NOTE:

If the disk capacity is not 600 GB, configure the disk capacity when Preparations for Installation > Preparing OS. For details, see How to Configure Disk Partitions When the Disk Capacity Is Insufficient in the Software Installation.

Disk type

You can use either of the following local storage type for big data storage based on site requirements:

  • SSDs provide high read and write performance. The storage capacity of SSDs is small and the unit storage cost is higher than that of common mechanical hard disks. Because the NVMe SSD does not support RAID 1, use the SAS SSD as the OS disk and metadata disk on the management node and control node.
  • Disks include SATA and SAS disks. They are the main storage types used by HDFS for data storage.
NOTE:
  • Disks are the main disk storage types on the big data platform. SSDs can be used for high-speed storage.
  • HDFS data nodes can use different disk types for data storage based on service requirements. For details, see Service Operation Guide > HDFS > Configuring HDFS Data Storage Policies. Disks must be planned for data nodes before the HDFS tiered storage function is used. Multiple disk types can be configured for a data node.
  • When using tiered storage, you are advised to configure multiple disk types and deploy heterogeneous disks (such as 12 SSD disks) separately, namely, on different nodes. Additionally, you should plan the disk quantity and capacity based on service requirements.
NOTICE:

Recommendations on disk selection for management nodes and control nodes:

  • OS: SAS disk or SSD is recommended.
  • Metadata: SAS disk or SSD is recommended for clusters with less than 1500 nodes, and SSD is recommended for clusters with more than 1500 nodes.

Running Environment Requirements

The following describes the local environment that can ensure the proper running of FusionInsight HD.

  • OS
    Table 3-4 FusionInsight HD-depended OS

    Server Type

    OS Software

    Supported Version

    Universal x86 servers

    SUSE

    • Recommended: SUSE Linux Enterprise Server 11 SP4 (SUSE 11.4)
    • Recommended: SUSE Linux Enterprise Server 12 SP1 (SUSE 12.1) (If you use this OS version, install the SUSE-SLE-SERVER-12-SP1-2016-930 patch provided by SUSE. Otherwise, NTP waiting times out and FusionInsight Manager installation fails.)
    • Available: SUSE Linux Enterprise Server 11 SP1 (SUSE 11.1)
    • Available: SUSE Linux Enterprise Server 11 SP2 (SUSE 11.2)
    • Available: SUSE Linux Enterprise Server 11 SP3 (SUSE 11.3)
    • Available: SUSE Linux Enterprise Server 12 (SUSE 12) (If you use this OS version, install the SUSE-SLE-SERVER-12-2016-933 patch provided by SUSE. Otherwise, the time synchronization between the active management node and other nodes fails.)
    • Available: SUSE Linux Enterprise Server 12 SP2 (SUSE 12.2)
    • Available: SUSE Linux Enterprise Server 12 SP3 (SUSE 12.3)

    Red Hat

    • Recommended: RedHat-6.6-x86_64 (Red Hat 6.6)
    • Recommended: RedHat-7.2-x86_64 (Red Hat7.2)
    • Available: RedHat-6.4-x86_64 (Red Hat 6.4)
    • Available: RedHat-6.5-x86_64 (Red Hat 6.5)
    • Available: RedHat-6.7-x86_64 (Red Hat 6.7)
    • Available: RedHat-6.8-x86_64 (Red Hat 6.8)
    • Available: RedHat-6.9-x86_64 (Red Hat 6.9)
    • Available: RedHat-7.0-x86_64 (Red Hat 7.0)
    • Available: RedHat-7.1-x86_64 (Red Hat 7.1)
    • Available: RedHat-7.3-x86_64 (Red Hat 7.3)
    • Available: RedHat-7.4-x86_64 (Red Hat 7.4) (If you use this OS version, install systemd-219-57.el7 provided by Red Hat. Otherwise, systemd may fail to provide services, causing system resources unavailable.)
    • Available: RedHat-7.5-x86_64 (Red Hat 7.5)
      NOTE:

      Elk does not support the Red Hat 7.5.

    CentOS

    • Available: CentOS-6.4 (CentOS 6.4)
    • Available: CentOS-6.5 (CentOS 6.5)
    • Available: CentOS-6.6 (CentOS 6.6)
    • Available: CentOS-6.7 (CentOS 6.7)
    • Available: CentOS-6.8 (CentOS 6.8)
    • Available: CentOS-6.9 (CentOS 6.9)
    • Available: CentOS-7.0 (CentOS 7.0)
    • Available: CentOS-7.1 (CentOS 7.1)
    • Available: CentOS-7.2 (CentOS 7.2)
    • Available: CentOS-7.3 (CentOS 7.3)
    • Available: CentOS-7.4 (CentOS 7.4)
    • Available: CentOS-7.5 (CentOS 7.5)
      NOTE:

      Elk does not support the CentOS 7.5.

    Euler

    • Available: EulerOS V2.0SP1 (EulerOS2.1)
    • Available: EulerOS V2.0SP2 (EulerOS2.2)
    • Available: EulerOS V2.0SP3 (EulerOS2.3)
    • Available: EulerOS V2.0SP5 (EulerOS2.5)
      NOTE:

      Elk only support the EulerOS V2.0SP3 system.

    NeoKylin

    • Available: NeoKylin-6.9
    • Available: NeoKylin-7.2
    • Available: NeoKylin-7.4
    • Available: NeoKylin-7.5
      NOTE:

      Elk does not support the NeoKylin system.

    Oracle

    • Available: Oracle Linux 7.4
    • Available: Oracle Linux 7.5
    • Available: Oracle Linux 7.6
      NOTE:

      Elk does not support the Oracle system.

    Huawei TaiShan ARM server

    NOTE:

    Elk only support the RedHat-7.5-aarch64 system.

    Red Hat

    • Available: RedHat-7.4-aarch64 (RedHat7.4) (If you use this OS version, install systemd-219-57.el7 provided by Red Hat. Otherwise, systemd may fail to provide services, causing system resources unavailable.)
    • Available: RedHat-7.5-aarch64 (RedHat7.5)

    CentOS

    • Available: CentOS-7.4-aarch64 (CentOS7.4) (If you use this OS version, install systemd-219-57.el7 provided by Red Hat. Otherwise, systemd may fail to provide services, causing system resources unavailable.)
    • Available: CentOS-7.5-aarch64 (CentOS7.5)

    Euler

    Available: EulerOS V2.0SP3 (EulerOS2.3)

  • Web browser
    • Supports Google Chrome.

      Windows OS: The Google Chrome 48 and later versions are supported. You are advised to use the latest version of Chrome.

    • Internet Explorer 9 Standards Mode or later is supported. You are advised to use Internet Explorer 11.
      NOTE:
      • If Solr is required, use Internet Explorer 10 or later.
      • If Hue is required, Internet Explorer may be incompatible with some content.
      • If Loader is required, Internet Explorer is not supported.
      • If HDFS is required, use Internet Explorer 11 or later.
      • If Metadata is required, use Internet Explorer 11 or later.
    • Supports Microsoft Edge 13.
    • Supported Mozilla Firefox versions:

      Windows: 43.x/50.x/58.x and later

      NOTE:

      If Hue is required, Mozilla Firefox may be incompatible with some content.

      • Linux:

        Firefox built-in CentOS 7.

        Firefox 17.0.4 provided by SUSE 11.3. The browser version is low. The layout of some pages is in disorder. You are advised to upgrade the browser to the latest version.

        NOTE:

        Hue does not support the browser of this version.

      NOTE:

      For Mozilla Firefox, you need to perform the following operations to ensure the This web page is being redirected to a new location. Would you like to resend the form data you have typed to the new location? dialog box is not prompted:

      1. Enter about:config in the address box, and press Enter. The configuration page is displayed.
      2. Search network.http.prompt-temp-redirect, and double-click the parameter to set it to false.
    NOTE:

    To access FusionInsight Manager using a browser of an earlier version (such as Google Chrome 21 and Mozilla Firefox 24.x), see section "Accessing FusionInsight Manager Using a Browser of an Earlier Version" in the Software Installation.

  • JDK

    The server and client support only built-in OpenJDK, version: 1.8(jdk-8u181-linux-x64), and therefore a JDK replacement is not allowed.

    Customers' applications that need to reference the JAR packages of SDK to run in the application processes support Oracle JDK, IBM JDK, and OpenJDK.
    • For x86 nodes that run clients, the following JDKs can be used:
      • Oracle JDK versions: 1.8
      • Recommended IBM JDK versions: 1.8.5.11
    • For nodes that run ARM and clients, the following JDK can be used:
      • OpenJDK: 1.8.0_161
Download
Updated: 2019-05-17

Document ID: EDOC1100074548

Views: 3206

Downloads: 36

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next