No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionInsight HD 6.5.0 Software Installation 02

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Real-Time Retrieval Scenario

Real-Time Retrieval Scenario

Real-time retrieval refers to real-time data writing and real-time query based on primary index keys for massive data. The requirement on query response is high and the query conditions are relatively simple. If the query terms are complicated, search for the primary index keys using key words in all-domain data and then use the primary index keys for query.

In real-time retrieval scenarios, components such as HDFS, HBase, Elasticsearch (or Solr), Kafka(optional), and GraphBase need to be deployed, as shown in the following figure.

Figure 2-5 Real-time retrieval scenario

The configuration in the real-time retrieval scenario is as follows:

Table 2-6 Real-time retrieval scenario

Node Type

Server Configuration

Number of Nodes

Description

Management node

  • CPU:
    • X86: 2-socket 8-core CPU or above
    • Huawei TaiShan server: dual-socket 32-core 1616 processor or more
  • Memory: 256 GB or above
  • Disk: 6 x 2.5-inch 600 GB SAS disks
  • RAID card: 1 GB LSI RAID 0/1 card (supporting three or more RAID 1 groups)
  • NIC: Two access switches are connected after the bond is configured.
    • Management plane: Two GE ports are bonded.
    • Service plane: Two 10GE ports are bonded.

2

Six disks on a single node form three RAID 1 groups. The functions are as follows (two nodes in total and same partitions on each node). For details about partitions, see Software Installation > Preparations for Installation > Preparing OS.

  • OS disk
  • /srv/BigData/dbdata_om partition
  • /srv/BigData/LocalBackup partition

Control node

  • CPU:
    • X86: 2-socket 8-core CPU or above
    • Huawei TaiShan server: dual-socket 32-core 1616 processor or more
  • Memory: 256 GB or above
  • Disk:
    • 3 or 5 control nodes: 10 x 2.5-inch 600 GB SAS disks
    • 9 or 11 control nodes: 6 x 2.5-inch 600 GB SAS disks
  • RAID card: 1 GB LSI RAID 0/1 card (supporting four or more RAID 1 groups)
  • NIC: Two access switches are connected after the bond is configured.
    • Management plane: Two GE ports are bonded.
    • Service plane: Two 10GE ports are bonded.

3/5/9/11

  • The number of control nodes is calculated based on the number of data nodes. For details, see Software Installation > Installation Introduction > Solution Introduction > Installation Solution > Node Deployment Scheme.
  • Eight disks on a single node form four RAID 1 groups. The functions are as follows (11 RAID 1 groups are distributed on three nodes). For details about partitions, see Software Installation > Preparations for Installation > Preparing OS.
    • OS disk x 3
    • /srv/BigData/zookeeper x 3
      NOTE:

      If the number of data nodes is less than or equal to 100, deploy three ZooKeeper nodes. If the number of data nodes is greater than 100, deploy five ZooKeeper nodes.

    • /srv/BigData/journalnode x 3
    • /srv/BigData/namenode x 2

Management + Control node

(Integrated deployment)

  • CPU:
    • X86: 2-socket 8-core CPU or above
    • Huawei TaiShan server: dual-socket 32-core 1616 processor or more
  • Memory: 256 GB or above
  • Disk: 10 x 2.5-inch 600 GB SAS disks
  • RAID card: 1 GB LSI RAID 0/1 card (supporting five or more RAID 1 groups)
  • NIC: Two access switches are connected after the bond is configured.
    • Management plane: Two GE ports are bonded.
    • Service plane: Two 10GE ports are bonded.

3

None

HBase node

  • CPU:
    • X86: 2-socket 8-core CPU or above
    • Huawei TaiShan server: dual-socket 32-core 1616 processor or more
  • Memory: 256 GB or above
  • Disk:
    • 2 x 2.5-inch 600 GB SAS disks
    • 12 x 3.5-inch 4 TB SATA disks
  • RAID card: 1 GB LSI RAID 0/1 card
    • Supports one or more RAID 1 groups.
    • Supports 12 or more RAID 0 groups or JBOD.
  • NIC: Two access switches are connected after the bond is configured.
    • Management plane: Two GE ports are bonded.
    • Service plane: Two 10GE ports are bonded.

The number of nodes is calculated based on the data volume.

HBase nodes store data and provide the query capability.

Calculation of node quantity:

Number of nodes = Planned data volume x 1.5 (Data expansion rate) x 1 (Data compression rate) x 3 (three copies)/0.8 (Disk utilization)/0.9 (Disk number system conversion)/(12 (Number of disks) x 4 TB (Disk capacity))

NOTE:

The data can be compressed according to the actual situation. The compression ratio is closely related to the data content and format.

Solr node

  • CPU:
    • X86: 2-socket 8-core CPU or above
    • Huawei TaiShan server: dual-socket 32-core 1616 processor or more
  • Memory: 256 GB or above
  • Disk:
    • 2 x 2.5-inch 600 GB SAS disks
    • 24 x 2.5-inch 600 GB (or above) SAS disks
  • RAID card: 1 GB LSI RAID 0/1 card
    • Supports one or more RAID 1 groups.
    • Supports four or more RAID 5 groups.
  • NIC: Two access switches are connected after the bond is configured.
    • Management plane: Two GE ports are bonded.
    • Service plane: Two 10GE ports are bonded.

The number of nodes is calculated based on the data volume.

Solr nodes store data and provide the retrieval capability.

Calculation of node quantity:

Number of nodes = Planned data volume x 1.5 (Data expansion rate) x 1 (Data compression rate) x 2 (two copies)/0.8 (Disk utilization)/0.9 (Disk number system conversion)/(24 (Number of disks) x 600GB (Disk capacity) x 5 / 6 (RAID coefficient))

NOTE:

The data can be compressed according to the actual situation. The compression ratio is closely related to the data content and format.

Elasticsearch node

  • CPU:
    • X86: 2-socket 8-core CPU or above
    • Huawei TaiShan server: dual-socket 32-core 1616 processor or more
  • Memory: 256 GB or above
  • Disk:
    • 2 x 2.5-inch 600 GB SAS disks
    • 24 x 2.5-inch 600 GB (or above) SAS disks
  • RAID card: 1 GB LSI RAID 0/1 card
    • Supports one or more RAID 1 groups.
    • Supports four or more RAID 5 groups.
  • NIC: Two access switches are connected after the bond is configured.
    • Management plane: Two GE ports are bonded.
    • Service plane: Two 10GE ports are bonded.

The number of nodes is calculated based on the data volume.

Elasticsearch nodes store data and provide the retrieval capability.

Calculation of node quantity:

Number of nodes = Planned data volume x 1.5 (Data expansion rate) x 1 (Data compression rate) x 2 (two copies)/0.8 (Disk utilization)/0.9 (Disk number system conversion)/(24 (Number of disks) x 600 GB (Disk capacity) x 5 / 6 (RAID coefficient))

NOTE:

The data can be compressed according to the actual situation. The compression ratio is closely related to the data content and format.

Kafka node (optional)

  • CPU:
    • X86: 2-socket 8-core CPU or above
    • Huawei TaiShan server: dual-socket 32-core 1616 processor or more
  • Memory: 128 GB or above
  • Disk:
    • 2 x 2.5-inch 600 GB SAS disks
    • 25 x 2.5-inch 1.2 TB SAS disks
  • RAID card: 1 GB LSI RAID 0/1 card
    • Supports one or more RAID 1 groups.
    • Supports four or more RAID 5 groups.
  • NIC: Two access switches are connected after the bond is configured.
    • Management plane: Two GE ports are bonded.
    • Service plane: Two 10GE ports are bonded.

The number of nodes is calculated based on the throughput.

Cache for Kafka to forward messages for stream processing.

  • Two 600 GB SAS disks on a node form a RAID 1 group and are used as OS disks.
  • Twenty-five 1.2 TB SAS disk on a node form RAID 0 or no RAID groups.
  • Calculation of node quantity:
    • Based on the throughput:

      X (Total throughput, MB/s)/100 (Maximum Producer throughput per node, MB/s)/0.85 (Reservation ratio)

    • Based on the storage capacity:

      X (Total throughput, MB/s) x 3600 x 24 x D (Number of days, 7 by default) x 2 (Number of copies)/1024/1024/(25 x 1.2)/0.85 (Reservation ratio)

Use the larger value. At least two devices must be configured.

GraphBase node

  • CPU:
    • X86: 2-socket 8-core CPU or above
    • Huawei TaiShan server: dual-socket 32-core 1616 processor or more
  • Memory: 128 GB or above
  • Disk: 2 x 2.5-inch 600 GB SAS disks
  • RAID card: 1 GB LSI RAID 0/1 card

    Supports one or more RAID 1 groups.

  • NIC: Two access switches are connected after the bond is configured.
    • Management plane: Two GE ports are bonded.
    • Service plane: Two 10GE ports are bonded.

The number of nodes is calculated based on the number of concurrent REST interfaces.

Cache for Kafka to forward messages for stream processing.

Number of nodes = Number of concurrent query requests for Layer 3 expansion/10

The L3 line expansion query refers to the expansion of the N layer relation of the common services of the graph service. (N=3)

At least two devices must be configured.

Download
Updated: 2019-05-17

Document ID: EDOC1100074555

Views: 6123

Downloads: 6

Average rating:
This Document Applies to these Products
Related Version
Related Documents
Share
Previous Next