No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionInsight HD V100R002C60SPC200 Product Description 06

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Software Deployment Scheme

Software Deployment Scheme

Software List

Table 3-5shows the versions of the open-source components required by FusionInsight HD V100R002C60SPC200.

Table 3-5 Software list

Component

Version

Hadoop

2.7.2

HBase

1.0.2

Hive

1.2.1

ZooKeeper

3.5.1

Phoenix

4.4.0

Oozie

4.2.0

Hue

3.9.0

Spark

1.5.1

Sqoop

1.99.3

Flume

1.6.0

Kafka 2.10-0.9.0.0
Solr 5.3.1

Kerberos

1.10.7

Streaming

0.10.0

Redis

3.0.5

Deployment Principles

Table 3-6 shows the deployment principles of service roles.

NOTE:
  • The dependency and association relationships between services in the cluster are as follows:
    • A depends on B, which indicates that if A is deployed in the cluster, B must be deployed in advance or at the same time. A and B can be deployed on the different nodes.
    • A associates with on B, which indicates that if A is deployed in the cluster, B must be deployed at the same time. A and B must be deployed on the same node.
  • During cluster installation, only one pair of NameNode and Zkfc roles can be installed. When HDFS Federation of the HDFS service is configured and multiple pairs of NameNode and Zkfc roles are required, you need to add the other pairs of roles manually after the cluster is installed.
Table 3-6 Memory requirements and deployment principles of service roles
Service Name Role Name Minimum Memory Requirement Dependency Deployment Principle
OMSServer OMSServer 10 GB - Deploy OMServers on two management nodes in active/standby mode.
LdapServer SlapdServer 500 MB to 1 GB -

To optimize performance, deploy LS and KrbServer on the same host in all clusters.

  • Analysis cluster: Deploy LSs on two control nodes in active/standby mode.
  • Backup cluster: Deploy LSs on two control nodes as backup services of the analysis cluster.
KrbServer KerberosServer 3 MB
  • KrbServer depends on LdapServer.
  • KerberosServer associated with KerberosAdmin.
Deploy KerberosServers on two control nodes in load sharing mode.
KerberosAdmin 2 MB Deploy KerberosAdmins on two control nodes in load sharing mode.
ZooKeeper QP (quorumpeer) 1 GB - Deploy three QPs in each cluster on control nodes. If you want to add nodes, ensure that the quantity is an odd number.
HDFS NN (NameNode) 4 GB
  • NameNode associated with Zkfc
  • Depends on ZooKeeper.
Deploy NNs on two control nodes in active/standby mode.
Zkfc (ZooKeeper FailoverController) 1 GB Deploy zkfcs on two control nodes in active/standby mode.
JN (JournalNode) 4 GB Deploy at least three JNs on control nodes. Each node stores a copy of backup date. To reserve three or more copies of backup data, deploy multiple JNs on control nodes or data nodes, ensure that the quantity is an odd number.
DN (DataNode) 4 GB Deploy DNs on at least three nodes. You are advised to deploy this service role on the data node.
Yarn RM (ResourceManager) 2 GB Depends on HDFS and ZooKeeper. Deploy RMs on two control nodes in active/standby mode.
NM (NodeManager) 2 GB Deploy NMs on data nodes. The number of NMs must be consistent with the number of HDFS DataNodes.
MapReduce JHS (JobHistoryServer) 2 GB Depends on Yarn, HDFS, and ZooKeeper. Deploy one JHS in each cluster on control node.
DBService DBServer 512 MB - Deploy DBServers on two control nodes in active/standby mode.
Hue Hue 1 GB Depends on DBService. Deploy Hues on two control nodes in active/standby mode.
Loader LS (LoaderServer) 2 GB Depends on MapReduce, Yarn, DBService, HDFS, and ZooKeeper.
  • Deploy LSs on two nodes in active/standby mode.
  • Loader must be deployed on two random NodeManagers.
Spark SR (SparkResource) - Depends on Yarn, Hive, HDFS, MapReduce, ZooKeeper, and DBService. SR does not have the actual process and does not consume memory. It must be deployed on all data nodes in non-active/standby mode.
JH (JobHistory) 2 GB Deploy this service role on two control nodes in non-active/standby mode.
JS(JDBCServer) 2 GB Deploy JSs on two control nodes in active/standby mode.
Hive HS (HiveServer) 4 GB Depends on DBService, MapReduce, HDFS, Yarn, and ZooKeeper. Deploy at least two HSs on control nodes. Multiple HSs can be deployed on control nodes in load sharing mode.
MS (MetaStore) 2 GB Deploy at least two MSs on control nodes. Multiple MSs can be deployed on control nodes in load sharing mode.
WebHCat 2 GB Deploy at least one WebHCats on control nodes. Multiple WebHCats can be deployed on control nodes in load sharing mode.
HBase HM (HMaster) 1 GB Depends on HDFS, ZooKeeper, and Yarn. Deploy HMs on two control nodes in active/standby mode.
RS (RegionServer) 6 GB Deploy RSs on data nodes. The number of RSs must be consistent with the number of HDFS DataNodes.
TS (ThriftServer) 1 GB Deploy three TSs on control nodes for each cluster. If there is a long delay when a TS accesses HBase and the delay cannot meet user requirements, you can deploy multiple of TSs on control nodes or data nodes.
FTP-Server FTP-Server 1 GB Depends on HDFS and ZooKeeper. Each instance provides 16 concurrent channels by default. If more concurrent channels are required, you can deploy multiple instances. A maximum of eight instances can be deployed on a control or data node.
Flume Flume 1 GB Depends on HDFS and ZooKeeper. It is recommended that the service roles be deployed on nodes only. It is recommended that Flume and DataNode be deployed on the different nodes to prevent data unbalance.
MonitorServer 128 MB Deploy MonitorServers on two control nodes in non-active/standby mode.
Kafka Broker 1 GB Depends on ZooKeeper. Deploy at least two Brokers on data nodes. If the data volume generated each day exceeds 2 TB, you are advised to deploy multiple Brokers on data nodes.
Metadata MetadataServer 512 MB Depends on DBService. Deploy one Metadata for the entire cluster on control node.
Oozie Oozie 1 GB Depends on DBService, Yarn, HDFS, MapReduce and ZooKeeper. Deploy Oozie on two control nodes in active/standby mode.
Solr SolrServerN (N is an integer ranging from 1 to 5) 2 GB Depends on ZooKeeper.
NOTE:
  • If Solr data is stored in the HDFS, the Solr service also depends on HDFS.
  • Solr data is stored in HDFS preferentially, and three Solr instances are deployed on each node.
  • It is recommended that a node whose real-time index speed is greater than 2 MB/s be deployed on a local disk, five Solr instances be deployed on each node, and a disk be mounted to each Solr instance independently.
  • Compared with that of a local disk, the performance of storage HDFS is decreased by 30% to 50%.
A maximum of five SolrServer instances can be deployed on each node. It is recommended that three or more nodes be deployed and the instances be evenly distributed on each node.
  • When the number of data entries is smaller than 1 billion, it is recommended that three instances be deployed on data nodes.
  • When the number of data entries ranges from 1 billion to 2 billion, it is recommended that 8 to 12 instances be deployed on data nodes.
  • When the number of data entries is greater than 2 billion, it is recommended that the instances be deployed on three or more independent nodes (nodes that are independent from the nodes where DataNode is deployed) and five instances be deployed on each node.
SolrServerAdmin 2 GB Deploy SolrServerAdmin on two data nodes in non-active/standby mode.
HBaseIndexer 512 MB Depends on HBase, HDFS, and ZooKeeper. Deploy HBaseIndexer on each node where the SolrServerX instance is located.
SmallFS FGCServer 6 GB Depends on MapReduce, YARN, HDFS and ZooKeeper. Deploy FGCServer on two data nodes in active/standby mode.
Streaming Logviewer 256 MB - Plan based on Supervisor deployment and deploy LogViewer on each node where Supervisor is deployed.
Nimbus 1 GB Depends on ZooKeeper. Deploy this service role on two control nodes in active/standby mode. This service role is associated with UI.
UI 1 GB Depends on ZooKeeper. Deploy this service role on two control nodes. This service role is associated with Nimbus. (The association relationship means that on a node where Nimbus is deployed, UI is also deployed.)
Supervisor 1 GB - Deploy this service role on at least one control node or data node. If a large number of computing capabilities are need to be provided, you can deploy multiple Supervisor roles. It is recommended that the service roles be deployed on control nodes only. Supervisors manage Workers. The number of Workers and memory can be configured.
NOTE:
The number of Supervisors to deployed can be calculated using the following formula, where the number of Workers in each topology is planned by the customer and the number of Workers configured in each Supervisor is four by default.

Number of Supervisors = Number of topologies x Number of Workers in each topology/Number of Workers configured for each Supervisor

Redis Redis_1, Redis_2, Redis_3…… 1 GB Depends on DBService. In single-master mode, deploy Redis on at least one data node. Deploy a Redis cluster on at least three data nodes.
Translation
Download
Updated: 2019-04-10

Document ID: EDOC1000104139

Views: 5989

Downloads: 64

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next