No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionInsight HD 6.5.0 Software Installation 02

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Generate the Configuration Files

Generate the Configuration Files

Configuration Files Required for the Node OS

Configuration

Configuration files required for the node OS configuration include preinstall.ini, partition configuration files of each node (hostN.ini), as shown in Table 7-2. Generate the required files manually according to the following code example, modify related parameters, and upload the modified files to specified position on each node.

NOTE:

If the preinstall does not need to be executed, all files in this topic do not need to be uploaded.

Table 7-2 List of configuration files

Configuration File Type

Configuration File

Usage

Server Installation Directory

OS configuration file

preinstall.ini

Configures the OS.

Generally, preinstall.ini is located on the management node (it is located on the active management node if there are active and standby management nodes). For example, the installation directory is:

/opt/FusionInsight_SetupTool/preinstall

Partition configuration file

hostN.ini(N=0, 1, 2, ...)

Configures the partition for each node.

Generally, hostN.ini is located on the management node (it is located on the active management node if there are active and standby management nodes). For example, the installation directory is:

/opt/FusionInsight_SetupTool/preinstall/partition/ini

OS configuration file

Parameters in the OS configuration file are described in Table 7-3.

Table 7-3 Parameter settings for preinstall.ini (11 nodes are deployed separately)

Parameter

Value Description

Example Value

g_hosts

Specifies the IP addresses of all servers (IP address of the management plane).

  • Use commas (,) to separate IP addresses, for example, 192.168.10.10,192.168.10.11.
  • Use hyphens (-) to indicate an IP address segment if the IP addresses are consecutive, for example, 192.168.10.[10-20].
  • Use hyphens (-) to indicate an IP address segment if the IP addresses are consecutive, and use commas (,) to separate IP address segments, for example, 192.168.10.[10-20,30-40].

192.168.10.[10-20]

g_user_name

Specifies the username for logging in to the server, which is set to root generally.

root

g_port

Specifies the SSH service port number, which is set to 22 generally.

22

g_parted

Specifies the partition formatting option.

  • 0: does not format the partition.
  • 1: formats the partition but not forcibly.
  • 2: formats the partition forcibly.
  • 3: does not format disks. Install the diskmgt process to monitor the disks.

2

g_parted_conf

Specifies the partition information about each server, of which the value is in the following format: Node IP address:Partition configuration file;Node IP address:Partition configuration file. (Use the IP address of the management plane as the node IP address.)

  • When the partition information about some nodes is the same, the format of the parameter value is Node IP address,Node IP address:Partition configuration file, for example, 192.168.10.10,192.168.10.11:host0.ini.
  • When the partition information about some nodes is the same and the IP addresses of the nodes are consecutive, use hyphens (-) to indicate an IP address segment if the IP addresses are consecutive in the parameter value, for example, 192.168.10.[20-30]:host10.ini.
  • When the partition information about some nodes is the same and the IP addresses of the nodes are consecutive in different IP address segments, use hyphens (-) to indicate an IP address segment if the IP addresses are consecutive, and use commas (,) to separate IP address segments in the parameter value, for example, 192.168.10.[20-30,40-50]:host10.ini.

192.168.10.10:host0.ini;192.168.10.11:host1.ini;192.168.10.12:host2.ini;192.168.10.13:host3.ini;192.168.10.14:host4.ini;192.168.10.15:host5.ini;192.168.10.16:host6.ini;192.168.10.17:host7.ini;192.168.10.18:host8.ini;192.168.10.19:host9.ini;192.168.10.20:host10.ini;

g_add_pkg

Specifies the option for installing the missing RPM packages.

  • 0: does not install the RPM package.
  • 1: installs the RPM package.

1

g_pkgs_dir

Specifies the source directory of the RPM package, for example:

  • redhat-6.4:/media/
  • suse-11.3:/media/

redhat-6.4:/media/

g_log_file

Specifies the log file generated during the pre-installation configuration, for example, /tmp/fi-preinstall.log.

/tmp/fi-preinstall.log

g_debug

Specifies the switch for the debug mode.

  • 0: disables the debug mode.
  • 1: enables the debug mode.

0

g_hostname_conf

The mapping between the management plane IP address, the service plane IP address, and the host name of the hosts.

The management plane IP address:The service plane IP address:The host name

192.168.10.10:192.168.20.10:host0;192.168.10.11:192.168.20.11:host1;192.168.10.12:192.168.20.12:host2;192.168.10.13:192.168.20.13:host3;192.168.10.14:192.168.20.14:host4;192.168.10.15:192.168.20.15:host5;192.168.10.16:192.168.20.16:host6;192.168.10.17:192.168.20.17:host7;192.168.10.18:192.168.20.18:host8;192.168.10.19:192.168.20.19:host9;192.168.10.20:192.168.20.20:host10;

g_swap_off

Disables the swap partition.

  • 0: The swap partition is not disabled.
  • 1: The swap partition is disabled.

1

g_wce_conf

Specifies whether to disable or enable the disk cache.

  • 0: The disk cache is disabled.
  • 1: The disk cache is not disabled.
NOTE:
  • If you have had the supercapacitor of the RAID controller card configured, enable the disk cache to improve disk performance.
  • In abnormal power-off scenarios, if the disk cache is enabled, data loss may occur. Therefore, exercise caution when enabling this function.

0

g_platform

Chip type

  • x86_64
  • aarch64

x86_64

Example of the parameter settings for preinstall.ini:

g_hosts="192.168.10.[10-20]" 
g_user_name="root" 
g_port=22 
g_parted=2
g_parted_conf="192.168.10.10:host0.ini;192.168.10.11:host1.ini;192.168.10.12:host2.ini;192.168.10.13:host3.ini; 
192.168.10.14:host4.ini;192.168.10.15:host5.ini;192.168.10.16:host6.ini;192.168.10.17:host7.ini; 
192.168.10.18:host8.ini;192.168.10.19:host9.ini;192.168.10.20:host10.ini;"
g_add_pkg=1
g_pkgs_dir="redhat-6.4:/media/"
g_log_file="/tmp/fi-preinstall.log"
g_debug=0
g_hostname_conf="192.168.10.10:192.168.10.10:;192.168.10.11:192.168.10.11:;192.168.10.12:192.168.10.12:;192.168.10.13:192.168.10.13:;192.168.10.14:192.168.10.14:;192.168.10.15:192.168.10.15:;192.168.10.16:192.168.10.16:;192.168.10.17:192.168.10.17:;192.168.10.18:192.168.10.18:;192.168.10.19:192.168.10.19:;192.168.10.20:192.168.10.20:;" 
g_swap_off=1
g_platform="x86_64"
Partition configuration file
Table 7-4 Parameter settings for the partition configuration file (hostN)

Parameter

Example Value

Value Description

mount

  • dbdata_om.conf
  • dbdata_service.conf
  • zookeeper.conf
  • journalnode.conf
  • namenode.conf
  • datanode1.conf

Specifies the configuration details of the partition to be mounted, of which the related file is preinstall/partition/ini-plugin/conf/*.conf.

required

y

Specifies whether this configuration item must be mounted.

  • y (yes): This configuration item must be mounted. If disks are insufficient or no disk meets the mount requirements, the pre-installation configuration fails.
  • n (no): This configuration item can be mounted or not. Whether this configuration item is mounted will not affect the pre-installation configuration.

care

y

Specifies whether this configuration item needs to be cared.

  • y: The disk management service will monitor this configuration item.
  • n: This configuration item needs to be mounted only during the pre-installation configuration. After that, the disk management service will not monitor this configuration item.

condition

n

  • *.sh: Specifies the disk requirements for the configuration item, for example, the disk must be a RAID 1 disk or the hard disks must meet the capacity requirements. The related file is preinstall/partition/ini-plugin/condition/*.sh.
  • n: Any available disk can meet the requirements.

the partition configuration file is similar to that shown in Table 7-5.

Table 7-5 Partition configuration file example (11 nodes are deployed separately)

Node

Partition Configuration File Example

  • host0.ini
  • host1.ini
#mount         required care condition
dbdata_om.conf y        y    dbdata_om_condition.sh #Each row indicates that a disk on this node is used to create partitions. In this example, partition /dev/sdb1 is created on /dev/sdb.       

host2.ini

#mount           required care condition 
zookeeper.conf   y        y    zookeeper_condition.sh #Each row indicates that a disk on this node is used to create partitions. In this example, partition /dev/sdb1 is created on /dev/sdb.
namenode.conf    y        y    namenode_condition.sh #Each row indicates that a disk on this node is used to create partitions. In this example, partition /dev/sdc1 is created on /dev/sdc.
journalnode.conf y        y    journalnode_condition.sh

host3.ini

#mount              required care condition 
dbdata_service.conf y        y    dbdata_om_condition.sh 
zookeeper.conf      y        y    zookeeper_condition.sh 
namenode.conf       y        y    namenode_condition.sh 
journalnode.conf    y        y    journalnode_condition.sh

host4.ini

#mount              required care condition 
dbdata_service.conf y        y    dbdata_om_condition.sh
zookeeper.conf      y        y    zookeeper_condition.sh
journalnode.conf    y        y    journalnode_condition.sh
  • host5.ini
  • host6.ini
  • host7.ini
  • host8.ini
  • host9.ini
  • host10.ini
#mount           required care condition 
datanode1.conf   y        y    n 
datanode2.conf   y        y    n 
datanode3.conf   y        y    n 
datanode4.conf   y        y    n 
datanode5.conf   y        y    n 
datanode6.conf   y        y    n
NOTE:

Determined by the number of hard disks on the node. The preceding items match sdb, sdc, sdd, sde, sdf, and sdg respectively.

Configuration File Required for the Manager

Installation

To install Manager on two management nodes, you need two configuration files for the active and standby management nodes respectively. To install a single-node Manager, you need only one configuration file for the management node, and you need to modify this file manually. Generate the required files manually according to the following code example, modify related parameters, and upload the modified files to specified position on each node.

Table 7-6 Configuration file required for the installation

Configuration File Type

Configuration File

Usage

Server Installation Directory

Manager installation configuration file

HostIP.ini

(Corresponds to the install.ini file in the /opt/FusionInsight_Manager/software directory generated by the installation software package)

Installs Manager.

Generally, HostIP.ini is located on the active and standby management nodes. For example, 192.168.10.10.ini is located on the active management node and 192.168.10.11.ini is located on the standby management node. For example, the installation directory is:

/opt/FusionInsight_Manager/software

Configuring HostIP.ini
Table 7-7 Parameter settings of the configuration file for the Manager installation

Parameter

Example Value

Value Description

ha_mode

double

Specifies the installation mode.

  • single: single-node installation
  • double: dual-node installation
  • local_ip1
  • local_ip2
  • local_ip3
  • local_ip4

192.168.10.10

When Manager is to be installed on two management nodes, set the parameters as follows:

  • local_ip1,local_ip2,local_ip3,local_ip4: Local IP address of the Manager HA channel (IP address of the management plane), ip2 to ip4 are not supported.
  • peer_ip1,peer_ip2,peer_ip3,peer_ip4: Peer IP address of the Manager HA channel (IP address of the management plane), ip2 to ip4 are not supported.
  • peer_ip1
  • peer_ip2
  • peer_ip3
  • peer_ip4

192.168.10.11

  • ws_float_ip
  • ws_float_ip_interface
  • ws_float_ip_netmask
  • ws_gateway
  • 10.10.100.201
  • eth0:WS
  • 255.255.0.0
  • 10.10.100.254

These parameters are mandatory when Manager is to be installed on two management nodes.

  • Specifies the external IP address.
    • This IP address is used to access FusionInsight Manager.
    • Generally, the external IP address uses the IP address of the external management network.
    • You can also set the external IP address to the IP address of the management plane or an IP address of the service plane.
    • Ensure that this IP address is unique and has not been used.
  • Specifies the bound interface.

    If the parameter value is eth0:WS, eth0 indicates the physical NIC that can provide services on the node. If this NIC is not enabled, change it to an enabled physical NIC on the node. The first 14 characters of NIC interface names cannot be the same.

  • Specifies the subnet mask.
  • Specifies the gateway.
NOTE:
  • If you are not sure whether the IP address is unique, run ping on the server in the same network segment to check whether the IP address has been used.
  • If you are not sure whether the interface is unique, run ifconfig on the server in the same network segment to check whether the interface has been used.
  • Run route on the server in the same network segment to query the subnet mask and the gateway of the network segment.
  • om_float_ip
  • om_float_ip_interface
  • om_float_ip_netmask
  • om_gateway
  • 192.168.10.200
  • eth1:OM
  • 255.255.255.0
  • 192.168.10.254

These parameters are mandatory when Manager is to be installed on two management nodes.

  • Specifies the internal IP address (IP address of the management plane).
  • Specifies the bound interface. The first 14 characters of NIC interface names cannot be the same.
  • Specifies the subnet mask.
  • Specifies the gateway.

ntp_server_ip

10.3.178.3,10.3.178.4

Specifies the IP address of the external Network Time Protocol (NTP) clock server. This parameter is optional.

NOTE:
  • This parameter is the unique entry to set the IP address of the NTP server. You must set this parameter if the NTP clock server is required.
  • If the NTP server is not specified, the active management node is specified as the NTP server by default.
  • If multiple NTP servers are configured, separate multiple IP addresses with commas (,), for example, 10.3.178.3,10.3.178.4.

om_mediator_ip

192.168.10.254

Specifies the third-party arbitration IP address, which must be an IP address that can be connected and is permanently valid. You are advised to use the gateway IP address (IP address of the management plane).

sso_ip

10.3.179.3

If Single Sign-on is adopted, set this parameter to the IP address of the third-party CAS server.

sso_port

7444

If Single Sign-on is adopted, set this parameter to the port number of the third-party CAS server.

bigdata_home

/opt/huawei/Bigdata

Software installation path.

bigdata_data_home

/srv/BigData

Path for storing data, including HDFS data and metadata of all components.

cluster_nodes_scale

11

Specifies the number of cluster nodes. It is used to adjust Manager-related process parameters. You can enter a value based on the actual node quantity.

The system will optimize the Manager configuration based on the number of cluster nodes. For details, see section "Optimizing Manager Configurations Based on the Number of Cluster Nodes" in Performance Optimization.

tls_protocol_min

tlsv1.1

Specifies the earliest version of the SSL or TLS protocol supported by LdapServer. If the value is set to tlsv1.1, LdapServer supports TLS version 1.1 and later.

Table 7-8 describes the default mapping relationship between the operating systems and this parameter. sslv3 indicates SSLv3 and later; tlsv1.0 indicates TLSv1.0 and later; tlsv1.1 indicates TLSv1.1 and later.

NOTICE:

If operating systems of the same type but different versions are installed in the cluster where you want to install FusionInsight HD, the tls_protocol_min parameter may have multiple values. In this case, the lowest value of tls_protocol_min prevails (sslv3 < tlsv1.0 < tlsv1.1).

For example, if a cluster has both SUSE 11.4 and SUSE 12.1, you must set tls_protocol_min to sslv3. Otherwise, after the system is installed, LdapClient cannot obtain data from LdapServer. As a result, the cluster is unavailable.

realm_parse_ip

10.10.10.10

Specifies the fixed mapped IP address for the domain name. The IP address is unreachable. Write realm_parse_ip hadoop.Kerberos.default_realm into the hosts file under /etc/ for enabling the DNS service in the cluster.

Log in to FusionInsight Manager and choose Cluster > Service > KrbServer > Configuration > All Configurations. Search for the default_realm parameter, and use its parameter value as the value of Kerberos.default_realm.

Example of the parameter settings for 192.168.10.10.ini:

[HA] 
ha_mode=double 
local_ip1=192.168.10.10 
local_ip2= 
local_ip3= 
local_ip4= 
peer_ip1=192.168.10.11 
peer_ip2= 
peer_ip3= 
peer_ip4= 
ws_float_ip=10.10.100.201 
ws_float_ip_interface=eth0:WS 
ws_float_ip_netmask=255.255.0.0 
ws_gateway=10.10.100.254 
om_float_ip=192.168.10.200 
om_float_ip_interface=eth1:OM 
om_float_ip_netmask=255.255.255.0 
om_gateway=192.168.10.254 
ntp_server_ip=10.3.178.3,10.3.178.4 
om_mediator_ip=192.168.10.254 
sso_ip=10.3.179.3 
sso_port=7444 
bigdata_home=/opt/huawei/Bigdata 
bigdata_data_home=/srv/BigData
cluster_nodes_scale=11
tls_protocol_min=tlsv1.1
realm_parse_ip=10.10.10.10
[/HA]
Table 7-8 Mapping between operating systems and TLS versions

Operating System

tls_protocol_min

SUSE 11.1

sslv3

SUSE 11.2

sslv3

SUSE 11.3

sslv3

SUSE 11.4

sslv3

SUSE 12

tlsv1.0

SUSE 12.1

tlsv1.1

SUSE 12.2

tlsv1.1

SUSE 12.3

tlsv1.1

Red Hat 6.4

sslv3

Red Hat 6.5

sslv3

Red Hat 6.6

tlsv1.0

Red Hat 6.7

tlsv1.1

Red Hat 6.8

tlsv1.1

Red Hat 6.9

tlsv1.1

Red Hat 7.0

sslv3

Red Hat 7.1

tlsv1.0

Red Hat 7.2

tlsv1.1

Red Hat 7.3

tlsv1.1

Red Hat 7.4

tlsv1.1

Red Hat 7.5

tlsv1.1

CentOS 6.4

sslv3

CentOS 6.5

sslv3

CentOS 6.6

tlsv1.0

CentOS 6.7

tlsv1.1

CentOS 6.8

tlsv1.1

CentOS 6.9

tlsv1.1

CentOS 7.0

sslv3

CentOS 7.1

tlsv1.0

CentOS 7.2

tlsv1.1

CentOS 7.3

tlsv1.1

CentOS 7.4

tlsv1.1

CentOS 7.5

tlsv1.1

EulerOS 2.1

tlsv1.1

EulerOS 2.2

tlsv1.1

EulerOS 2.3

tlsv1.1

EulerOS 2.5

tlsv1.1

NeoKylin-6.9

tlsv1.1

NeoKylin-7.2

tlsv1.1

NeoKylin-7.4

tlsv1.1

Oracle Linux 7.4

tlsv1.1

Oracle Linux 7.5

tlsv1.1

Oracle Linux 7.6

tlsv1.1

Parameter Settings for the Cluster

During cluster installation, parameters need to be set for some service roles to ensure the proper running of the cluster.

Table 7-9 Parameter settings for service roles during the installation

Service Name

Role Name

Parameter

Description

OMSServer

OMSServer

Use the default value.

LdapServer

SlapdServer

Use the default value.

Metadata

MetadataServer

Use the default value.

Flink

FlinkResource

Use the default value.

HBase

HM (HMaster)

Use the default value.

RS (RegionServer)

Use the default value.

TS (ThriftServer)

Use the default value.

Hive

HS (HiveServer)

Use the default value.

MS (MetaStore)

Use the default value.

WH (WebHCat)

Use the default value.

Flume

Flume

Use the default value.

MonitorServer

Use the default value.

FTP-Server

FTP-Server

  • ftp-enabled
  • ftp-server-ip
  • Options:
    • true (The ftp protocol is enabled.)
    • false (The ftp protocol is disabled. This is the default value.)
NOTE:

To ensure data security, it is recommended that the ftps protocol that provides higher security be used during data transmission.

  • Indicates the IP address that provides the FTP service on the node where the FTP-Server service is installed. The default value is the service plane IP address of the node. If the default value is to be changed, the user needs to add an IP address to the network port of the node. The user also needs to delete the added IP address when uninstalling the FTP-Server service.

Spark

SR (SparkResource)

Use the default value.

JH (JobHistory)

Use the default value.

JS (JDBCServer)

Use the default value.

Spark2x

SparkResource2x

Use the default value.

JobHistory2x

Use the default value.

JDBCServer2x

Use the default value.

Loader

LS (LoaderServer)

  • loader.float.ip
  • mapreduce.client.submit.file.replication
  • Specifies the floating IP address of loader.

    The floating IP address must be the unique and unused IP address in the same network segment(service plane).

  • The default value of this parameter is 10.
    • If the number of DataNodes deployed in the cluster is greater than or equal to 10, do not change the parameter value.
    • If the number of DataNodes deployed in the cluster is smaller than 10, set this parameter to the actual number of DataNodes in the cluster.

Hue

Hue

  • HUE_FLOAT_IP
  • hdfs_umask
  • Specifies the floating IP address of Hue. The floating IP address must be the unique and unused IP address in the same network segment(service plane).
  • The umask used when creating files and directories. Can be in octal or in symbolic. For example,
    • 022 (octal for u=rwx, g=r-x, o=r-x in symbolic)
    • u=rwx, g=rwx, o= (symbolic for 007 in octal)

      The default value is 077.

DBService

DBServer

  • dbservice.floatip
  • dbservice.mediator.ip

Set the following parameters when DBService is deployed. These parameters can ensure the successful installation of DBService.

  • Specifies the floating IP address of DBService.

    The floating IP address must be the unique and unused IP address in the same network segment(service plane).

  • Specifies the arbitration IP address of DBService.

    This IP address is used to check whether DBServer and the network segment are properly connected. You are advised to set it to the gateway address or another fixed and available IP address. This parameter cannot be set to the IP address or the floating IP address of the node where active/standby DBService is located.

Mapreduce

JHS (JobHistoryServer)

Use the default value.

Solr

SolrServerAdmin

  • SOLRHASERVER_FLOAT_IP

Specifies the floating IP address of SolrServerAdmin.

  • The floating IP address must be the unique and unused IP address in the same network segment(service plane).

SolrServerN (N is an integer ranging from 1 to 5)

HBaseIndexer

Use the default value.

Elasticsearch

EsMaster

Use the default value.

EsNode1-9

Use the default value.

Kafka

Broker

Use the default value.

Storm

Logviewer

Use the default value.

Nimbus

Use the default value.

UI

Use the default value.

Supervisor

Use the default value.

Redis

Redis_1, Redis_2……

REDIS_SECURITY_ENABLED

When Redis is deployed, configure the following parameters based on actual services:

Installation mode of the Redis component. Value:

  • true (security mode)
  • false (normal mode)
NOTE:

The security mode deteriorates the performance. You are advised to install the normal mode if there is no special requirement. (If Redis is accessed in normal mode, authentication or authorization is not required. Do not use Redis to store important data; otherwise, security risks may exist.)

Oozie

oozie

Use the default value.

Yarn

RM (ResourceManager)

Use the default value.

NM (NodeManager)

  • yarn.nodemanager.local-dirs
  • yarn.nodemanager.log-dirs
  • yarn.nodemanager.resource.cpu-vcores
  • yarn.nodemanager.resource.memory-mb
  • yarn.nodemanager.resource.percentage-physical-cpu-limit

Set the local directory, log directory, system resource usage (number of CPU cores and memory size) of NodeManager on each node based on the disk partition plan.

  • Specifies the directories in which intermediate results are stored. You can specify multiple directories to share the I/O loads on the disk.

    Determines where the localized files should store. The default value of this parameter is %{@auto.detect.datapart.nm.localdir}.If there is a data partition, generate a path list similar to /srv/BigData/hadoop/data1/nm/localdir,/srv/BigData/hadoop/data2/nm/localdir based on the data partition. If there is no data partition, generate the default path /srv/BigData/yarn/data1/nm/localdir. In addition to using the expression, you can also enter a complete path list, for example (e.g. /srv/BigData/yarn/data1/nm/localdir or /srv/BigData/yarn/data1/nm/localdir,/srv/BigData/yarn/data2/nm/localdir), then data will be stored in all named directories, typically on different devices. To balance the load of disk IO, you'd better provide several paths each corresponding to a individual disk. An application's localized file directory will be found in a relative path /usercache/%{user}/appcache/application_%{appid}. Individual container's work directories, called container_%{contid}, are subdirectories under this path. To add a new directory, such as /srv/BigData/yarn/data2/nm/localdir, delete files under /srv/BigData/yarn/data2/nm/localdir first. Then, grant the same read/write permissions as those for /srv/BigData/hadoop/data1/nm/localdir to /srv/BigData/hadoop/data1/nm/localdir, and modify /srv/BigData/yarn/data1/nm/localdir as /srv/BigData/yarn/data1/nm/localdir,/srv/BigData/yarn/data2/nm/localdir. You can add a directory but do not modify or delete existing directories. Otherwise, data of the node manager will be lost, and services will become unavailable.

    [Default] %{@auto.detect.datapart.nm.localdir}

    [Note] After the storage directory of the NodeManager role is changed, the storage directories of all NodeManager instances are changed. After the storage directory of a single NodeManager instance is changed, the storage directories of other NodeManager instances remain the same.

  • Specifies the directories in which logs are stored. Multiple directories can be specified.

    Determines where the container logs should store. The default value of this parameter is %{@auto.detect.datapart.nm.logs}. If there is a data partition, generate a path list similar to /srv/BigData/hadoop/data1/nm/containerlogs,/srv/BigData/hadoop/data2/nm/containerlogs based on the data partition. If there is no data partition, generate the default path /srv/BigData/yarn/data1/nm/containerlogs. In addition to using the expression, you can also enter a complete path list, for example (e.g. /srv/BigData/yarn/data1/nm/containerlogs or /srv/BigData/yarn/data1/nm/containerlogs,/srv/BigData/yarn/data2/nm/containerlogs), then data will be stored in all named directories, typically on different devices. To balance the load of disk IO,you'd better provide several paths each corresponding to a individual disk. An application's localized log directory will be found in a relative path /application_%{appid}.Individual container's log directories, called container_{$contid}, are subdirectories under this path.Each container directory contains the stderr, stdin, and syslog files generated by the container. To add a new directory, such as /srv/BigData/yarn/data2/nm/containerlogs, delete files under /srv/BigData/yarn/data2/nm/containerlogs first. Then, grant the same read/write permissions as those for /srv/BigData/yarn/data1/nm/containerlogs to /srv/BigData/yarn/data1/nm/containerlogs, and modify /srv/BigData/yarn/data1/nm/containerlogs as /srv/BigData/yarn/data1/nm/containerlogs,/srv/BigData/yarn/data2/nm/containerlogs. You can add a directory but do not modify or delete existing directories. Otherwise, data of the node manager will be lost, and services will become unavailable.

    [Default] %{@auto.detect.datapart.nm.logs}

    [Note] Exercise caution when changing this value. If the value is incorrect, the service may fail to start. After the storage directory of the DataNode role is changed, the storage directories of all DataNode instances are changed. After the storage directory of a single DataNode instance is changed, the storage directories of other DataNode instances remain the same.

  • Specifies the number of CPU cores allocated to container. It is recommended to set this parameter to 0.75 to 1 times the number of logical CPU cores.
  • Indicates the total physical memory capacity available for YARN on the node. The default value is 8192 MB. You are advised to set the parameter value to 75% to 90% of the node's total physical memory capacity. If the node has the permanent processes of other services, reduce this parameter value to reserve sufficient resources for the processes.
  • Specifies the percentage of the CPU resource allocated to container. It is recommended to specify 75% of the CPU resource.

HDFS

NN (NameNode)

  • dfs.namenode.name.dir
  • dfs.namenode.edits.dir

Specifies the directory for storing NameNode data. The default directory is /srv/BigData/namenode.

  • Change the storage directory based on the actual partitions. Otherwise, disk usage will be too high to support normal service operating. For example, change the directory to /srv/BigData/hadoop/dataN/dfs/name (N ≥ 1, directories are separated by commas), and set the owner of this directory to user omm. You need to set this parameter to a customized directory when customizing the installation directory.
  • You need to set this parameter to a customized directory when customizing the installation directory.

Zkfc (ZooKeeper FailoverController)

Use the default value.

JN (JournalNode)

dfs.journalnode.edits.dir

You need to set this parameter to a customized directory when customizing the installation directory.

DN (DataNode)

  • dfs.datanode.data.dir
  • dfs.datanode.data.dir.perm
  • Specifies the directory for storing DataNode data. The default directory is /srv/BigData/hadoop/data1/dn. This directory stores the largest amount of data.

    Change the storage directory based on the actual partitions. Otherwise, disk usage will be too high to support normal service operating. For example, change the directory to /srv/BigData/hadoop/dataN/dfs/data (N ≥ 1, directories are separated by commas), and set the owner of this directory to user omm. You need to set this parameter to a customized directory when customizing the installation directory.

  • Permission of dfs.datanode.data.dir. The default value is 700. After the value is changed, other users on the OS will have related permission of the directory, which brings security risks. You are advised to retain the default value.

Router

Router configuration is not required in non-Federation scenarios. For details, see section "Federation Configuration" in Service Operation Guide by choosing Federation.

HttpFS

Use the default value.

SmallFS

FGCServer

Use the default value.

ZooKeeper

QP (quorumpeer)

dataDir

You need to set this parameter to a customized directory when customizing the installation directory.

GC_OPTS

If the number of Solr service nodes exceeds 50, it is recommended that the GC_OPTS parameter of the Zookeeper quorumpeer instance be set to the following values: -Xmx be set to 8G and -XX:MaxDirectMemorySize be set to 2G.

KrbServer

KerberosServer

Use the default value.

KerberosAdmin

Use the default value.

GraphBase

GraphServer

loadbalancer.floatip

Specifies the floating IP address of GraphBase.

The floating IP address must be the unique and unused IP address in the same network segment (service plane).

LoadBalancer

Use the default value.

NOTE:

When the physical memory of the node where a service role locates is smaller than 24 GB, set GC_Profile of the service role to low.

Download
Updated: 2019-05-17

Document ID: EDOC1100074555

Views: 6645

Downloads: 6

Average rating:
This Document Applies to these Products
Related Version
Related Documents
Share
Previous Next