FusionInsight HD 6.5.1 Service Operation Guide 03

About This Document
DBService
- Enabling SSL Encrypted Transmission
- Disabling SSL Encrypted Transmission
- Log Description
Elasticsearch
- Elasticsearch Usage Suggestions
  - Service Planning Suggestions
  - Development Specifications
- Creating an Elasticsearch Role
- Switching the Elasticsearch Security Mode
- Curl Command Usage for Linux
- Integrated Third-Party SDK Plug-Ins
- Interface Authentication Whitelist Configuration
- Custom Data Directory
- Connecting Elasticsearch to Other Components
- Index Owner Group Synchronization
- Migrating Data
- Switching the Elasticsearch Local Access Authentication Mode
- Enabling the Elasticsearch Memory Circuit Breaking
- Introducing Logs
- FAQ
- Performance Optimization
Flink
- Flink Configuration Management
- Security Management
- User Management
- Log Description
- Performance Optimization
  - Optimization DataStream
- Reference
  - Shell Commands
Flume
- Overview
- Installing a Flume Client
- Flume Service Configuration Guide
- Using Environment Variables in the properties.properties File
- Non-Encrypted Transmission
- Encrypted Transmission
  - Configuring the Encrypted Transmission
  - Typical Scenario: Collecting Local Static Logs and Uploading Them to HDFS
- Viewing Flume Client Monitoring Information
- Stopping or Uninstalling the Flume Client
- Flume Client Cgroup Usage Guide
- Secondary Development Guide for Flume Third-Party Plug-ins
- Guide for Interconnection Between Flume and Kafka in Security Mode
- Guide to Connecting the Flume to the Security Hive
- Flume Service Model Configuration Guide
  - Overview
  - Service Model Configuration Guide
- Log Description
- FAQs
FTP-Server
- Using FTP to Access HDFS
- Using FTPS to Access HDFS
- Log Description
GraphBase
- Managing Users
- Multi-Graph Permission Control
  - Introduction
  - Creating a GraphBase Role
- Log Description
- Refined GraphBase Graph Monitoring
HBase
- Creating an HBase Role
- Using an HBase Client
- Configuring the HBase Disaster Recovery
- HBase DR Service Switchover
- Switchover of Active and Standby HBase DR Clusters
- (Official Website) BulkLoad Tool
- (Huawei Enhanced) BulkLoad Tool
- Configuring Connection Number Limit
- Configuring Secure HBase Replication
- Configuring Region In Transition Recovery Chore Service
- Configuring Region Double Assignment Recovery Chore Service
- Configuring Disable Table Regions Max Retry Attempts on Timeout
- Log Description
- Performance Optimization
- FAQ
HDFS
- Configuring Memory Management
- Creating an HDFS Role
- Using an HDFS Client
- Running the DistCp Command
- Overview of HDFS File System Directories
- Changing the DataNode Storage Directory
- Configuring HDFS Directory Permission
- Configuring NFS
- Planning HDFS Capacity
- Configuring HDFS Data Storage Policies
- Configuring ulimit for HBase and HDFS
- Configuring DataNode Capacity Balancing
- Configuring the Replica Replacement Policy When Disk Capacities Are Different on a DataNode
- Configuring the Replica Replacement Policy When the Disk Capacity of Each DataNode Is Different
- Configuring the Number of Files in a Single HDFS Directory
- Setting the Recycle Bin Mechanism
- Setting File and Directory Permissions
- Setting the Maximum Lifetime and Renewal Interval of a Token
- Configuring Failed Disk Volumes
- Configuring Encrypted Channels
- Reducing the Probability of Abnormal Client Application Operation When the Network Is Not Stable
- Setting NameNode Blacklist
- Setting Rack Group BPP for Data Storage
- Optimizing HDFS NameNode RPC QoS
- Optimizing HDFS DataNode RPC QoS
- Configuring the LZC Compression
- Configuring Reserved Percentage of DataNode Disk Usage
- Configuring HDFS NodeLabel
- Configuring HDFS Mover
- Configuring HDFS DiskBalancer
- Configuring Capacity Balancing in the Heterogeneous DataNode Scenario
- Configuring EC Storage on HDFS
- Configuring Observer NameNode to Process Read Requests
- Configuring DataNode Group
- Configuring Separated Deployment of JournalNodes in the Federation Scenario
- Using HDFS Files to Perform Concurrent Operations
- Using fastCopy
- Log Description
- Performance Optimization
  - Improving Write Performance
  - Optimizing JVM Parameters
  - Improving Read Performance Using Client Metadata Cache
  - Improving the ClientConnections to NameNode Using Current Active Cache
- FAQ
  - Why NameNode Startup Is Slow?
  - MapReduce Job Failed in Multiple NameService Environment
  - Why Cannot the Number of Data Blocks Be Reported When the DataNode Is Normal?
  - HDFS WebUI Cannot Properly Update Information About Damaged Data
  - Why Does the Distcp Command Fail in the Secure Cluster, Causing an Exception?
  - Why Does DataNode Fail to Start When the Number of Disks Specified by dfs.datanode.data.dir Equals dfs.datanode.failed.volumes.tolerated?
  - Why Does an Error Occur During DataNode Capacity Calculation When Multiple data.dir Are Configured in a Partition?
  - Why Does the Standby NameNode Fail to Restart Upon a Power Outage During Storage of Metadata?
  - Why Data in the Buffer Is Lost If a Power Outage Occurs During Storage of Small Files
  - Why Does Array Border-crossing Occur During FileInputFormat Split?
  - When LAZY_PERSIST Is Set as the Layered Storage Policy, Why the Storage Device Type of All Replicas Is DISK?
  - The HDFS Client Is Unresponsive When the NameNode Is Overloaded for a Long Time
  - Can I Delete or Modify the Data Storage Directory in DataNode?
  - Why NameNode UI Shows Missing Blocks After Successful Rollback
  - Why "java.net.SocketException: No buffer space available" Exception Occurs While Writing Data to HDFS?
  - Why Are There Two Standby NameNodes After I Restart the Active NameNode?
  - When Does a Balance Process in HDFS, Shut Down and Fail to be Executed Again?
  - "This page can't be displayed" Is Displayed When Internet Explorer Fails to Access the Native HDFS UI
  - NameNode Startup Failure Due to Inconsecutive EditLog Sequence Number
Hive
- Common Hive Parameters
- Permission Management
  - Hive Permission
  - Creating a Hive Role
  - Configuring Permission for Hive Tables, Columns, or Databases
  - Configuring the Rights to Use Other Components for Hive
- Using a Hive Client
- Using HDFS Colocation to Store Hive Tables
- Using the Hive Column Encryption Function
- Customizing Row Separators
- Deleting Single-row Records from Hive on HBase
- Configuring HTTPS/HTTP-based REST Interfaces
- Enabling or Disabling the Transform Function
- Access Control of a Dynamic Table View on Hive
- Specifying Whether Admin Rights Are Required for Creating Temporary Functions
- Using Hive to Read Data in a Relational Database
- Supporting Traditional Relational Database Syntax in Hive
- Switching Between Multiple Hive Instances in Hive Editor
- Creating User-defined Hive Functions
- Connecting Hive to Elasticsearch
- Enhancing beeline Reliability
- Viewing Table Structures Using the show create table Statement as Users with the select Permission
- Writing a Directory into Hive with the Old Data Removed to the Recycle Bin
- Enabling the Function of Creating a Subdirectory for the Directory With the 700 Permission Containing No Lower-Level Directories
- Creating Databases and Creating Tables in the Default Database Only as the Hive Administrator
- Disabling of Specifying the location Keyword When Creating an Internal Hive Table
- Enabling the Function of Creating a Foreign Table in a Directory that Can Only Be Read
- Authorizing Over 32 Roles in Hive
- Restricting the Maximum Number of maps for Hive Tasks
- HiveServer Lease Isolation
- Log Description
- Performance Optimization
  - Creating Table Partitions
  - Optimizing Join
  - Optimizing Group By
  - Optimizing Data Storage
  - Optimizing SQL Statements
  - Optimizing the Query Function Using Hive CBO
- FAQ
  - How to Delete UDFs on Multiple HiveServers at the Same Time
  - Drop Operation Cannot Be Performed for a Backed Up Hive Table
  - How to Perform Operations on Local Files with Hive User-Defined Functions
  - How to Forcibly Stop the MapReduce Tasks Executed by Hive
  - Table Creation Fails Because Hive Complex Fields' Names Contain Special Characters
  - How Do I Monitor Hive Table Size
  - How to Protect Key Directories to Prevent Data Loss Caused by Misoperations of the insert overwrite Statement?
  - Does Hive Support Transaction Enabling?
  - Disabling Freezing of Hive on Spark Tasks When HBase Is Not Installed
  - An Error Occurred when the WHERE Condition is Used to Query Tables that Exceed the 32,000 Partition
Hue
- Hue Common Parameters
- Hue WebUI
- Log Description
- FAQ
  - How to Resolve the Problem that HQL Statements Fail to Be Run In Hue When Internet Explorer Is Used?
  - Why Does the Entered use database Statement Become Invalid When Hive Is Used?
Kafka
- Creating a Kafka Role
- Kafka Common Parameters
- Viewing Kafka Topic Monitoring Information
- Managing Kafka User Permissions
- Safety Instruction on Using Kafka
- Kafka Specifications
- Using a Kafka Client
- Managing Kafka Topics
- Configuring Kafka HA and High Reliability Parameters
- Changing the Broker Storage Directory
- Checking the Consumption Status of Consumer Group
- Kafka Balancing Tool Instructions
- Kafka Token Authentication Mechanism Implemented Using a Script
- Log Description
- Performance Optimization
  - Kafka Performance Tuning
- FAQ
  - How to Resolve the Problem That a Kafka Topic Cannot Be Deleted
  - Kafka Nodes Startup Failure Due to Unexpected Cluster Power-off
Loader
- Loader Common Parameters
- Creating a Loader Role
- Importing Data
- Exporting Data
- Job Management
- Operator Help
- Client Tool Description
- Log Description
- FAQ
  - How to Resolve the Problem that Failed to Save Data When Using Internet Explorer 10 or Internet Explorer 11 ?
  - Differences Among Connectors Used During the Process of Importing Data from the Oracle Database to HDFS
Mapreduce
- Converting MapReduce from the Single Instance Mode to the HA Mode
- Configuring the Log Archive and Cleanup Mechanism
- Reducing Client Application Failure Rate
- Transmitting MR Tasks from Windows to Linux
- Using Distributed Cache
- Configuring MapReduce Shuffle Address
- Configuring the Cluster Administrator ACL
- Log Description
- Performance Optimization
- FAQ
Metadata
- Creating a Metadata Role
- Viewing Metadata
- Configuring Automatic Metadata Extraction
- Log Description
Oozie
- Overview
- Using an Oozie Client
- Configuring the Oozie Access Channel Protocol
- Using Oozie Client to Submit an Oozie Job
- Using Hue to Submit an Oozie Job
- Log Description
- FAQ
Redis
- Redis Common Parameters
- Creating a Redis Role
- Redis Cluster Management
- Redis Data Backup Tool
- Redis Data Restoration Tool
- Log Description
SmallFS
- Configuring SmallFS Management Frequency
- SmallFS Common Parameters
- Log Description
- FAQ
Solr
- Creating a Solr Role
- Using a Solr Client
- Common Service Operations
- Log Description
- Performance Optimization
- FAQ
Spark
- Using a Spark Client
- Installing SparkR
- Enabling the Spark On HBase Feature
- Enabling the Spark On Solr Feature
- Using Spark Fair Scheduler
- Modifying a JDBCServer Job Queue
- Submitting a Job to a Specified Yarn Queue by Running a spark-submit Command
- Unique Parameters in yarn-cluster Mode
- Scenario-Specific Configuration
- Advanced Parameters
  - Configuring Join Condition Transfer
- SparkSQL Permission Management(Security Mode)
- SQL-based Unified Search
- CarbonData
- Log Description
- Performance Optimization
- FAQ
Spark2x
- Basic Operation
- CarbonData User Guide
- Log Description
- Small File Combination Tool
- Performance Optimization
- FAQ
Storm
- Storm Common Parameters
- Storm WebUI
- Viewing worker Logs
- Using a Storm Client
- Submitting a Storm Topology
- Managing a Storm Topology
- Configuring a Storm Service User Password Policy
- Migrating Applications from Streaming to Storm
- Migrating Applications from Storm to Flink
- Log Description
- Performance Optimization
  - Storm Performance Tuning
Yarn
- Yarn Common Parameters
- Creating a Yarn Role
- Using a Yarn Client
- Configuring Resources for a NodeManager Role Instance
- Changing the NodeManager Storage Directory
- Configuring YARN Strict Permission Control
- Setting the Container Log Aggregation Function
- Using CGroups with YARN
- Configuring the Number of AM Reconfiguration Attempts
- Configure the AM to Automatically Adjust the Allocated Memory
- Configuring the Access Channel Protocol
- Configuring Memory Usage Detection
- Setting the Additional Scheduler WebUI
- Configuring YARN Restart
- Configuring AM Work Preserving
- Configuring the Localizer Log Level
- Configuring Users Who Run Tasks
- Configuring the Function of GPU Sharing by Percentage
- Log Description
- Performance Optimization
  - Task Preemption
  - Setting the Job Priority
  - Optimizing Node Configuration
  - Optimizing JVM Parameters
- FAQ
  - Why Mounted Directory for Container is Not Cleared After the Completion of the Job While Using CGroups?
  - Why the Job Fails with HDFS_DELEGATION_TOKEN Expired Exception?
  - Local Logs Cannot Be Deleted After the YARN Is Restarted
  - Why the Task Does Not Fail Even Though AppAttempts Restarts for More Than Two Times?
  - Why Is an Application Moved Back to the Original Queue After ResourceManager Restarts?
  - Why Does not YARN Release the Blacklist Even All Nodes Are Added to the Blacklist?
  - Why Does the Switchover of ResourceManager Occur Continuously?
  - Why Does a New Application Fail When a NodeManager Has Been in Unhealthy Status for 10 Minutes?
  - Why Does an Error Occur When I Query the ApplicationID of a Completed or Non-existing Application Using the RESTful APIs?
  - Why does the Fault of a Single NodeManager Cause MapReduce Task Failure in the Superior Scheduling Mode?
  - Why Are Applications Suspended After They Are Moved From Lost_and_Found Queue to Another Queue?
  - How to Limit the Size of App's Diagnostic Message Which Is Stored in the ZKstore?
  - Why Does the MR Job Fail When the Non-ViewFS File System Is Configured to ViewFS
ZooKeeper
- ZooKeeper Common Parameters
- Using a ZooKeeper Client
- Changing the ZooKeeper Storage Directory
- ZooKeeper Permission Configuration Guidance
- Setting Connection Control
- Setting Response Timeout
- Setting Bind IP for Client
- Setting Bind Port Range for Client
- Setting Client Specific Configuration in a Single JVM
- Setting ZooKeeper Server Principal
- Setting ACL Comparison to Use Short Name
- Setting a Quota for Znode
- Changing the ZooKeeper Transaction Log Directory
- Log Description
- FAQ
Unified SQL (Fiber)
- Configuring the Client
- Using the fiber-beeline Client
- Introduction to the fiber-beeline Command
- FAQ
  - Why Can I Still Switch to the Unavailable Spark JDBC Driver that Has Been Connected Before in fiber-beeline?
  - Why Does the "Permission denied" Exception Occur When I Use Spark JDBC Driver to Create a Temporary Table in fiber-beeline?
Federation
- Federation Overview
- Federation Configuration
- Components That Use Federation
- Federation Configuration Examples

Rate and give feedback:

Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).

About This Document

Purpose

This document describes how to operate each component in detail.

Intended Audience

This document is intended for O&M engineers.

Symbol Conventions

The symbols that may be found in this document are defined as follows.

Symbol	Description
	Indicates a hazard with a high level of risk which, if not avoided, will result in death or serious injury.
	Indicates a hazard with a medium level of risk which, if not avoided, could result in death or serious injury.
	Indicates a hazard with a low level of risk which, if not avoided, could result in minor or moderate injury.
	Indicates a potentially hazardous situation which, if not avoided, could result in equipment damage, data loss, performance deterioration, or unanticipated results. NOTICE is used to address practices not related to personal injury.
	Supplements the important information in the main text. NOTE is used to address information not related to personal injury, equipment damage, and environment deterioration.

Change History

Changes between document issues are cumulative. The latest document issue contains all the changes made in earlier issues.

Issue 03 (2020-01-15)

This issue is the third official release.

Issue 02 (2019-09-23)

This issue is the second official release.

Issue 01 (2019-07-15)

This issue is the first official release.

Download

Updated: 2020-01-15

Document ID: EDOC1100094223

Downloads: 57

Average rating:

This Document Applies to these Products

Reminder

Enterprise

Consumer

Corporate

Carrier

Africa

Latin America

Middle East

Asia Pacific

North America

Europe

About This Document

Purpose

Intended Audience

Symbol Conventions

Change History

Related Version

Related Documents

About Huawei

How to Buy

Partner

Resources

Others

Enterprise App