No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionInsight HD 6.5.0 Product Description 02

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Support for DA200 Data Compression Card

Support for DA200 Data Compression Card

Introduction

FusionInsight HD allows the DA200 data compression card to be used on x86 servers. The DA200 data compression card compresses structured data and unstructured text data in Gzip compression mode to improve the compression ratio and reduce disk usage.

The Gzip compression/decompression mode provides a higher compression ratio, higher compression rate, and higher-density storage than the traditional Snappy compression mode. This helps release CPU resources in the compression/decompression scenario.

NOTE:

The DA200 compression card does not support pictures, and audio and video data.

Applications

Choose Support > Carrier IT > Carrier Intelligent Computing > FusionCube > FusionCube BigData Machine,download the driver of the DA200 compression card, and use either of thefollowing methods to deploy the driver:

  • Installingthe DA200 compression card > Loading the DA200 compression card driver >Restarting the operating system > Installing the FusionInsight HD clusterand services
  • Installingthe DA200 compression card > Installing the FusionInsight HD cluster andservices > Loading the DA200 compression card driver > Restarting theoperating system
NOTE:

If a compression card is added during cluster use, restart the FusionInsight HD operating system after the DA200 compression card driver is installed. This helps enable the hard compression function.

To restart the operating system, log in to the active management node as user root and run the reboot command to restart the operating system.

In Gzip compression mode, the compression capability of the DA200 compression card is invoked to compress/decompress structured and unstructured text data during data write on HDFS. Table 4-8 lists the support of each service for the DA200 data compression card.

Table 4-8 Support for the DA200 compression card

Service Name

Support for DA200 Data Compression Card

Compression Library Path

Description

Manager

N/A

-

FusionInsight Manager is not involved. When the DA200 data compression card is used, the status display, model query, and alarm management functions of the DA200 data compression card must be provided.

Yarn

Yes

zlib path of the ./hadoop checknative result

-

HDFS

Yes

zlib path of the ./hadoop checknative result

-

MapReduce

Yes

zlib path of the ./hadoop checknative result

-

HBase

Yes

/opt/client/HBase/hbase/lib/native/

-

Hive

Yes

{BIGDATA_HOME}/FusionInsight_6.5.0/FusionInsight-Hive-3.1.0/hadoop/lib/native/

The Gzip compression algorithm for the ORC file format of Hive does not use the compression library in the system or Hadoop native directory. Therefore, the DA200 compression card is not supported (ORC Java library path: /jre/lib/amd64/libzip.so).

Spark/Spark2

Yes

zlib path of the ./hadoop checknative result

-

Oozie

No

-

-

Flink

No

-

-

Zookeeper

N/A

-

-

Kafka

N/A

-

-

Elasticsearch

N/A

-

-

Flume

No

-

-

FTP-Sever

No

-

-

Hue

N/A

-

-

Loader

N/A

-

-

Redis

No

-

-

Solr

No

-

-

Storm

No

-

-

Constraints

  • Only x86 clusters support the DA200 data compression card (Gzip).
    NOTE:

    In x86 and ARM hybrid deployment scenarios, the DA200 data compression card is not supported.

  • For the HDFS, MapReduce, Hive, HBase, Yarn, and Spark components deployed in the same cluster on the x86 server, the same type of nodes must use the same compression policy.

    For example: All HDFS nodes use the unified soft compression or DA200 data compression card. If the DA200 data compression card (Gzip) is selected, the DA200 compression must be installed.

Download
Updated: 2019-05-17

Document ID: EDOC1100074548

Views: 4058

Downloads: 37

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next