No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search


To have a better experience, please upgrade your IE browser.


FusionInsight HD 6.5.0 Product Description 02

Rate and give feedback :
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).


Data Block Colocation

In the offline data statistics collection scenario, Join is a commonly used computing function and is implemented in MapReduce as follows:

  1. The Map task sorts two-table records into Join keys and values, implements Hash partition by Join Key, and sends the data to different Reduce tasks for processing.
  2. Reduce tasks read data in the left table recursively in the nested loop mode and poll each line of the right table. If Join key values are identical, Join results are output.

The biggest problem of the preceding method is the great performance reduction of the Join operation. This is because that a large amount of network data transfer is required during transferring the data stored in different nodes from MAP to Reduce. Figure 4-14 shows this process.

Figure 4-14 Data transmission in the non-colocation scenario

Datasheets are stored in physical file system by HDFS block. Therefore, if two to-be-joined blocks are put into the same machine accordingly after they are partitioned with Join Key, you can obtain the results directly from Map Join in the local node without any data transfer in the Reduce process of the Join operation. This will greatly improve the performance.

Through this feature, the same distribution ID can be specified for FileA and FileB that need association and summary computing so that all blocks are distributed at the same location. Computing can be performed without cross-node data reading, which greatly improves the MR Join performance.

Figure 4-15 Data block distribution in colocation and non-colocation scenarios

Damaged Hard Disk Volume Configuration

In open-source release, if multiple data storage volumes are configured for DataNode, DataNode stops providing services by default when one volume is damaged. If the configuration item dfs.datanode.failed.volumes.tolerated is set to specify the number of damaged volumes that can be tolerated, DataNode continues to provide services when the number of damaged volumes does not exceed the threshold.

The value of dfs.datanode.failed.volumes.tolerated must be greater than or equal to 0. The default value is 0, as shown in Figure 4-16.

Figure 4-16 Item being set to 0

For example, three data storage volumes are mounted to a DataNode, and dfs.datanode.failed.volumes.tolerated is set to 1. In this case, if one data storage volume of DataNode is unavailable, this DataNode can still provide services, as shown in Figure 4-17.

Figure 4-17 Item being set to 1

This native configuration item has some defects. When the number of data storage volumes in each DataNode is inconsistent, you need to configure each DataNode independently instead of generating the unified configuration file for all nodes.

Assume that there are three DataNodes in the cluster. The first node has three data directories, the second node has four, and the third node has five. If you want to ensure that DataNode services are available when only one data directory is available, you need to perform the configuration as shown in Figure 4-18.

Figure 4-18 Attribute configuration before being strengthened

In the HDFS of FusionInsight version, this configuration item is strengthened, with a value option -1 being added. When this configuration item is set to -1, DataNode can provide services as long as there is one available data storage volume.

Figure 4-19 shows how to configure the configuration item.

Figure 4-19 Attribute configuration after being strengthened

HDFS Startup Acceleration

During the startup of NameNode in HDFS, the metadata file fsimage is loaded. The fsimage file exists Safemode to complete the startup until DataNode starts and reports certain percentage of data blocks. If the number of files stored on the HDFS reaches the million or billion level, the two processes are time-consuming and will lead to a long startup time of the NameNode. Therefore, the process of loading the matedate fsimage is optimized.

In the open-source HDFS, the fsimage file stores all types of metadata. Each type of metadata (such as the file metadata and folder metadata) is independently stored in a section. Sections are loaded in serial mode. If a large number of files and folders are stored on the HDFS, loading of sessions is time-consuming and will lead to a long HDFS startup time. In the HDFS of the Huawei, the NameNode divides each type of metadata by segments and stores the data in multiple sections when generating the fsimage file. When the NameNode starts, sections are loaded in parallel mode. This measure greatly reduces the startup time of the NameNode and therefore significant accelerates the HDFS startup.

Label-based Block Placement Policies

Users need to flexibly configure the node to store the HDFS data blocks based on the data features by configuring one label expression corresponding to the HDFS directory or file and each DataNode corresponding to one or more labels so that specific DataNodes are specified for data block storage. If the label-based data block placement policy is used for selecting the DataNode to store the specified files, the range of DataNodes will be selected based on the label expression of the files and appropriate nodes will be selected from the range of DataNodes.

  • Users can store the replica of data blocks to the nodes with different labels accordingly. For example, store two replicas of the data block in one file to the node labeled with L1, and store other replicas of the data block to the nodes labeled with L2.
  • Users can set the policy in case of block placement failure, for example, select a node from all nodes randomly.

As shown in Figure 4-20.

  • The data in /HBase is stored in A, B, and D.
  • The data in /Spark is stored in A, B, D, E, and F.
  • The data in /user is stored in C, D, and F.
  • The data in /user/shl is stored in A, E, and F.
    Figure 4-20 Example of label-based block placement policy

HDFS Auto Data Movement

Hadoop has been used for batch processing of immense data in a long time. The current HDFS model is used to fit the needs of batch processing applications very well because such applications focus more on throughput than delay.

However, as Hadoop is increasingly used for upper-layer applications that demand frequent random I/O access such as Hive and HBase, low latency disks such as solid state disk (SSD) are favored in delay-sensitive scenarios. To cater to the trend, HDFS supports a variety of storage types. Users can choose a storage type according to their needs.

Storage policies vary depending on how frequently data is used. For example, the data that is frequently used is marked as ALL_SSD or HOT, whereas the data that is rarely used is marked as COLD or even FROZEN.

Low latency disks are far more expensive than spinning disks. Data typically sees heavy initial usage with decline in usage over a period of time. Therefore, it can be useful if data that is no longer used is moved out from expensive disks to cheaper ones such as Disk/Archive.

A typical example is storage of detail records. New detail records are imported into SSD because they are frequently queried by upper-layer applications. As access to these detail records declines, they are moved to cheaper storage.

Before automatic data movement is achieved, you have to manually determine by service type whether data is frequently used, manually set a data storage policy, and manually trigger the HDFS Auto Data Movement Tool.

If aged data can be automatically identified and moved to cheaper storage such as Disk/Archive, users will see significant cost cuts and data management efficiency improvement.

The HDFS Auto Data Movement Tool is at the core of HDFS Auto Data Movement. It automatically sets a storage policy depending on how frequently data is used. Specifically, functions of the HDFS Auto Data Movement Tool include:

  • Mark a data storage policy as All_SSD, One_SSD, Hot, Warm, Cold, or FROZEN according to age, access time, and manual data movement rules.
  • Define the rules for distinguishing frequently used data from rarely used data based on data age, access time, and manual data movement rules.
  • Define the action to be taken if age-based rules are met.
    • MARK: the action for identifying whether data is frequently or rarely used and setting a data storage policy.
    • MOVE: the action for invoking the HDFS Auto Data Movement Tool and moving data across tiers.
    • SET_REPL: The action for setting new replication factor for a file.
    • MOVE_TO_FOLDER: The action for moving files to a target folder.
    • DELETE: The action for deleting a file or directory.
    • SET_NODE_LABEL: The action for setting the NodeLabel for a node.

With the HDFS Auto Data Movement feature, you only need to define age based on access time rules. HDFS Auto Data Movement Tool matches data according to age-based rules, sets storage policies, and moves data. In this way, data management efficiency and cluster resource usage are improved.

Updated: 2019-05-17

Document ID: EDOC1100074548

Views: 3039

Downloads: 35

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Previous Next