No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionInsight HD 6.5.0 Product Description 02

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Yarn

Yarn

Basic Concept

Overview

YARN is the resource management system of Hadoop 3.0. It is a general resource manage module that manages and schedules resources for applications. YARN can be used not only in the MapReduce framework, but also in other frameworks such as Tez, Spark, and Flink. YARN shares some similarities with Mesos and Torque.

In the earliest Hadoop MapReduce framework, the FIFO Scheduler is used to schedule jobs. FIFO indicates that jobs of all users are submitted to one queue, and then are executed according to their priorities and the sequence of the time when each job is submitted. There is only one job queue in Hadoop. Jobs are queued according to the time when they are submitted. A new job is added to the queue tail. After a job runs, the job at the head of the job queue is the next job to run. This scheduling policy is simple, easy to be implemented, and reduces burdens of the JobTracker. However, the urgency of jobs is not considered in this scheduling policy. In addition, this policy is unfavorable for the running of small jobs. Therefore, Superior YARN Scheduler and Capacity Scheduler, which are able to allocate resources, are developed.

Architecture

The YARN model consists of ResourceManager, ApplicationMaster, and NodeManager, as shown in Figure 2-112.

Figure 2-112 Apache YARN architecture

Table 2-25 describes the functions of each module shown in Figure 2-112.

Table 2-25 YARN modules

Module

Function

Client

Client of a YARN application. You can submit a task to ResourceManager and query the operating status of an application using the client.

ResourceManager (RM)

A component responsible for unified management and allocation of resources. It receives the resource report information from various nodes (NodeManager) and allocates resources to various applications based on the specified policy.

NodeManager (NM)

An agent on each node of YARN. It manages the computing nodes in the Hadoop cluster as follows:
  • Communicates with the ResourceManger.
  • Monitors the lifecycle management of Container.
  • Monitors resource (memory and CPU) usage of each Container.
  • Monitors the node health status.
  • Manages the auxiliary services used by the log and other applications.

ApplicationMaster (AM)

A component responsible for all tasks in the lifecycle of an application. It performs the following tasks:
  • Negotiates with the Scheduler of RM to obtain resources.
  • Further allocates the assigned resources to internal tasks (secondary resource allocation).
  • Communicates with the NM to start or stop tasks.
  • Monitors the running status of all tasks.
  • Applies for resources to restart a task if the task has failed.

Container

A resource abstraction in YARN. It encapsulates multi-dimensional resources (including only memory and CPU currently) on a certain node. When the AM applies for resources from the RM, the RM returns resources to the AM in a container. The YARN allocates one container for each task and the task can only use the resources encapsulated in the container.

In Yarn, resource schedulers organize resources through hierarchical queues. This ensures that resources are allocated and shared among queues, thereby improving the usage of cluster resources. The core resource allocation model of Superior YARN Scheduler is the same as that of Capacity Scheduler, as shown in the following figure.

A scheduler maintains the information of a cluster of queues. A user can submit applications to one or more queues. During each NM heartbeat, the scheduler selects a queue according to a specific scheduling rule, selects an application in the queue, and then allocates resources to the application. If resources fail to be allocated to an application due to the limit some parameters, the scheduler will select the next application. After the scheduler selects an application, the application may concern many resource requests. The scheduler first satisfies the requests for local resources, and then for resources on the same rack, and finally for resources from any machine.

Figure 2-113 Resource allocation model
Principle

The new Hadoop MapReduce framework is named MRv2 or YARN. The new Hadoop MapReduce framework consists of ResourceManager, ApplicationMaster, and NodeManager.

  • ResourceManager (RM): The RM is a global resource manager. It is responsible for resource management and allocation of the entire system. It is comprised of two components, Scheduler and Applications Manager. The Scheduler allocates system resources to all running applications based on the restrictions such as capacity and queue (for example, allocates a certain amount of resources for a queue and executes a specific number of jobs). The Scheduler allocates resources based on the demand of applications, with container being used as the resource allocation unit. A container is a dynamic resource allocation unit. It encapsulates the memory, CPU, disk, and network resources to restrict the volume of resources used by each task. A container is a pluggable component. Users can design new containers based on their demand. YARN provides multiple types of Scheduler, such as Fair Scheduler and Capacity Scheduler. The Applications Manager manages all applications in the system. It performs tasks such as submitting applications, negotiating with the Scheduler on the resources required to start the ApplicationMaster, and monitoring the running status of the ApplicationMaster and restarting it when it fails to work properly.
  • NodeManager (NM): The NM is the resource and task manager on each node. It periodically reports the resource usage of the local node and the running status of each Container to the RM. It also receives and processes requests, such as Container start and stop requests, sent from the AM.
  • ApplicationMaster (AM): The AM is responsible for all tasks in the lifecycle of an application. It performs the following tasks:
    • Negotiates with the Scheduler of RM to obtain resources.
    • Further allocates the assigned resources to internal tasks (secondary resource allocation).
    • Communicates with the NM to start or stop tasks.
    • Monitors the running status of all tasks. Applies for resources to restart a task if the task has failed.
Principle of Capacity Scheduler

Capacity Scheduler is a multi-tenant scheduler. The unit of scheduling is a queue. The minimum and maximum resources that can be used by each queue are set. In addition, the maximum resources that can be used by each user are set to prevent resource abuse. Surplus resources of a queue are temporarily shared with other queues.

Capacity Scheduler supports multiple queues. A certain number of resources are configured for each queue. Each queue adopts the FIFO scheduling policy. To prevent one user's applications from exclusively using the resources in a queue, Capacity Scheduler sets a limit on the number of resources used by jobs submitted by one user. During scheduling, the number of computing resources required for each queue is first calculated. Then the Capacity Scheduler selects the queue that requires the least resources. Capacity Scheduler supports the following features:

  • Capacity guarantee. As the administrator, you can set the lower and upper limits of resource usage for each queue. All applications submitted to this queue share the resources.
  • Flexibility. Temporarily, resource surplus of a queue can be shared with other queues. However, such resources must be released in case of new application submission to the queue. Such flexible resource allocation helps notably improve resource usage.
  • Multi-tenancy. A cluster can be shared by multiple users or applications. To avoid exclusive resource usage by a single application, user, or queue, the administrator can apply multiple constraints (for example, limit on concurrent tasks of a single application).
  • Security assurance. An ACL list is provided for each queue to strictly limit user access. You can specify the users who can view your application status or control the applications. As the administrator, you can specify a queue administrator and a cluster system administrator.
  • Dynamic update of configuration files. As the administrator, you can dynamically modify configuration parameters as required to achieve online cluster management.

HA Overview

Background

The ResourceManager is responsible for managing resources and scheduling tasks in a cluster. In the version earlier than Hadoop 2.4, the ResourceManager is the single point of failure (SPOF) in a YARN cluster. The High Availability feature adds redundancy in the form of an Active/Standby ResourceManager pair to remove this otherwise single point of failure.

Implementation
Figure 2-114 ResourceManager HA architecture

ResourceManager HA is realized through an Active/Standby architecture, as shown in Figure 2-114. The ResourceManager HA solution is similar to the HDFS HA solution. That is, at any time point, only one of the ResourceManagers is in Active state. When the Active ResourceManager fails, the failover can be triggered automatically or manually to switch Active/Standby states.

When automatic failover is not enabled, administrators must run the yarn rmadmin command in the command line interface (CLI) to manually switch one of the ResourceManagers to the Active state. Upon a planned maintenance event or a fault, they are expected to first switch the Active ResourceManager to the Standby state and the Standby ResourceManager to the Active state.

With automatic failover enabled, the ResourceManagers have an option to embed the Zookeeper-based ActiveStandbyElector to decide which ResourceManager should be the Active node. When the Active ResourceManager goes down or becomes unresponsive, another ResourceManager is automatically elected to be the Active node which then takes over.

When ResourceManagers in the cluster are deployed in HA mode, the configuration yarn-site.xml used by clients needs to list all the ResourceManagers. Clients (including ApplicationMasters and NodeManagers) try connecting to the ResourceManagers in a round-robin mode until they hit the Active ResourceManager. That is, the clients must provide the fault tolerance mechanism by themselves. If the Active ResourceManager goes down, clients continue the round-robin polling until they hit the "new" Active ResourceManager.

After the standby ResourceManager becomes active, upper-layer applications can recover to their status when the fault occurs. (For details, see ResourceManger Restart.) When ResourceManager Restart is enabled, the ResourceManager restarts, loads the status information on the previous active ResourceManager, and takes over container status information on all NodeManagers to continue to execute the running status. In this way, status information can be saved by periodically executing checkpoint operations, avoiding data loss. Ensure that both active and standby ResourceManagers can access the status information. Currently, three means are provided for sharing status information: FileSystemRMStateStore, LeveldbRMStateStore, and ZKRMStateStore. Among them, only ZKRMStateStore supports the Fencing mechanism. By default, FusionInsight uses ZooKeeper for sharing.

For more information about the YARN HA, see content at:

http://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

Relationship with Other Components

Relationship Between Spark and YARN

The Spark computing and scheduling can be implemented using YARN mode. Spark enjoys the computing resources provided by YARN clusters and runs tasks in a distributed way. Spark on YARN involves two modes: yarn-cluster and yarn-client.

  • yarn-cluster mode

    Figure 2-115 describes the operation framework.

    Figure 2-115 Spark on yarn-cluster operation framework

    Spark on yarn-cluster implementation process:

    1. The client generates the Application information, and then sends the information to ResourceManager.
    2. ResourceManager allocates a Container(ApplicationMaster) to Spark Application, and then starts Driver on the Container node.
    3. ApplicationMaster applies for resources from ResourceManager to run Executor.

      ResourceManager allocates a Container to ApplicationMaster. Then ApplicationMaster communicates with the related NodeManager, and starts Executor on the obtained Container. After Executor is started, it registers in Driver and applies for tasks.

    4. Driver allocates tasks to Executor.
    5. Executor executes tasks and reports the operating status to Driver.
  • yarn-client mode

    Figure 2-116 describes the operation framework.

    Figure 2-116 Spark on yarn-client operation framework

    Spark on yarn-client implementation process:

    NOTE:

    In yarn-client mode, the Driver is deployed and run on the Client. In yarn-client mode, is not compatible with the old version of the client. Recommend the use of yarn-cluster mode.

    1. The client sends the Spark application request to ResourceManager, and ResourceManager returns the results. The results include information such as ApplicationId and the upper limit as well as lower limit of available resources. The client packages all information required to start ApplicationMaster, and sends the information to ResourceManager.
    2. After receiving the request, ResourceManager finds a proper node for ApplicationMaster and starts it on this node. ApplicationMaster is a role in YARN, and the process name in Spark is ExecutorLauncher.
    3. Based on the resource requirements of each task, ApplicationMaster can apply for a series of Containers to run tasks from ResourceManager.
    4. After receiving the newly allocated Container list (from ResourceManager), ApplicationMaster sends information to the related NodeManager to start the Container.

      ResourceManager allocates a Container to ApplicationMaster. Then ApplicationMaster communicates with the related NodeManager, and starts Executor on the obtained Container. After Executor is started, it registers in Driver and applies for tasks.

      NOTE:

      The running containers will not be hung to release resources.

    5. Driver allocates tasks to Executor. Executor executes tasks and reports the operating status to Driver.
Relationship Between MapReduce and YARN

MapReduce is a computing framework running on the YARN, which is used for batch processing. MRv1 is implemented based on MapReduce in Hadoop 1.0, which is composed of programming models (new and old programming interfaces), running environment (JobTracker and TaskTracker), and data processing engines (MapTask and ReduceTask). This framework is still weak in extendability, error tolerance (JobTracker single fault), and compatibility with multiple frameworks. (Currently, only the Mapreduce computing framework is supported.) MRv2 is implemented based on MapReduce in Hadoop 2.0. The source code reuses MRv1 programming models and data processing engine implementation, and the running environment is composed of ResourceManager and ApplicationMaster. ResourceManager is a brand new resource manager system, and ApplicationMaster is responsible for cutting MapReduce job data, assigning tasks, applying for resources, scheduling tasks, and tolerating errors.

Relationship Between ZooKeeper and YARN

Figure 2-117 describes the relationship between ZooKeeper and YARN.

Figure 2-117 Relationship between ZooKeeper and YARN
  1. ResourceManager tries to write the election information into ZooKeeper when the system starts. The ResourceManager which writes the information into ZooKeeper successfully first is elected as the active ResourceManager. The other one is elected as the standby ResourceManager. The standby ResourceManager monitors the election information of the active ResourceManager through ZooKeeper at regular time.
  2. The active ResourceManager creates a Statestore directory in ZooKeeper to store Application information. The standby ResourceManager obtains Application information from the Statestore directory and restores the data when the active ResourceManager is faulty.
Relationship Between SmallFS and YARN

SmallFS will periodically run merge, delete, and cleanup tasks. Suck tasks are MapReduce tasks running on YARN to merge, delete, and clean up data in the HDFS.

Download
Updated: 2019-05-17

Document ID: EDOC1100074548

Views: 4059

Downloads: 37

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next