hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7417) Hadoop Management System (Umbrella)
Date Sat, 25 Jun 2011 01:02:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054787#comment-13054787

Eric Yang commented on HADOOP-7417:

A brief description of Hadoop Management System design:


h4. Setup

HMS Agent is a list of rpm packages which can be deployed as part of OS image through PXE
boot.  HMS Beacon is a daemon which runs on each zookeeper nodes to broadcast the location
of the zookeeper.  HMS Agent and controllers are standalone daemons, which resolve zookeeper
location through HMS Beacon (zeroconf).

h4. Operation

Operator can issue command through HMS client and pass through HMS controller REST API.  HMS
command is serialized into JSON messages and queued in Zookeeper.  Multiple HMS controllers
watch the command queue for commands.  When a command triggers the controller to execute,
HMS controllers compete to create a lock for the command, and corresponding cluster to execute
the command.  If locks are successfully created, the controller begin to translate the command
into a list of actions to perform on the managed nodes.  HMS controller watches for the status
queues and coordinate actions to perform on HMS agents.  HMS managed agents download software
through yum repository or bit torrent through peer exchange.  HMS agent reports installation
status and configuration status back to agent status queue for HMS controller to orchestrate
the cluster deployment.  Once, all actions are finalized, HMS controller store the deployment
command history in the cluster node.  

In the event of node failures (to be implemented), operator can re-image the defected node.
When the agent join back, HMS agent can send status to controller to replay the installation
and configuration history to recover.

h4. Monitoring Proposal

For large clusters deployment, monitoring setup could be complex.  HMS can simplify this by
orchestrate Hadoop 0.20.2+1 (append branch) + HBase 0.90.3 + Pig 0.8.1 + Chukwa 0.5 deployment
using the proposed RPM packages for HADOOP-6255, ZOOKEEPER-999, HBASE-3606, PIG-1857, CHUKWA

> Hadoop Management System (Umbrella)
> -----------------------------------
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
> The primary goal of Hadoop Management System is to build a component around management
and deployment of Hadoop related projects. This includes software installation, configuration,
application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message