hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyunsik Choi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HAMA-251) Add job manager to BSPMaster
Date Sat, 08 May 2010 07:56:47 GMT

    [ https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865430#action_12865430
] 

Hyunsik Choi commented on HAMA-251:
-----------------------------------

Before I complete the UML diagram and the design of the job manager, I would like to discuss
my plan with you. Overall, the job management system of BSP is resemble to that of MR. The
below summary includes not only a job management but also an overview of bsp cluster. It is
not completed yet, and I'll update this summary later.
----
Summary: 

One BSP cluster consists of one BSPMaster, multiple GroomServer and one or more zookeeper
servers in a network environment. Like JobTracker of MR, the bsp master plays the following
roles:
 * Maintaining groom server status.
 * Controlling supersteps in a cluster.
 * Maintaining job progress information.
 * Assining tasks to groom servers.
 * Job Scheduing
 * Controlling fault.
 * Providing users with the cluster control interface.

Like TaskTracker of MR, a Groom Server (shortly referred to as groom) is a process that performs
bsp tasks assigned by BSPMaster. Each groom contacts the BSPMaster, and it takes assigned
tasks and reports its status by means of periodical piggybacks with BSPMaster. Each groom
is designed to run with HDFS or other distributed storages. Basically, a groom server and
a data node should be run on one physical node.

A BSP Master and multiple grooms are started by the script. Then, the bsp master starts up
with a RPC server for groom servers. Groom servers starts up with a BSPPeer instance - later,
BSPPeer needs to be integrated with GroomServer - and a RPC proxy to contact the bsp master.
After started, each groom periodically sends a heartbeat message that encloses its groom server
status, including maximum task capacity, unused memory, and so on.

Each time the bsp master receives a heartbeat message, it brings up-to-date groom server status
- the bsp master makes use of groom servers' status in order to effectively assign tasks to
idle groom servers - and returns a heartbeat response that contains assigned tasks and others
actions that a groom server has to do. For now, I'll develop a FIFO job scheduler and very
simple task assignment algorithms.

> Add job manager to BSPMaster
> ----------------------------
>
>                 Key: HAMA-251
>                 URL: https://issues.apache.org/jira/browse/HAMA-251
>             Project: Hama
>          Issue Type: Sub-task
>    Affects Versions: 0.2.0
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>             Fix For: 0.2.0
>
>
> In regard to BSPMaster, I plan to design as follows:
>  * BSPMaster is responsible to manage jobs submitted by clients.
>  * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
>  * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of
BSPMaster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message