incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyunsik Choi (JIRA)" <>
Subject [jira] Commented: (HAMA-251) Add job manager to BSPMaster
Date Sat, 15 May 2010 03:22:42 GMT


Hyunsik Choi commented on HAMA-251:

1) How to programming BSP with HAMA to process data on HDFS?

A task is paired with an input split. The input split is a HDFS block. The more HDFS block
is close to the reader, the more network cost is reduced. Thus, the scheduler should assign
as many tasks paired with local block as possible. For this, BSPMaster needs to hold network
topology and gets groom servers (with a data node) in which certain DFS blocks reside during
assigning tasks to groom servers. This explain is scope of this issue.

2) How to configure its BSP job?

In my progress, job configuration is performed in BSPJob and JobContext. BSPjob is a subclass
of JobContext. JobContext has a Configuration instance. Like new MapReduce API, BSPJob is
a writable job instance, and JobContext is readonly view. Parameters that users set are contained
in the Configuration instance in JobContext. As you know, A Configuration instance can be
written into a xml file, and it can be restored to a Configuration instance.

You can check the writeXml() in Configuration, BSPJob and JobContext.

3) How to submit Job through client interface?

Do you mean command line interface as the below comment?

If so, it is somewhat trivial. If we have internal implementation, we can make any type interfaces.
In addition, I made a really simple job submission interface for test as you can see BSPJob
and BSPExample. Right now, it is enough to test job submission.

If you mean how jobs is disseminated, the job submission is through In order
to submit job, a user firstly needs to set some parameters (e.g., Input Path, Output Path,
and class names for java reflection) to a BSPJob instance; my progress supports only to set
a class name for BSP computation. When the user submit the BSPJob instance, BSPJob contacts
the BSPMaster, and then it get new job id from BSPMaster. The BSPJob packs configurations
with JobID to a xml file and copies it to BSPMaster's local disk via DFS remote copy. A Jar
file that contains the job class is also copied to BSPMaster's local disk. 

Upon receiving the submitted job, BSPMaster adds the job to a simple scheduler. The scheduler
assigns tasks to groom servers via responses of heartbeat. So far, I have designed that each
groom server has only one task. For optimal scheduling, BSPMaster needs status of groom servers.
The status of groom servers have to include the max task capacity and the number of running
task for each groom servers. For this, BSPMaster collects the statuses of all groom servers
that join the BSP cluster through heartbeat. Finally, the task information is transmitted
to groom servers via the response of heartbeat.

You can check the heartbeat(), submitJob() in BSPJob, and submitJobInternal() in JobClient.

4) Then, Who manage the Job execution? 

In overall, BSPMaster controls all groom servers. BSPMaster sends a lunch action to groom
servers through responses of heartbeat. Then, each groom runs BSPPeer with a job class. During
processing job, each groom sends status of a running task to BSPMaster via heartbeat. BSPMaster
keeps status of all running tasks in a map data structure. Later, this information can be
provided to users via any kind interfaces, such as terminal and web. If a user kills certain
job, the user can send a kill action via BSPJob in the same way. These communications are
performed with Hadoop RPC. The patch already contains some of these interaction.

You can check the heartbeat () in BSPMaster and transmitHeartBeat in GroomServer.

> Add job manager to BSPMaster
> ----------------------------
>                 Key: HAMA-251
>                 URL:
>             Project: Hama
>          Issue Type: New Feature
>          Components: bsp
>    Affects Versions: 0.2.0
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>             Fix For: 0.2.0
>         Attachments: HAMA-251_01.patch
> In regard to BSPMaster, I plan to design as follows:
>  * BSPMaster is responsible to manage jobs submitted by clients.
>  * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
>  * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above features of

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message