hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Filipe Manana (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HAMA-282) Add superstep counter
Date Mon, 08 Nov 2010 21:44:22 GMT

    [ https://issues.apache.org/jira/browse/HAMA-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929751#action_12929751

Filipe Manana commented on HAMA-282:

Hi Edward,

Thanks for looking into it.
Yes, the failling test went unnoticed before the addition of the GroboTestingJUnit library.

About the line:

superstepCounter -= tip.getTaskStatus(status.getTaskId()).getSuperstepCount();

My reasoning was the following:

A job in progress has one or more tasks (1 task per peer/groom server), which means the job's
superstep count matches the sum of all its tasks' superstep counts. Right?
TaskInProgress seems to represent more than one task (the current task attempt and previous
(failed) attempts, like Hadoop MapReduce). So I thought:

In getSuperstepCounter() of JobInProgress, iterate over the list of TaskInProgress, get the
TaskStatus of each TaskInProgress, get the superstep counter from each TaskStatus and sum
them all.
The problem with this is that in order to get the TaskStatus of a TaskInProgress, I need to
supply the TaskStatus ID, which I only seem to know when JobInProgress.updateTaskStatus()
is invoked.
Also, I thought that to update the superstep count of the job, I need to decrement it by the
number of supersteps of the previous TaskStatus and then increment it with the number of supersteps
in the new TaskStatus - an alternative would be to set to zero the superstep count of each
TaskStatus (in GroomServer / Peer) after each heartbeat.

Also, in your approach, you do:

  public void updateTaskStatus(TaskInProgress tip, TaskStatus status) {
    tip.updateStatus(status); // update tip
    superstepCounter = status.getSuperstepCount();

Like this, I think the superstep count of the job will be the superstep count of the last
completed task.
For e.g., if you have 2 tasks for a job (each one running on a different groom/peer), and
each one has 2 supersteps done when they finish, the job's superstep count should be 2+2=4
With that approach (your suggestion) I think it will end up as 2.
But correct me if I am wrong :)

> Add superstep counter
> ---------------------
>                 Key: HAMA-282
>                 URL: https://issues.apache.org/jira/browse/HAMA-282
>             Project: Hama
>          Issue Type: New Feature
>          Components: bsp
>    Affects Versions: 0.2.0
>            Reporter: Edward J. Yoon
>            Assignee: Filipe Manana
>             Fix For: 0.2.0
>         Attachments: hama-282.patch, HAMA-282_v02.patch, hama-282_v03.patch, hama-282_v04.patch

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message