incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Jungblut (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-431) MapReduce NG integration
Date Tue, 04 Oct 2011 08:07:34 GMT

    [ https://issues.apache.org/jira/browse/HAMA-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119952#comment-13119952
] 

Thomas Jungblut commented on HAMA-431:
--------------------------------------

Yes that is correct.
But I don't see the improvement, if a task fails, the checkpointer also fails within the task.
If you seperate the checkpointer as a seperate process which guards several tasks, it can
fail the tasks it guards if the process is not working properly. Armstrong is just referring
to the need of redundancy to absorb failure, but with a single process which is guarding several
tasks you have introduced another point of failure which can have a lot more impact than a
single task which fails.

Each task attempt should write the checkpoints with its taskID, attemptID and superstep (as
name?) into HDFS so it can be restarted from outside.
That's just my opinion on that, but you're the fault-tolerance professional ;)

But I would leave this outside for now and we can open another issue that will add this. In
this issue we can talk about the benefits of another process.
                
> MapReduce NG integration
> ------------------------
>
>                 Key: HAMA-431
>                 URL: https://issues.apache.org/jira/browse/HAMA-431
>             Project: Hama
>          Issue Type: New Feature
>            Reporter: Thomas Jungblut
>            Assignee: Thomas Jungblut
>         Attachments: job_state.dot, task_phase.dot, task_state.dot
>
>
> We should take a look at how to integrate Hama's BSP Engine to Hadoop's nextGen application
platform.
> Can be currently found in the 0.23 branch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message