hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ChiaHung Lin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-431) MapReduce NG integration
Date Wed, 14 Sep 2011 11:45:09 GMT

    [ https://issues.apache.org/jira/browse/HAMA-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104430#comment-13104430

ChiaHung Lin commented on HAMA-431:


Regarding the missing processes, which we call stragglers in mapreduce, isn't the API such
that there should be no progress till all the processes perform the barrier sync?
Yes, in that case there would have no progress. However, it differs from the barrier sync
with zookeeper in that there could always have different stragglers do not poll each round
due to networking loading, etc. For instance, with time interval e.g. 1 secs each GroomServer
polls to check if he can proceed; unfortunately due to network congestion, master server always
receives parts of response (not response from all GroomServers). So the rate of barrier sync
with no progress probably could be higher than expected. Or we will have master to help coordinate
between stragglers, but this seems the tasks that should be handled by zookeeper service.
In addition, if it is going to have multiple masters, to replicate the poll information should
also be taken into account. 

I was just thinking some issues that maybe we need to consider beforehand if it is decided
to work toward this direction. Thanks Vinod, that inspires me a lot. 

> MapReduce NG integration
> ------------------------
>                 Key: HAMA-431
>                 URL: https://issues.apache.org/jira/browse/HAMA-431
>             Project: Hama
>          Issue Type: New Feature
>            Reporter: Thomas Jungblut
>            Assignee: Thomas Jungblut
> We should take a look at how to integrate Hama's BSP Engine to Hadoop's nextGen application
> Can be currently found in the 0.23 branch.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message