incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-431) MapReduce NG integration
Date Tue, 13 Sep 2011 13:01:09 GMT

    [ https://issues.apache.org/jira/browse/HAMA-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103601#comment-13103601
] 

Vinod Kumar Vavilapalli commented on HAMA-431:
----------------------------------------------

ChiaHung,

bq. With polling, it seems that chances are the polling would not reach the agreement (there
could always have 1 process missing) in an unfortunate timing case. Also, as the processes
increase probably it would increase the loading for master to deal with polling tasks. 
Regarding the missing processes, which we call stragglers in mapreduce, isn't the API such
that there should be no progress till all the processes perform the barrier sync?
Regarding the load, even MR AM which uses a Hadoop RPC server has similar requirements, in
the order of ten's of thousands of tasks. That amount of scalability should be enough for
Hama's case also. And like Thomas mentioned, each BSPMaster is needed to serve the same job's
BSPPeers, so that should help too.

bq. In addition, my understanding is the integration with MRV2 would be just an additional
support so that MR job/ application can be submitted without rewriting to use hama for computation.
It is not clear to me. But if you are talking of the ability to run the current BSP jobs without
rewriting them, then yes, we will support API level compatibility.

> MapReduce NG integration
> ------------------------
>
>                 Key: HAMA-431
>                 URL: https://issues.apache.org/jira/browse/HAMA-431
>             Project: Hama
>          Issue Type: New Feature
>            Reporter: Thomas Jungblut
>            Assignee: Thomas Jungblut
>
> We should take a look at how to integrate Hama's BSP Engine to Hadoop's nextGen application
platform.
> Can be currently found in the 0.23 branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message