incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Jungblut (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-431) MapReduce NG integration
Date Tue, 13 Sep 2011 08:56:09 GMT

    [ https://issues.apache.org/jira/browse/HAMA-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103466#comment-13103466
] 

Thomas Jungblut commented on HAMA-431:
--------------------------------------

{quote}
With polling, it seems that chances are the polling would not reach the agreement (there could
always have 1 process missing) in an unfortunate timing case. Also, as the processes increase
probably it would increase the loading for master to deal with polling tasks. 
{quote}

This is correct, but depends highly on the polling intervall. As far as I understood each
BSPJob gets its own Application Master. So there is no "Master-Machine" anymore like our BSPMaster
or Jobtracker. 

We have two options:
- fix the barrier sync with zookeeper and use it in the AM and Peers
- do the polling in Application Master

{quote}
In addition, my understanding is the integration with MRV2 would be just an additional support
so that MR job/ application can be submitted without rewriting to use hama for computation.

{quote}
That is right as well. I think we should make this a configuration based decision whether
YARN (or the URL) has been set or not.

> MapReduce NG integration
> ------------------------
>
>                 Key: HAMA-431
>                 URL: https://issues.apache.org/jira/browse/HAMA-431
>             Project: Hama
>          Issue Type: New Feature
>            Reporter: Thomas Jungblut
>            Assignee: Thomas Jungblut
>
> We should take a look at how to integrate Hama's BSP Engine to Hadoop's nextGen application
platform.
> Can be currently found in the 0.23 branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message