incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Jungblut (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-431) MapReduce NG integration
Date Tue, 06 Sep 2011 18:35:10 GMT

    [ https://issues.apache.org/jira/browse/HAMA-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098250#comment-13098250
] 

Thomas Jungblut commented on HAMA-431:
--------------------------------------

Sorry for my late feedback and thank you for your help and information.

Currently our codebase has a majority of the "old" architecture of Hadoop. We changed parts
of the computation model, but task execution and job livecycle stays the same as it was in
the "old" Hadoop architecture. We put a synchronization service on top of it which is working
(most of the time it is not working:P) with zookeeper. In addition we have RPC connections
between the servers to message each other.

I suggest to implement our BSPMaster as an "Application Master". He must take care of allocating
new Containers which then will be a "Groom" in our namespace. Each groom needs a ZNode and
some kind of identifier.

But there is a question of security in my mind. Do you mind when we don't care about security
in the first version? I'm not an expert in these authentication systems like Kerberos.

So everything is actually implemented some way, but we need to port this code to YARN. I have
alot of time tomorrow so I just start. I also think we are going to split this task up to
several smaller pieces, so our other developers can contribute to it, too.

But I have a more general question:
Should we make this task part of our framework? Like another maven module which can be plugged-in
into Hadoop?

> MapReduce NG integration
> ------------------------
>
>                 Key: HAMA-431
>                 URL: https://issues.apache.org/jira/browse/HAMA-431
>             Project: Hama
>          Issue Type: New Feature
>            Reporter: Thomas Jungblut
>            Assignee: Thomas Jungblut
>
> We should take a look at how to integrate Hama's BSP Engine to Hadoop's nextGen application
platform.
> Can be currently found in the 0.23 branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message