hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Fenchel <jfen...@gmail.com>
Subject Re: [jira] [Commented] (HAMA-726) Hama on Mesos
Date Sun, 09 Feb 2014 05:52:25 GMT
Dave: Thanks for the mesos pointers.


I am not sure how far you intend to go with HAMA-848, but here are my

Much of the code between YARN and Mesos is not going to be reusable; they
each have their own method for doing RPC and claiming resources. However, I
do feel there is an opportunity for us to work together in turning the use
of these frameworks from special job types to configuration parameters, if
that is something in which you are interested.

Along that thought, I have attempted to make the "mesos option"
interoperable with a given TaskScheduler and TaskAllocationStrategy. To do
this I have done the following:

1) Added "getPreferredGrooms(TaskInProgress task, ...)" to JobInProgress.
The idea here is to be able to ask the job, if you could run anywhere where
would you prefer to be.

2) Added a "TaskWorkerManager" interface. The main idea here is to have a
class that is capable of creating TaskWorkers which are modeled off of the
TaskWorker in SimpleTaskScheduler. When call is executed on a taskWorker
the given framework should do its best to execute the jobInProgress. This
is where I dropped in and separated out a SimpleTaskWorkerManager and am
now creating a MesosTaskWorkerManager. If this is something you agree with,
we could add a YARNTaskWorkerManager too.

Things that I am presently not sure about and may cause issues in the above
1) What does job recovery look like under a Mesos or YARN framework?
2) To what extent do we want to be able to consider available resources
from within TaskAllocationStrategy. There appears to have been some effort
to do, but the current SimpleTaskWorker fails to provide resources beyond
just the available grooms and the current allocation strategy ignores
everything but the available grooms. It becomes difficult to answer this
question reasonably for both frameworks; while in yarn we can request what
resources we want, in mesos, we must accept or reject resource offers and
consequently we may not have a full picture of the available resources on
the cluster.

Anyway, sorry for the long winded reply. This is my current picture of what
should happen. However, I am not attached to any of it, am open to new and
brighter ideas, and would love criticism if you(or any others) have any
thoughts on this. I have attached a patch with the above changes to make
the framework compatible with various TaskWorkerManagers. However, the
mesos portion is still a work in progress. Additionally, at this point, I
make no guarantees on the patch's sanity aside from it compiling(which it
does/should compile). I am just including so you can get a better picture
of where I am going and, if you are interested, jump on bored with this

-Jeff Fenchel

On Sat, Feb 8, 2014 at 9:00 PM, Skater Xu (JIRA) <jira@apache.org> wrote:

>     [
> https://issues.apache.org/jira/browse/HAMA-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13895822#comment-13895822]
> Skater Xu commented on HAMA-726:
> --------------------------------
> Hello Jeff:
>          I am working on https://issues.apache.org/jira/browse/HAMA-848,
> (I am doing it in my spare time).
>          I have read some mesos docs before, I am think whether we can
> share some thinking to have mesos and yarn integration share sth?
> Regards,
> Skater
> > Hama on Mesos
> > -------------
> >
> >                 Key: HAMA-726
> >                 URL: https://issues.apache.org/jira/browse/HAMA-726
> >             Project: Hama
> >          Issue Type: New Feature
> >          Components: bsp core
> >            Reporter: Suraj Menon
> >            Assignee: Jeff Fenchel
> >
> > Since we are going to refactor our code to work on YARN, it would be
> nice to keep our abstraction levels to make it feasible to work on Mesos.
> > Documentation: http://incubator.apache.org/mesos/documentation.html
> > Framework development guide:
> > https://github.com/mesos/mesos/wiki/App-Framework-development-guide
> --
> This message was sent by Atlassian JIRA
> (v6.1.5#6160)

View raw message