hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suraj Menon <surajsme...@apache.org>
Subject Re: [DISCUSS] Our 0.6 roadmap
Date Mon, 07 May 2012 10:32:40 GMT
Hi,

I think Thomas has a point here. Spilling, sorting and merging should
be configurable on our MessagingQueue.
If we have the above three features supported in our MessageQueue,
implementing mapreduce would be relatively trivial. I am going to
resume my efforts though :) for the sake of learning it and to
implement/validate schemes for the above three features on messaging
queues in the process.

I would like to share two ideas that Thomas and myself discussed last
week. First one is having a library of Superstep classes that users
could reuse. Say as an example, when we have real-time usage
implemented, we can provide a stream splitter/filter Superstep that
filters and splits traffic to other peers.(this specific problem might
even need a JNI implementation).  The second one is having a library
of other computation models that we can implement over BSP model. The
two paradigms we had in our mind during the conversation were
mapreduce and actor model.
Should we start having these from 0.6?

-Suraj

On Sun, May 6, 2012 at 3:21 AM, Thomas Jungblut
<thomas.jungblut@googlemail.com> wrote:
>
> >
> >  2. (New feature) hama-mapreduce
>
>
> I don't want to disgard Suraj's work on this, but don't you think this is
> not more than an example?
> If people run HDFS, why should they use the Hama MapReduce if they could
> just start up tasktrackers and run a full mapreduce.
> I think if we can split out the sort and merging and make it as robust as
> in MapReduce it can be a useful feature.
> Don't take it personally, it is just my opinion.
>
> Do you think Mikalai is finished with the random graph gen and the SpMV for
> the next release?
> Otherwise I want to contribute my kmeans (giraph is going to put some
> example up there as well from a stanford student), there is a top k-SSSP
> algorithm in work (not by me) and some other very secret thing ;)
>
> At least I wanted to add (with suraj of course!) the remote memory access
> and the local storage of objects in the new Superstep API.
>
>
> 2012/5/4 Praveen Sripati <praveensripati@gmail.com>
>
> > Not sure what additional algorithms to be implemented. But, any which
> > show significance improvements over MR/Hadoop would be useful in the
> > adoption of BSP/Hama.
> >
> > Praveen
> >
> > On Fri, May 4, 2012 at 2:44 PM, Suraj Menon <menonsuraj5@gmail.com> wrote:
> > > We can rope in fault tolerance in offline mode. Testing the feature would
> > > be more tedious. I hope we have 2 months for the next release?
> > >
> > > On Fri, May 4, 2012 at 3:16 AM, Edward J. Yoon <edwardyoon@apache.org
> > >wrote:
> > >
> > >> Hi,
> > >>
> > >> I think it's time to discuss roadmap of our next release.
> > >>
> > >>  1. Improve web UI
> > >>  2. (New feature) hama-mapreduce
> > >>  3. Improve real-time usage (and documentation)
> > >>  4. Improve RPC messenger
> > >>
> > >> and ?
> > >>
> > >> --
> > >> Best Regards, Edward J. Yoon
> > >> @eddieyoon
> > >>
> >
>
>
>
> --
> Thomas Jungblut
> Berlin <thomas.jungblut@gmail.com>

Mime
View raw message