Return-Path: X-Original-To: apmail-incubator-hama-dev-archive@minotaur.apache.org Delivered-To: apmail-incubator-hama-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A65D597D6 for ; Tue, 14 Feb 2012 22:49:38 +0000 (UTC) Received: (qmail 86387 invoked by uid 500); 14 Feb 2012 22:49:38 -0000 Delivered-To: apmail-incubator-hama-dev-archive@incubator.apache.org Received: (qmail 86345 invoked by uid 500); 14 Feb 2012 22:49:38 -0000 Mailing-List: contact hama-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hama-dev@incubator.apache.org Delivered-To: mailing list hama-dev@incubator.apache.org Received: (qmail 86336 invoked by uid 99); 14 Feb 2012 22:49:38 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Feb 2012 22:49:38 +0000 Received: from localhost (HELO [192.168.123.123]) (127.0.0.1) (smtp-auth username edwardyoon, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Feb 2012 22:49:38 +0000 Subject: Re: [DISCUSS] Hama 0.5 roadmap References: From: "Edward J. Yoon" Content-Type: text/plain; charset=us-ascii X-Mailer: iPad Mail (9A405) In-Reply-To: Message-Id: Date: Wed, 15 Feb 2012 07:49:37 +0900 To: "hama-dev@incubator.apache.org" Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) +1 :) Sent from my iPad On Feb 15, 2012, at 4:55 AM, Thomas Jungblut wrote: >>=20 >> Maybe 2~3 months later? >=20 >=20 > I would love that schedule, but I don't think we are going to handle this > timelimit with our current throughput. > Lin (may I call you like that?:D) has to add more detailed descriptions to= > the tasks so that we can also work on them. > So realistically we can make it in 5-6 months, our regular release schedul= e. > I know there is a business behind, but it doesn't help us to hurry. >=20 > HAMA-511 should not be a blocker for 0.5 release, it should be considered >> as a long term task I think. >=20 >=20 > +1. >=20 > We have to stabilize ourselves first rather than finding ways to >> differentiate ourselves from the competition or considering new paradigms= . >=20 >=20 > To be honest, the whole graph domain is conquered by giraph. And it is > perfectly fine, because they are focused on it. > Anyways, we have to push Hama into another direction. We can support graph= > processing, but our great success should be in iterative algorithms which > can easily implemented with BSP. > The first is to make my K-Means "tasteful" to Mahout, they are a great > driver, especially for researches. > The second idea is to support this dryad functionality, there is no > framework which has this ability out there and since Hortonworks is > supporting Microsoft, I think we can get some new people for Hama. > And the third one is to improve the real-time processing. This will be > greatly driven by the second idea, however we have to add some more simple= r > API for these task. This must be evaluated then (lets say in 0.6.0). >=20 > speculative task execution >=20 >=20 > Sorry, I seem to have not answered your question at all in the mail you've= > linked. > It is a cool feature, but I guess this should come along with > fault-tolerance, e.G. if we detect that a task is longer running than the > other. >=20 > A future target for Hama is a distributed cache like in BSPLib where you > can get and put objects. > I am having an eye on Apache Direct Memory, however they are in early stag= e > of incubation, so this may take a bit of time. >=20 > Everything else has been targeted so far. >=20 > What about graduation? > In my opinion we have stabilized so far with our community, I expect two > new comitters soon, a third one also seem to get on its way for > contribution. > The other tasks seems to be ticked off as well. >=20 > 2012/2/14 Edward J. Yoon >=20 >> Are you looking for this link? >> http://wiki.apache.org/hama/GroomServerFaultTolerance >>=20 >>>> There are many tasks required to work on and to be integrated in order >>>> to get (GroomServer) fault tolerance ready. Tasks include: >>>> - GroomServer status/ resource monitor >>>> - Failure Detection >>>> - Checkpointed data integration >>>> - Refactoring bsp() (if necessary) >>>> - Master decision making >>=20 >> Hmm, yes. and I missed message compressor. >>=20 >> Could you please split them into more smaller task so that we can help yo= u? >>=20 >>> I also would like to know why we rejected the idea of speculative task >>> execution? >>=20 >> I wanted to talk about speculative task execution before but, the idea >> of speculative task execution is not discussed/reported yet. ( >> http://markmail.org/thread/sq7neayhstqufrsz ) >>=20 >> To support this, we should add 'Progress' feature first. Currently, >> job/task progress checker is not implemented yet. >>=20 >>> How serious is the feature of real-time processing for Hama? I am told >> that >>> some are already using it for the purpose and read Thomas's blog on the >>> same. Are we deferring it until we have a design for offline processing >> or >>> should we keep it in mind for fault tolerance? >>=20 >> I think, yes if possible. But in some cases, maybe turning off >> recovery mode is the best. >>=20 >> I don't understand perfectly yet, so would you please describe the >> issues which must be discussed/considered? >>=20 >> On Tue, Feb 14, 2012 at 3:15 AM, Suraj Menon >> wrote: >>> +1 on HAMA 511 should not be blocker. >>>=20 >>> Also, I lost the wiki link that explains the fault tolerant design. It >>> would be helpful to undestand the recovery design. I believe that we wil= l >>> have the recovery BSP tasks scheduled to start running(in high >> probability) >>> on node with data where the checkpointed messages are written on HDFS >> with >>> a single input split? >>> I also would like to know why we rejected the idea of speculative task >>> execution? >>> I am currently working on HAMA-445 and HAMA-498. Thanks to Chiahung, I >> have >>> 2-3 good papers to read already :). >>>=20 >>> How serious is the feature of real-time processing for Hama? I am told >> that >>> some are already using it for the purpose and read Thomas's blog on the >>> same. Are we deferring it until we have a design for offline processing >> or >>> should we keep it in mind for fault tolerance? >>>=20 >>>=20 >>> Thanks, >>> Suraj >>>=20 >>>=20 >>>=20 >>> On Mon, Feb 13, 2012 at 12:25 PM, Chia-Hung Lin >> wrote: >>>=20 >>>> There are many tasks required to work on and to be integrated in order >>>> to get (GroomServer) fault tolerance ready. Tasks include: >>>> - GroomServer status/ resource monitor >>>> - Failure Detection >>>> - Checkpointed data integration >>>> - Refactoring bsp() (if necessary) >>>> - Master decision making >>>>=20 >>>> Currently I am working on the first one, and with a patch for 2nd on >>>> jira already. In my viewpoint, it might be difficult to get those >>>> tasks done within 2-3 months. >>>>=20 >>>> On 13 February 2012 17:05, Edward J. Yoon >> wrote: >>>>> Hi, >>>>>=20 >>>>> I think, it's time to discuss about our 0.5 roadmap more clearly. >>>>>=20 >>>>> IMO, I'd like to release Hama 0.5 with only fault tolerant processing,= >>>>> clearly defined BSP and Pregel interfaces. Maybe 2~3 months later? >>>>> And, HAMA-511 should not be a blocker for 0.5 release, it should be >>>>> considered as a long term task I think. >>>>>=20 >>>>> There's a lot of new M/R alternatives but no stable alternatives and >>>>> no dominant player at the moment. We have to stabilize ourselves first= >>>>> rather than finding ways to differentiate ourselves from the >>>>> competition or considering new paradigms. >>>>>=20 >>>>> Please feel free to leave your opinion! >>>>>=20 >>>>> -- >>>>> Best Regards, Edward J. Yoon >>>>> @eddieyoon >>>>=20 >>=20 >>=20 >>=20 >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon >>=20 >=20 >=20 >=20 > --=20 > Thomas Jungblut > Berlin