incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tommaso Teofili <tommaso.teof...@gmail.com>
Subject Re: Please review new APIs.
Date Wed, 02 Nov 2011 12:23:36 GMT
2011/11/2 Thomas Jungblut <thomas.jungblut@googlemail.com>

> Another point while fixing the local runner:
>
> Are we now input driven?
> I see in the code that the user defined task number is overriden by the
> number of splits.
> Was this your intention? This will actually make realtime processing with
> no static input a real pain.
> For example if you want a similar behaviour in Hadoop M/R you'll need to
> create dummy splits, and this is not what we should aim at.
>
> We could simply check if the user define the NullInputFormat or nothing and
> then use the number of tasks the user has configured.
>

For sure I agree we should allow the former programming model with no input
without explicitly instantiating dummy inputs/splits. What about providing
two basic (different) implementations?
Tommaso


>
> 2011/11/2 Tommaso Teofili <tommaso.teofili@gmail.com>
>
> > 2011/11/2 Edward J. Yoon <edwardyoon@apache.org>
> >
> > > > I'm sure that not every job actually needs a cleanup or a setup.
> > >
> > > You're right. Almost BSP applications should override bsp() method
> > > but, setup() and cleaner() methods are not as you said. Let's fix
> > > them.
> > >
> >
> > Agreed +1
> >
> >
> > >
> > > > Generally I would suggest to integrate the OutputCollector and the
> > > > RecordReader into the BSPPeerImpl.
> > > > So our peer is like the context in Hadoop.
> > >
> > > Good idea.
> > >
> >
> > +1 here too
> >
> > Tommaso
> >
> >
> > >
> > > On Wed, Nov 2, 2011 at 9:03 PM, Thomas Jungblut
> > > <thomas.jungblut@googlemail.com> wrote:
> > > > Yes. When I reworked that API, I made a default implementation in our
> > > > abstract BSP class.
> > > > So the user has to override the methods for himself, if he needs to.
> > > > I'm sure that not every job actually needs a cleanup or a setup.
> > > >
> > > > Generally I would suggest to integrate the OutputCollector and the
> > > > RecordReader into the BSPPeerImpl.
> > > > So our peer is like the context in Hadoop.
> > > > But that is just a minor thing. It is a great improvement ;)
> > > >
> > > > 2011/11/2 Edward J. Yoon <edwardyoon@apache.org>
> > > >
> > > >> There're bsp(), setup() and cleaner() methods.
> > > >>
> > > >> What is you suggestion?
> > > >>
> > > >> On Wed, Nov 2, 2011 at 8:47 PM, Thomas Jungblut
> > > >> <thomas.jungblut@googlemail.com> wrote:
> > > >> > Have a look at the combiner class. I know that this is just a
> > "test",
> > > but
> > > >> > it is really messy if the user does not use the methods, but
is
> > > forced to
> > > >> > override them.
> > > >> >
> > > >> > 2011/11/2 Edward J. Yoon <edwardyoon@apache.org>
> > > >> >
> > > >> >> Why?
> > > >> >>
> > > >> >> On Wed, Nov 2, 2011 at 8:21 PM, Thomas Jungblut
> > > >> >> <thomas.jungblut@googlemail.com> wrote:
> > > >> >> > I totally dislike that BSP class now has abstract methods
> instead
> > > of
> > > >> >> > default implementations.
> > > >> >> >
> > > >> >> > 2011/11/2 Edward J. Yoon <edwardyoon@apache.org>
> > > >> >> >
> > > >> >> >> Hi all,
> > > >> >> >>
> > > >> >> >> As you know, recently combiners and IO are added.
> > > >> >> >>
> > > >> >> >> Please review them from user viewpoint.
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> http://svn.apache.org/repos/asf/incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/PiEstimator.java
> > > >> >> >>
> > > >> >> >> I'm testing multiple tasks and IO features on 100
nodes
> cluster
> > > using
> > > >> >> >> 10 tasks per node. If there's no issue, I'll close
HAMA-258.
> > > >> >> >>
> > > >> >> >> Thanks.
> > > >> >> >>
> > > >> >> >> --
> > > >> >> >> Best Regards, Edward J. Yoon
> > > >> >> >> @eddieyoon
> > > >> >> >>
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > --
> > > >> >> > Thomas Jungblut
> > > >> >> > Berlin <thomas.jungblut@gmail.com>
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> --
> > > >> >> Best Regards, Edward J. Yoon
> > > >> >> @eddieyoon
> > > >> >>
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Thomas Jungblut
> > > >> > Berlin <thomas.jungblut@gmail.com>
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best Regards, Edward J. Yoon
> > > >> @eddieyoon
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Thomas Jungblut
> > > > Berlin <thomas.jungblut@gmail.com>
> > > >
> > >
> > >
> > >
> > > --
> > > Best Regards, Edward J. Yoon
> > > @eddieyoon
> > >
> >
>
>
>
> --
> Thomas Jungblut
> Berlin <thomas.jungblut@gmail.com>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message