hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavan Sudheendra <pavan0...@gmail.com>
Subject Re: How is pig so much faster than my java MR job?
Date Tue, 03 Sep 2013 07:28:26 GMT
Yes.. Caching is set to 1000 .
Number of mappers in pig is 21, don't know about number of reducers..
Probably less than that..

It does many things basically.. The logic consists of the MR job populating
2 tables, one table which has 4m rows and 1 table which has 50,000 rows..


On Tue, Sep 3, 2013 at 12:33 PM, Anoop John <anoop.hbase@gmail.com> wrote:

> You are using Scan caching in ur MR java code?   How many mapper and
> reducers in case of pig?  How is ur Java MR job written..  a bit more on
> its logic pls.
>
> -Anoop-
>
> On Tue, Sep 3, 2013 at 11:33 AM, Pavan Sudheendra <pavan0591@gmail.com
> >wrote:
>
> > Hi all,
> > I'm doing a kind of table join across 3 tables in the MR job ( plus doing
> > some computation).. It took nearly 19 hours to run with 21 mappers and 21
> > reducers.. But with pig it ran in less than 2 hours..
> > We are using HBase both as source and sink.. Is this normal?
> >
> >
> > On Mon, Sep 2, 2013 at 7:20 PM, Adrien Mogenet <adrien.mogenet@gmail.com
> > >wrote:
> >
> > > You should have a kind of debug/explain mode in Pig, and will show you
> > how
> > > it does clever things to optimize its excution path.
> > >
> > >
> > > On Mon, Sep 2, 2013 at 3:45 PM, Dhaval Shah <
> prince_mithibai@yahoo.co.in
> > > >wrote:
> > >
> > > > Java MR code is not optimized/efficiently written while Pig is highly
> > > > optimized? Can you give us more details on what exactly you are
> trying
> > to
> > > > do and how your Java MR code is written, how many MR jobs for Java vs
> > Pig
> > > > and so on
> > > >
> > > > Sent from Yahoo! Mail on Android
> > > >
> > > >
> > >
> > >
> > > --
> > > Adrien Mogenet
> > > http://www.borntosegfault.com
> > >
> >
> >
> >
> > --
> > Regards-
> > Pavan
> >
>



-- 
Regards-
Pavan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message