Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of vladrodionov@gmail.com
 designates 74.125.82.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CADcMMgGnu61D_oRGkPeyUDOqnUfDQX2jku-BWkYeCyL1WEHkUA@mail.gmail.com>
References: 
 <CAFAd71UiasTbFrb5HYeVaifFGbC=J1TJtEPgpz_wMrPz_t5UgA@mail.gmail.com>
	<CAFAd71VTUF7QcrWHxf5uoV7Vaq-BX4o2wDE4aLa-9ugxMmkydA@mail.gmail.com>
	<CADcMMgEYFJwbWisynXzWPF0MF2h6Wch+HxxUufD6ygSZvWMVGA@mail.gmail.com>
	<CAFAd71VDE2+DL6mWeBvuD1MCjVf=4dnadOCbwt5vu0_c=f8jSw@mail.gmail.com>
	<CADcMMgFr00H5YmSj7X=yEOpAFtjM2JP8VODhJt-OT0_o8yWB+Q@mail.gmail.com>
	<CAFAd71VhAzsvHQHEZBPvsGQV6fAmSw8xHNw73n3HEVmK16VGCw@mail.gmail.com>
	<CADcMMgFtouZ28svDdk=gWgrJMUbh4AG31gmc__sj=0j6zhpV+A@mail.gmail.com>
	<CAFAd71VQ_-4aazsvy6ChfkGkyFXJ0VHx7fQwwXB4UhiATUCRog@mail.gmail.com>
	<CADcMMgGnu61D_oRGkPeyUDOqnUfDQX2jku-BWkYeCyL1WEHkUA@mail.gmail.com>
Date: Thu, 24 Jul 2014 13:23:59 -0700
Message-ID: 
 <CAAg3a2osymd5MCjdeqPUKNgU=S7EvPaChveVVZkmMQKMhTYoyw@mail.gmail.com>
Subject: Re: how to do parallel scanning in map reduce using hbase as input?
From: Vladimir Rodionov <vladrodionov@gmail.com>
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c24092a69a6504fef63ca6

--001a11c24092a69a6504fef63ca6
Content-Type: text/plain; charset=UTF-8

I am working on improving inter-region scan performance and have the patch
already. The patch will be committed as soon as all tests are done. This
should improve M/R over HBase performance because now you will be able to
create input splits with granularities lower than a region without loss of
a performance.

See :

https://issues.apache.org/jira/browse/HBASE-7336
https://issues.apache.org/jira/browse/HBASE-5979

for more information on the subject.

-Vladimir Rodionov


On Tue, Jul 22, 2014 at 3:31 PM, Stack <stack@duboce.net> wrote:

> On Mon, Jul 21, 2014 at 11:11 PM, Li Li <fancyerii@gmail.com> wrote:
>
> > On Tue, Jul 22, 2014 at 1:57 PM, Stack <stack@duboce.net> wrote:
> > > On Mon, Jul 21, 2014 at 10:53 PM, Li Li <fancyerii@gmail.com> wrote:
> > >
> > >> Sorry, I enter tab and it send my unfinished post. See the following
> > >> mail for answers of other questions.
> > >>
> > >> I forget the exception's detail. It throws exception in terminal.
> > >
> > >
> > > What exception is thrown?
> > I forget it. maybe I can retry it with 8 mapper configuration. it
> > seems like out of memory exception
> >
>
>
> Who OOME'd?  The map task or hbase?
>
>
>
> > >
> > >
> > >
> > >> The
> > >> default io.sort.mb is 100 and I set it to 500 to speed up reducer.
> > >
> > >
> > > Do you have to have a reducer?  If you could skip the shuffle...
> > I have 8 reducers
> >
>
>
> Do you have to reduce?
>
> Would more reducers make your job run faster?
>
>
>
> > >
> > >
> > >
> > >> So
> > >> I set mapred.child.java.opts to 1g
> > >> The datanode/regionserver has 16GB memory but free memory
> > >
> > >
> > > Does the RS use the 16G?
> > the RS use 8G and there are datanode and tasktracker in this machine
> > >
> >
>
>
> How much for DN and TT?  They don't need much usually.
>
>
>
> > >
> > >
> > >> for
> > >> map-reduce is about 5gb. So I can't add more mappers
> > >>
> > >>
> > >> How much RAM in these machines?
> > 16GB
>
>
>
> These your machines or EC2?  Can you get bigger machines if EC2?
>
> St.Ack
>

--001a11c24092a69a6504fef63ca6--