Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com
 designates 74.125.82.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOF-KfhZp6rHD6BxeNsV-_ozrhWrL5r4B=9Rr=b7dQw_=R+0zw@mail.gmail.com>
References: 
 <CAOF-Kfg5asaSfi4DaMo04ie8ha8YeSrVXgxTMw_Ee2DLWmXmpw@mail.gmail.com>
	<CALte62xysT82SFsP1e7vU5wvTvR5-3eKwGbRspuSk5KYip7dtA@mail.gmail.com>
	<CAFLnt_r2xopCaiTyAC5xhD0xb=gZapdgBVJAyTno9BNnfdQShw@mail.gmail.com>
	<CAOF-KfhZp6rHD6BxeNsV-_ozrhWrL5r4B=9Rr=b7dQw_=R+0zw@mail.gmail.com>
Date: Sun, 9 Oct 2011 08:44:17 -0700
Message-ID: 
 <CALte62w6KiQt_+FbWEMvxMCxe33MdxyuCpDtFS=_7qM2qpVWBw@mail.gmail.com>
Subject: Re: speeding up rowcount
From: Ted Yu <yuzhihong@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e9210d99f104aedf8c00

--0016e6d7e9210d99f104aedf8c00
Content-Type: text/plain; charset=ISO-8859-1

Excellent question.
There seems to be a bug for RowCounter.

In TableInputFormat:
        if (conf.get(SCAN_CACHEDROWS) != null) {
          scan.setCaching(Integer.parseInt(conf.get(SCAN_CACHEDROWS)));
        }
But I don't see SCAN_CACHEDROWS in either TableMapReduceUtil or RowCounter.

Mind filing a bug ?

On Sun, Oct 9, 2011 at 8:30 AM, Rita <rmorgan466@gmail.com> wrote:

> Thanks for the responses.
>
> Where do I set the high Scan cache values?
>
>
> On Sun, Oct 9, 2011 at 11:19 AM, Himanshu Vashishtha <
> hvashish@cs.ualberta.ca> wrote:
>
> > Since a MapReduce is a separate process, try with a high Scan cache
> value.
> >
> > http://hbase.apache.org/book.html#perf.hbase.client.caching
> >
> > Himanshu
> >
> > On Sun, Oct 9, 2011 at 9:09 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > > I guess your hbase.hregion.max.filesize is quite high.
> > > If possible, lower its value so that you have smaller regions.
> > >
> > > On Sun, Oct 9, 2011 at 7:50 AM, Rita <rmorgan466@gmail.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> I have been doing a rowcount via mapreduce and its taking about 4-5
> > hours
> > >> to
> > >> count a 500million rows in a table. I was wondering if there are any
> map
> > >> reduce tunings I can do so it will go much faster.
> > >>
> > >> I have 10 node cluster, each node with 8CPUs with 64GB of memory. Any
> > >> tuning
> > >> advice would be much appreciated.
> > >>
> > >>
> > >> --
> > >> --- Get your facts first, then you can distort them as you please.--
> > >>
> > >
> >
>
>
>
> --
> --- Get your facts first, then you can distort them as you please.--
>

--0016e6d7e9210d99f104aedf8c00--