Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates
 209.85.160.50 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=TohoALW2QHXFVytkqAkGOl3sau/atBVuR+KaA1ey2mMAIkUY+Y9ByswCia+M+MXcRP
         XGeWz3Q2gvfpNtZ0jOVOFnGueOl8w6Ltg4aL73NbSiEEap6C7Xa2A4ZBsIONXjmPGPi1
         Asr+2jm7xVwyf8rKCC3GwiFOLpPGk034Qu6nc=
MIME-Version: 1.0
In-Reply-To: <c7a45cc60911201620q5979ccd8ob4cf3de96bdf1e76@mail.gmail.com>
References: <C72C3161.967E%silberst@yahoo-inc.com>
	 <7c962aed0911201204u156e7610idd8882401901af87@mail.gmail.com>
	 <e6aa01cc0911201235p7b96231ale4f4545a772e4a17@mail.gmail.com>
	 <c7a45cc60911201343n52492294y5e725873d5acfeca@mail.gmail.com>
	 <78568af10911201345s420b2ca6v6a271775dd40b9b6@mail.gmail.com>
	 <31a243e70911201355y30c6bbf3r4daec71df9ae05fc@mail.gmail.com>
	 <c7a45cc60911201620q5979ccd8ob4cf3de96bdf1e76@mail.gmail.com>
Date: Fri, 20 Nov 2009 16:36:11 -0800
Message-ID: <78568af10911201636m65650c9bx64b522f3c7c7d3fb@mail.gmail.com>
Subject: Re: Record limit in scan api?
From: Ryan Rawson <ryanobjc@gmail.com>
To: hbase-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The problem with this setting, is there is no good 'one size fits all'
value.  If we set it to 1, we do a RPC for ever row, clearly not
efficient for small rows.  If we set it to something as seemingly
innocuous as 5 or 10, then map reduces which do a significant amount
of processing on a row can cause the scanner to time out. The client
code will also give up if its been more than 60 seconds since the
scanner was last used, it's possible this code might need to be
adjusted so we can resume scanning.

I personally set it to anywhere between 1000-5000 for high performance
jobs on small rows.

The only factor is "can you process the cached chunk of rows in  <
60s".  Set the value as large as possible to not violate this and
you'll achieve max perf.

-ryan

On Fri, Nov 20, 2009 at 4:20 PM, Dave Latham <latham@davelink.net> wrote:
> Thanks for your thoughts. =A0It's true you can configure the scan buffer =
rows
> on an HTable or Scan instance, but I think there's something to be said t=
o
> try to work as well as we can out of the box.
>
> It would be more complication, but not by much. =A0To track the idea and =
see
> what it would look like, I made an issue and attached a proposed patch.
>
> Dave
>
> On Fri, Nov 20, 2009 at 1:55 PM, Jean-Daniel Cryans <jdcryans@apache.org>=
wrote:
>
>> And on the Scan as I wrote in my answer which is really really convenien=
t.
>>
>> Not convinced on using bytes as a value for caching... It would be
>> also more complicated.
>>
>> J-D
>>
>> On Fri, Nov 20, 2009 at 1:45 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>> > You can set it on a per-HTable basis. =A0HTable.setScannerCaching(int)=
;
>> >
>> >
>> >
>> > On Fri, Nov 20, 2009 at 1:43 PM, Dave Latham <latham@davelink.net>
>> wrote:
>> >> I have some tables with large rows and some tables with very small ro=
ws,
>> so
>> >> I keep my default scanner caching at 1 row, but have to remember to s=
et
>> it
>> >> higher when scanner tables with smaller rows. =A0It would be nice to =
have
>> a
>> >> default that did something reasonable across tables.
>> >>
>> >> Would it make sense to set scanner caching as a count of bytes rather
>> than a
>> >> count of rows? =A0That would make it similar to the write buffer for
>> batches
>> >> of puts that get flushed based on size rather than a fixed number of
>> Puts.
>> >> Then there could be some default value which should provide decent
>> >> performance out of the box.
>> >>
>> >> Dave
>> >>
>> >> On Fri, Nov 20, 2009 at 12:35 PM, Gary Helmling <ghelmling@gmail.com>
>> wrote:
>> >>
>> >>> To set this per scan you should be able to do:
>> >>>
>> >>> Scan s =3D new Scan()
>> >>> s.setCaching(...)
>> >>>
>> >>> (I think this works anyway)
>> >>>
>> >>>
>> >>> The other thing that I've found useful is using a PageFilter on scan=
s:
>> >>>
>> >>>
>> http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/=
filter/PageFilter.html
>> >>>
>> >>> I believe this is applied independently on each region server (?) so
>> you
>> >>> still need to do your own counting in iterating the results, but it =
can
>> be
>> >>> used to early out on the server side separately from the scanner
>> caching
>> >>> value.
>> >>>
>> >>> --gh
>> >>>
>> >>> On Fri, Nov 20, 2009 at 3:04 PM, stack <stack@duboce.net> wrote:
>> >>>
>> >>> > There is this in the configuration:
>> >>> >
>> >>> > =A0<property>
>> >>> > =A0 =A0<name>hbase.client.scanner.caching</name>
>> >>> > =A0 =A0<value>1</value>
>> >>> > =A0 =A0<description>Number of rows that will be fetched when calli=
ng next
>> >>> > =A0 =A0on a scanner if it is not served from memory. Higher cachin=
g
>> values
>> >>> > =A0 =A0will enable faster scanners but will eat up more memory and=
 some
>> >>> > =A0 =A0calls of next may take longer and longer times when the cac=
he is
>> >>> empty.
>> >>> > =A0 =A0</description>
>> >>> > =A0</property>
>> >>> >
>> >>> >
>> >>> > Being able to do it per Scan sounds like something we should add.
>> >>> >
>> >>> > St.Ack
>> >>> >
>> >>> >
>> >>> > On Fri, Nov 20, 2009 at 11:43 AM, Adam Silberstein
>> >>> > <silberst@yahoo-inc.com>wrote:
>> >>> >
>> >>> > > =A0 Hi,
>> >>> > > Is there a way to specify a limit on number of returned records =
for
>> >>> scan?
>> >>> > > =A0I
>> >>> > > don=B9t see any way to do this when building the scan. =A0If the=
re is,
>> that
>> >>> > > would be great. =A0If not, what about when iterating over the re=
sult?
>> =A0If
>> >>> I
>> >>> > > exit the loop when I reach my limit, will that approximate this
>> clause?
>> >>> > I
>> >>> > > guess my real question is about how scan is implemented in the
>> client.
>> >>> > > =A0I.e.
>> >>> > > How many records are returned from Hbase at a time as I iterate
>> through
>> >>> > the
>> >>> > > scan result? =A0If I want 1,000 records and 100 get returned at =
a
>> time,
>> >>> > then
>> >>> > > I=B9m in good shape. =A0On the other hand, if I want 10 records =
and get
>> 100
>> >>> > at
>> >>> > > a
>> >>> > > time, it=B9s a bit wasteful, though the waste is bounded.
>> >>> > >
>> >>> > > Thanks,
>> >>> > > Adam
>> >>> > >
>> >>> >
>> >>>
>> >>
>> >
>>
>