hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michelan Arendse <miche...@addynamo.com>
Subject RE: Improving HBase scanner
Date Tue, 04 May 2010 14:51:58 GMT
Sorry if I have mislead you to think that the client times out. 

I'm expecting an hour's worth of data between startDate and toDate. When it gets to the scanner.next()
method it spends an awful amount of time to retrieve the data. Is there a way to speed up
the fetching of data from HBase?

-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: 04 May 2010 04:37 PM
To: hbase-user@hadoop.apache.org
Subject: Re: Improving HBase scanner

If long periods between next invocations, up the scanner lease.   See:

  <property>
    <name>hbase.regionserver.lease.period</name>
    <value>60000</value>
    <description>HRegion server lease period in milliseconds. Default is
    60 seconds. Clients must report in within this period else they are
    considered dead.</description>
  </property>

St.Ack


On Tue, May 4, 2010 at 7:04 AM, Michelan Arendse <michelan@addynamo.com> wrote:
> Yes I am waiting long periods between invocation of next. I didn't know that I am fetching
too much data at once.
>
> I am using HBase 0.20.3. This is my code:
>
> scan.setTimeRange(fromDate.getTime(), toDate.getTime());
> ResultScanner scanner = table.getScanner(scan);
>
> while( (result = scanner.next()) != null) {
> channelRow = getChannelDeliveryRow(Bytes.toString(result.getRow()));
>      channelRowList.add(channelRow);
> }
>
> This is some of the output from the log file:
> 2010-05-04 15:27:44,546 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache
LRU eviction started.  Attempting to free 62791520 bytes
> 2010-05-04 15:27:44,552 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache
LRU eviction completed. Freed 62797944 bytes.  Priority Sizes: Single=279.4997MB (293076672),
Multi=224.35243MB (235250576),Memory=0.0MB (0)
>
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: 04 May 2010 03:55 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Improving HBase scanner
>
> Are you waiting too long between invocations of next?  (i.e. > the
> scanner lease period?)  Or, perhaps you are fetching too much in the
> one go.  If you fetch 1000 at a time -- scanner caching -- and you
> don't get the next batch within the scanner lease period, again you
> will timeout.
>
> St.Ack
>
> On Tue, May 4, 2010 at 1:46 AM, Michelan Arendse <michelan@addynamo.com> wrote:
>> Hi
>>
>> I would like to know how configure HBase to improve the scanner fetching data from
the table or another method of using scanner, as my database is very large and scanner times
out.
>>
>> Kind Regards,
>>
>> Michelan Arendse
>> Junior Developer | AD:DYNAMO // happy business ;-)
>> Office 0861 Dynamo (0861 396266)  | Fax +27 (0) 21 465 2587
>>
>> Advertise Online Instantly - www.addynamo.com<http://www.addynamo.com> <http://www.addynamo.com>
>>
>>
>

Mime
View raw message