hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Owhadi <eric.owh...@esgyn.com>
Subject RE: Question on hbase.client.scanner.timeout.period
Date Tue, 25 Aug 2015 15:03:00 GMT
Hello St.Ack,
Thanks for your pointer, but I had already investigated JIRA
https://issues.apache.org/jira/browse/HBASE-13090
Unfortunately, this heartbeat will protect against rpc timeout, not server
side lease timeout that we are experiencing right now. I have not seen an
active JIRA fixing our issue.
Only https://issues.apache.org/jira/browse/HBASE6121 is complaining about
the exact same issue, but was never resolved.

The heartbeat JIRA in 13090 protect for situation where server scanner takes
so long to retrieve the highly filtered information, that it exceeds the RPC
timeout (hbase.rpc.timeout).
The timeout we are experiencing is the hbase.client.scanner.timeout.period,
also deprecatedly known as hbase.regionserver.lease.period
The mechanism is different: here, region server scanners wants to protect
themselves against dead clients that would not perform "close", and allow
releasing server side scanner resources. To do that, a lease mechanism is
implemented, and if between 2 next() call, more than
hbase.regionserver.lease.period occurs, the server side scanner will have
been forced closed by this lease timeout safety mechanism. On late next()
call, client will receive a DNRIOE of type unknownScannerException, and the
client will assess that it is coming most likely from the lease timeout (and
not from a region move), therefore throwing an exception instead of reset
scanner (for the region move scenario).

Hbase 1.1 does not address, as far as I have researched, the
hbase.client.scanner.timeout.period issue we are facing.

And yes, we will move to Hbase 1.1, and 1.0 as Cloudera and Hortonworks are
having version mismatch on the next official builds trafodion will support.

So my question is still open?

Best regards,
Eric Owhadi



-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Monday, August 24, 2015 11:07 PM
To: HBase Dev List
Subject: Re: Question on hbase.client.scanner.timeout.period

On Mon, Aug 24, 2015 at 4:48 PM, Eric Owhadi <eric.owhadi@esgyn.com> wrote:

> Hello everyone,
> We have been facing a situation on trafodion, where we are hitting the
> hbase.client.scanner.timeout.period scenario:
> basically, when doing queries that require spilling to disk because of
> high complexity of what is involved, the underlying hbase scanner
> serving one of the operation involved in the complex query cannot call
> the next() withing the timeout specify... too busy taking care of other
> business.
> This is legit scenario, and I was wondering why in the code, special
> care is done to make sure that client side, if a DNRIOE of type
> unknownScannerException shows up, and the
> hbase.client.scanner.timeout.period time elapsed, we make sure to
> throw a scannerTimeoutException, instead of just let it go and reset
> scanner.
>
> Scanners were redone in hbase 1.1. Can Trafodion come up onto hbase 1.1?
See https://blogs.apache.org/hbase/entry/scan_improvements_in_hbase_1 for
summary.
St.Ack



> I imagine that the lease time out implementation on region server side
> is supposed to protect from resource leak of scanner object server
> side. But I am not sure why we would make it so that client side throw
> this timeout exception, when in fact what just happened was that
> client was too busy to call next() on time.
>
> I am sure there is a reason, but cannot figure it out :-).
>
> BTW, I found this JIRA, talking about exact same thing:
> https://issues.apache.org/jira/browse/HBASE61-21 but with no resolution.
>


> Any help understanding the reason of the timeout thrwown client side
> instead of an automatic reset would be much appreciated, Best regards,
> Eric Owhadi
>

Mime
View raw message