hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vidhyashankar Venkataraman <vidhy...@yahoo-inc.com>
Subject Re: A possible bug in the scanner.
Date Wed, 13 Apr 2011 17:03:08 GMT
>> Even without the TTL expiration being applied, I think I've heard of this in other
cases where a very
>> restrictive filter was used on a large table scan.
Thanks, I was about to say that in a follow-up mail! We use a filter to scan records, produce
a list of delete records and bulk load them back to HBase. And the same problem will exist
even in that case.

And in response to JD's suggestions, this problem 'might' be related (mid-way I see JD's comment
on scanner timeouts during GCs which is quite analogous to the problem that I had pointed
out): I can't quite pinpoint exactly what the bug tries to address and if any fix has come
out of it. JD, can you let me know the status of the JIRA?

Thank you
V

On 4/13/11 9:15 AM, "Gary Helmling" <ghelmling@gmail.com> wrote:

Hi Vidhya,

So it sounds like the timeout thread is timing out the scanner when it takes
more than 60 seconds reading through the large column family store file
without returning anything to the client?  Even without the TTL expiration
being applied, I think I've heard of this in other cases where a very
restrictive filter was used on a large table scan.

If this is the case, it certainly seems like we should handle it better.  We
could do something as simple as refreshing the scanner timestamp every X
rows when iterating server side.

I'll check the code and open a JIRA (if we don't have one existing).  Thanks
for detailing the problem.

--gh



On Wed, Apr 13, 2011 at 7:44 AM, Vidhyashankar Venkataraman <
vidhyash@yahoo-inc.com> wrote:

> Hi
>   We had enabled scanner caching but I don't think it is the same issue
> because scanner.next in this case is blocking: the scanner is busy in the
> region server but hasn't returned anything yet since a row to be returned
> hasn't been found yet (all rows have expired but are still there since they
> havent been compacted yet).
>
> Vidhya
>
> On 4/13/11 1:44 AM, "Ted Yu" <yuzhihong@gmail.com> wrote:
>
> Have you read the following thread ?
> "ScannerTimeoutException when a scan enables caching, no exception when it
> doesn't"Did you enable caching ? If not, it is different issue.
>
> On Wed, Apr 13, 2011 at 12:40 AM, Vidhyashankar Venkataraman <
> vidhyash@yahoo-inc.com> wrote:
>
> > (This could be a known issue. Please let me know if it is).
> >
> > We had a set of uncompacted store files in a region. One of the column
> > families had a store file of 5 Gigs. The other column families were
> pretty
> > small (a few megabytes at most).
> >
> >  It so turned out that all these files had rows whose TTL had expired.
> Now
> > when this region was scanned (which should yield a result of a null set),
> we
> > got Scanner timeouts and UnknownScannerExceptions.
> >
> > And when we tried scanning the region without the large column family,
> the
> > scanner returned back safely with no result.
> >
> > So, I major compacted it and the scan started working correctly.
> >
> > So it looks like timeouts happen if the scanner does not return any
> output
> > for a specified time.
> > Which isn't exactly the correct thing to do, because it could be the case
> > that the scanner was indeed busy but it just so happened that there are
> no
> > rows yet to return back to the client.
> >
> > We can try increasing the scanner timeout, but this doesn't resolve the
> > underlying problem. Is this a know issue?
> >
> > Thank you
> > Vidhya
> >
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message