hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Wolf <opus...@gmail.com>
Subject Re: Confirming a Bug
Date Sun, 18 Mar 2012 23:40:39 GMT
Excellent!   Thank you very much (other) Lars.

I have only tested this one one dataset, and only on a few values of 
caching.  I certainly get different results with 10,000 5,000 and 1,000 
caching.  1,000 gives me the same results as default.  I also get 
different results when I add families to the Scan.

I seem to be surpassing some maximum buffer size.  The number of results 
is always the correct value - some multiple of the cache size.  For 
example, the correct value was 24,452, but when caching was set to 
10,000, I got 4,452 results.  When I then removed a family from the 
scan, I got 14,452 results.

I'll try to write a standalone program to reproduce this.  I'll get back 
to you soon.

P

P.S.  I just want to check.  The following code counts the number of 
results.  I don't need to do anything to "get the next cache" or 
something do I?

          Iterator<Result>   it = scanner.iterator();
          while (it.hasNext()) {
              Result result = it.next();
              ...
          }




On 3/18/12 5:51 PM, lars hofhansl wrote:
> Hi Peter,
>
> (this is the other Lars)
>
>
> Does this depend on your dataset at all? Does not it also happen for smaller value of
scanner caching?
>
>
> Any chance that you can reproduce this in a unittest and file a jira?
> If you do (specifically the test), I'll promise I'll look at it this week :)
>
>
> -- Lars (H)
>
>
>
> ________________________________
>   From: Peter Wolf<opus111@gmail.com>
> To: user@hbase.apache.org
> Sent: Sunday, March 18, 2012 7:13 AM
> Subject: Re: Confirming a Bug
>
> Hi Lars,
>
> I don't think so...  My behavior is definitely tied to the amount of
> data in each Result.  There definitely seems to be some sort of
> threshold.  Changing the caching amount produces a completely repeatable
> behavior.  10,000, 5,000, and 1000 each produce different repeatable
> results, and changing the families added as produces different reliable
> results. There is no "sometimes" or "occasional", and if there was a
> Major Compaction, it wouldn't happen that often.
>
> https://issues.apache.org/jira/browse/HBASE-5121
> https://issues.apache.org/jira/browse/HBASE-2856
>
> Note that with all my families added each result is a few 1000 bytes
> big.  Is that unusually large?
>
> Thanks
> P
>
>
>
> On 3/18/12 5:28 AM, Lars George wrote:
>> Hi Peter,
>>
>> Could you be hitting HBASE-5121? Or even HBASE-2856?
>>
>> Lars
>>
>> On Mar 17, 2012, at 20:46, Peter Wolf<opus111@gmail.com>   wrote:
>>
>>> Hello,
>>>
>>> A couple of days ago, I asked about strange behavior in my "Scan.addFamiliy reduces
results" thread.
>>>
>>> I want to confirm that I did find a bug, and if so, how to submit a bug report.
>>>
>>> The basic strangeness is that changing the amount of caching, changes the number
of results.  In the original thread, this was confused by the fact that adding different families
also changed the number of results.  We thought it was a filtering problem.
>>>
>>> However, changing nothing but the setCaching() value changes the number of results.
 Furthermore, the result difference is a multiple of the setCaching() value.
>>>
>>> Here is the pseudo code:
>>>
>>>           Scan scan = new Scan(...);
>>>           scan.addFamily(...);
>>>           Filter filter = ...
>>>           scan.setFilter(filter);
>>>
>>> -->       scan.setCaching(10000);<--
>>>
>>>           scanner = hTable.getScanner(scan);
>>>           Iterator<Result>   it = scanner.iterator();
>>>           while (it.hasNext()) {
>>>               Result result = it.next();
>>>               ...
>>>           }
>>>
>>>
>>> Thank you
>>> Peter


Mime
View raw message