hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <michael_se...@hotmail.com>
Subject Re: Lease exception when I execute large scan with filters.
Date Sat, 12 Apr 2014 21:14:07 GMT
You do realize that it is an internal feature and that the public API can change to not present
access to it.
However, that wouldn’t be a good idea because you would want to be able to change it and
in some cases review the versions of a cell.  How else do you describe versioning which is
unique to HBase and/or other specific databases, yet temporal modeling is not? 

In fact if memory servers… going back to 2009-10 IIRC the ‘old API’ vs the ‘new API’
for Hadoop where the ‘new API’ had a subset of the exposed classes / methods than the
old API? (It was an attempt to simplify the API… ) So again, APIs can change. 

The point is that you should be modeling your data on time if it is time sensitive data. Using
versioning bypasses this with bad consequences. 

By all means keep abusing the cell’s versioning. 
Just don’t complain about poor performance and your HBase tossing exceptions left and right.
I mean I can’t stop you from mixing booze, coke and meth. All I can do is tell you that
its not a good idea and not recommended. 

If you want a good definition of why HBase has versioning… go ask StAck, Ted, Nick or one
of the committers since they are more familiar with the internal workings of HBase than I.
When you get a good answer, then have the online HBase book updated.

-Mike

PS… if you want a really good example of why not to use versioning to store temporal data…

What happens if you’re storing 100 versions of a cell and you find out that you have a duplicate
entry with the wrong timestamp and you want to delete that one version.
How do you do that? Going from memory, and I could very well be wrong, but the tombstone marker
is on the cell, not the version, right? 

If it is on the version, what happens to the versions of the cell that are older than the
tombstone marker?
Sorry, its been a while since I’ve been intimate with HBase. Doing a bit of other things
at the moment, and I’m already overtaxing my last remaining living brain cell.  ;-) 


On Apr 12, 2014, at 9:14 PM, Brian Jeltema <bdjeltema@gmail.com> wrote:

> I don't want to be argumentative here, but by definition is's not an internal feature
because it's part of the
> public API. We use versioning in a way that makes me somewhat uncomfortable, but it's
been quite
> useful. I'd like to see a clear explanation of why it exists and what use cases it was
intended to support.
> 
> Brian
> 
>> Since you asked… 
>> 
>> Simplest answer… your schema should not rely upon internal features of the system.
 Since you are tracking your data along the lines of a temporal attribute it should be part
of the schema. In terms of a good design, by making it a part of the schema, you’re defining
that the data has a temporal property/attribute. 
>> 
>> Cell versioning is an internal feature of HBase. Its there for a reason. 
>> Perhaps one of the committers should expand on why its there.  (When I asked this
earlier, never got an answer. ) 
>> 
>> 
>> Longer answer… review how HBase stores the rows, including the versions of the
cell. 
>> You’re putting an unnecessary stress on the system. 
>> 
>> Its just not Zen… ;-) 
>> 
>> The reason I’m a bit short on this topic is that its an issue that keeps coming
up, over and over again because some idiot keeps looking to take a shortcut without understanding
the implications of their decision. Just like salting the key. (Note:  prepending a truncated
hash isn’t the same as using a salt.  Salting has a specific meaning and the salt is orthogonal
to the underlying key. Any relationship between the salt and the key is purely random luck.)

>> 
>> Does that help? 
>> (BTW, this should be part of any schema design talk… yet somehow I think its not
covered… ) 
>> 
>> -Mike
>> 
>> PS. Its not weird that the cell versions are checked. It makes perfect sense. 
>> 
>> On Apr 12, 2014, at 2:55 PM, Guillermo Ortiz <konstt2000@gmail.com> wrote:
>> 
>>> Well, It was just a example why I could keep a thousand versions or a cell.
>>> I didn't know that HBase was checking each version when I do a scan, it's a
>>> little weird when data is sorted.
>>> 
>>> You get my attention with your comment, that it's better to store data over
>>> time with new columns that with versions. Why is it better?
>>> Versions looks that there're very convenient for that use case. So, does it
>>> work better a rowkey with 3600 columns, that a rowkey with a column with
>>> 3600 versions? What's the reason for avoiding a massive use of versions?
>>> 
>>> 
>>> 2014-04-12 15:07 GMT+02:00 Michael Segel <michael_segel@hotmail.com>:
>>> 
>>>> Silly question...
>>>> 
>>>> Why does the idea of using versioning to capture temporal changes to data
>>>> keep being propagated?
>>>> 
>>>> Seriously this issue keeps popping up...
>>>> 
>>>> If you want to capture data over time... use a timestamp as part of the
>>>> column name.  Don't abuse the cell's version.
>>>> 
>>>> 
>>>> 
>>>> On Apr 11, 2014, at 11:03 AM, gortiz <gortiz@pragsis.com> wrote:
>>>> 
>>>>> Yes, I have tried with two different values for that value of versions,
>>>> 1000 and maximum value for integers.
>>>>> 
>>>>> But, I want to keep those versions. I don't want to keep just 3
>>>> versions. Imagine that I want to record a new version each minute and store
>>>> a day, those are 1440 versions.
>>>>> 
>>>>> Why is HBase going to read all the versions?? , I thought, if you don't
>>>> indicate any versions it's just read the newest and skip the rest. It
>>>> doesn't make too much sense to read all of them if data is sorted, plus the
>>>> newest version is stored in the top.
>>>>> 
>>>>> 
>>>>> On 11/04/14 11:54, Anoop John wrote:
>>>>>> What is the max version setting u have done for ur table cf?  When
u set
>>>>>> some a value, HBase has to keep all those versions.  During a scan
it
>>>> will
>>>>>> read all those versions. In 94 version the default value for the
max
>>>>>> versions is 3.  I guess you have set some bigger value.   If u have
not,
>>>>>> mind testing after a major compaction?
>>>>>> 
>>>>>> -Anoop-
>>>>>> 
>>>>>> On Fri, Apr 11, 2014 at 1:01 PM, gortiz <gortiz@pragsis.com>
wrote:
>>>>>> 
>>>>>>> Last test I have done it's to reduce the number of versions to
100.
>>>>>>> So, right now, I have 100 rows with 100 versions each one.
>>>>>>> Times are: (I got the same times for blocksize of 64Ks and 1Mb)
>>>>>>> 100row-1000versions + blockcache-> 80s.
>>>>>>> 100row-1000versions + No blockcache-> 70s.
>>>>>>> 
>>>>>>> 100row-*100*versions + blockcache-> 7.3s.
>>>>>>> 100row-*100*versions + No blockcache-> 6.1s.
>>>>>>> 
>>>>>>> What's the reasons of this? I guess HBase is enough smart for
not
>>>> consider
>>>>>>> old versions, so, it just checks the newest. But, I reduce 10
times the
>>>>>>> size (in versions) and I got a 10x of performance.
>>>>>>> 
>>>>>>> The filter is scan 'filters', {FILTER => "ValueFilter(=,
>>>>>>> 'binary:5')",STARTROW => '1010000000000000000000000000000000000101',
>>>>>>> STOPROW => '6010000000000000000000000000000000000201'}
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On 11/04/14 09:04, gortiz wrote:
>>>>>>> 
>>>>>>>> Well, I guessed that, what it doesn't make too much sense
because
>>>> it's so
>>>>>>>> slow. I only have right now 100 rows with 1000 versions each
row.
>>>>>>>> I have checked the size of the dataset and each row is about
700Kbytes
>>>>>>>> (around 7Gb, 100rowsx1000versions). So, it should only check
100 rows
>>>> x
>>>>>>>> 700Kbytes = 70Mb, since it just check the newest version.
How can it
>>>> spend
>>>>>>>> too many time checking this quantity of data?
>>>>>>>> 
>>>>>>>> I'm generating again the dataset with a bigger blocksize
(previously
>>>> was
>>>>>>>> 64Kb, now, it's going to be 1Mb). I could try tunning the
scanning and
>>>>>>>> baching parameters, but I don't think they're going to affect
too
>>>> much.
>>>>>>>> 
>>>>>>>> Another test I want to do, it's generate the same dataset
with just
>>>>>>>> 100versions, It should spend around the same time, right?
Or am I
>>>> wrong?
>>>>>>>> 
>>>>>>>> On 10/04/14 18:08, Ted Yu wrote:
>>>>>>>> 
>>>>>>>>> It should be newest version of each value.
>>>>>>>>> 
>>>>>>>>> Cheers
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, Apr 10, 2014 at 9:55 AM, gortiz <gortiz@pragsis.com>
wrote:
>>>>>>>>> 
>>>>>>>>> Another little question is, when the filter I'm using,
Do I check
>>>> all the
>>>>>>>>>> versions? or just the newest? Because, I'm wondering
if when I do a
>>>> scan
>>>>>>>>>> over all the table, I look for the value "5" in all
the dataset or
>>>> I'm
>>>>>>>>>> just
>>>>>>>>>> looking for in one newest version of each value.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 10/04/14 16:52, gortiz wrote:
>>>>>>>>>> 
>>>>>>>>>> I was trying to check the behaviour of HBase. The
cluster is a
>>>> group of
>>>>>>>>>>> old computers, one master, five slaves, each
one with 2Gb, so,
>>>> 12gb in
>>>>>>>>>>> total.
>>>>>>>>>>> The table has a column family with 1000 columns
and each column
>>>> with
>>>>>>>>>>> 100
>>>>>>>>>>> versions.
>>>>>>>>>>> There's another column faimily with four columns
an one image of
>>>> 100kb.
>>>>>>>>>>> (I've tried without this column family as well.)
>>>>>>>>>>> The table is partitioned manually in all the
slaves, so data are
>>>>>>>>>>> balanced
>>>>>>>>>>> in the cluster.
>>>>>>>>>>> 
>>>>>>>>>>> I'm executing this sentence *scan 'table1', {FILTER
=>
>>>> "ValueFilter(=,
>>>>>>>>>>> 'binary:5')"* in HBase 0.94.6
>>>>>>>>>>> My time for lease and rpc is three minutes.
>>>>>>>>>>> Since, it's a full scan of the table, I have
been playing with the
>>>>>>>>>>> BLOCKCACHE as well (just disable and enable,
not about the size of
>>>>>>>>>>> it). I
>>>>>>>>>>> thought that it was going to have too much calls
to the GC. I'm not
>>>>>>>>>>> sure
>>>>>>>>>>> about this point.
>>>>>>>>>>> 
>>>>>>>>>>> I know that it's not the best way to use HBase,
it's just a test. I
>>>>>>>>>>> think
>>>>>>>>>>> that it's not working because the hardware isn't
enough, although,
>>>> I
>>>>>>>>>>> would
>>>>>>>>>>> like to try some kind of tunning to improve it.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 10/04/14 14:21, Ted Yu wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Can you give us a bit more information:
>>>>>>>>>>>> HBase release you're running
>>>>>>>>>>>> What filters are used for the scan
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> 
>>>>>>>>>>>> On Apr 10, 2014, at 2:36 AM, gortiz <gortiz@pragsis.com>
wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> I got this error when I execute a full scan
with filters about a
>>>>>>>>>>>> table.
>>>>>>>>>>>> 
>>>>>>>>>>>>> Caused by: java.lang.RuntimeException:
org.apache.hadoop.hbase.
>>>>>>>>>>>>> regionserver.LeaseException:
>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.LeaseException:
lease
>>>>>>>>>>>>> '-4165751462641113359' does not exist
>>>>>>>>>>>>>  at
>>>> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>  at org.apache.hadoop.hbase.regionserver.HRegionServer.
>>>>>>>>>>>>> next(HRegionServer.java:2482)
>>>>>>>>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>>>>>>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>>>>>>>>>> NativeMethodAccessorImpl.java:39)
>>>>>>>>>>>>>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>>>>>>>>>> DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>>>>  at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>  at
>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(
>>>>>>>>>>>>> WritableRpcEngine.java:320)
>>>>>>>>>>>>>  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(
>>>>>>>>>>>>> HBaseServer.java:1428)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have read about increase the lease
time and rpc time, but it's
>>>> not
>>>>>>>>>>>>> working.. what else could I try?? The
table isn't too big. I have
>>>>>>>>>>>>> been
>>>>>>>>>>>>> checking the logs from GC, HMaster and
some RegionServers and I
>>>>>>>>>>>>> didn't see
>>>>>>>>>>>>> anything weird. I tried as well to try
with a couple of caching
>>>>>>>>>>>>> values.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>> *Guillermo Ortiz*
>>>>>>>>>> /Big Data Developer/
>>>>>>>>>> 
>>>>>>>>>> Telf.: +34 917 680 490<
>>>> https://mail.google.com/mail/u/0/html/compose/static_files/blank_quirks.html#
>>>>> 
>>>>>>>>>> Fax: +34 913 833 301<
>>>> https://mail.google.com/mail/u/0/html/compose/static_files/blank_quirks.html#
>>>>> 
>>>>>>>>>> C/ Manuel Tovar, 49-53 - 28034 Madrid - Spain
>>>>>>>>>> 
>>>>>>>>>> _http://www.bidoop.es_
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> --
>>>>>>> *Guillermo Ortiz*
>>>>>>> /Big Data Developer/
>>>>>>> 
>>>>>>> Telf.: +34 917 680 490<
>>>> https://mail.google.com/mail/u/0/html/compose/static_files/blank_quirks.html#
>>>>> 
>>>>>>> Fax: +34 913 833 301<
>>>> https://mail.google.com/mail/u/0/html/compose/static_files/blank_quirks.html#
>>>>> 
>>>>>>> C/ Manuel Tovar, 49-53 - 28034 Madrid - Spain
>>>>>>> 
>>>>>>> _http://www.bidoop.es_
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> *Guillermo Ortiz*
>>>>> /Big Data Developer/
>>>>> 
>>>>> Telf.: +34 917 680 490
>>>>> Fax: +34 913 833 301
>>>>> C/ Manuel Tovar, 49-53 - 28034 Madrid - Spain
>>>>> 
>>>>> _http://www.bidoop.es_
>>>>> 
>>>> 
>>>> 
>> 
>> 
> 
> 


Mime
View raw message