hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lars hofhansl <lhofha...@yahoo.com>
Subject Re: Is it necessary to set MD5 on rowkey?
Date Wed, 19 Dec 2012 20:51:59 GMT
Doesn't Alex' blog post do that?




________________________________
 From: Michael Segel <michael_segel@hotmail.com>
To: user@hbase.apache.org; lars hofhansl <lhofhansl@yahoo.com> 
Sent: Wednesday, December 19, 2012 11:46 AM
Subject: Re: Is it necessary to set MD5 on rowkey?
 
Ok, 

Maybe I'm missing something.
Why don't you walk me through the use of a salt example. 


On Dec 19, 2012, at 12:37 PM, lars hofhansl <lhofhansl@yahoo.com> wrote:

> I would disagree here.
> It depends on what you are doing and blanket statements about "this is very, very bad"
typically do not help.
> 
> Salting (even round robin) is very nice to distribute write load *and* it gives you a
natural way to parallelize scans assuming scans are of reasonable size.
> 
> If the typical use case is point gets then hashing or inverting keys would be preferable.
As usual: It depends.
> 
> -- Lars
> 
> 
> 
> ________________________________
> From: Michael Segel <michael_segel@hotmail.com>
> To: user@hbase.apache.org 
> Sent: Tuesday, December 18, 2012 3:29 PM
> Subject: Re: Is it necessary to set MD5 on rowkey?
> 
> Alex, 
> And that's the point. Salt as you explain it conceptually implies that the number you
are adding to the key to ensure a better distribution means that you will have inefficiencies
in terms of scans and gets. 
> 
> Using a hash as either the full key, or taking the hash, truncating it and appending
the key may screw up scans, but your get() is intact. 
> 
> There are other options like inverting the numeric key ... 
> 
> And of course doing nothing. 
> 
> Using a salt as part of the design pattern is bad. 
> 
> With respect to the OP, I was discussing the use of hash and some alternatives to how
to implement the hash of a key. 
> Again, doing nothing may also make sense too, if you understand the risks and you know
how your data is going to be used.
> 
> 
> On Dec 18, 2012, at 11:36 AM, Alex Baranau <alex.baranov.v@gmail.com> wrote:
> 
>> Mike,
>> 
>> Please read *full post* before judge. In particular, "Hash-based
>> distribution" section. You can find the same in HBaseWD small README file
>> [1] (not sure if you read it at all before commenting on the lib). Round
>> robin is mainly for explaining the concept/idea (though not only for that).
>> 
>> Thank you,
>> Alex Baranau
>> ------
>> Sematext :: http://blog.sematext.com/ :: Hadoop - HBase - ElasticSearch -
>> Solr
>> 
>> [1] https://github.com/sematext/HBaseWD
>> 
>> On Tue, Dec 18, 2012 at 12:24 PM, Michael Segel
>> <michael_segel@hotmail.com>wrote:
>> 
>>> Quick answer...
>>> 
>>> Look at the salt.
>>> Its just a number from a round robin counter.
>>> There is no tie between the salt and row.
>>> 
>>> So when you want to fetch a single row, how do you do it?
>>> ...
>>> ;-)
>>> 
>>> On Dec 18, 2012, at 11:12 AM, Alex Baranau <alex.baranov.v@gmail.com>
>>> wrote:
>>> 
>>>> Hello,
>>>> 
>>>> @Mike:
>>>> 
>>>> I'm the author of that post :).
>>>> 
>>>> Quick reply to your last comment:
>>>> 
>>>> 1) Could you please describe why "the use of a 'Salt' is a very, very bad
>>>> idea" in more specific way than "Fetching data takes more effort". Would
>>> be
>>>> helpful for anyone who is looking into using this approach.
>>>> 
>>>> 2) The approach described in the post also says you can prefix with the
>>>> hash, you probably missed that.
>>>> 
>>>> 3) I believe your answer, "use MD5 or SHA-1" doesn't help bigdata guy.
>>>> Please re-read the question: the intention is to distribute the load
>>> while
>>>> still being able to do "partial key scans". The blog post linked above
>>>> explains one possible solution for that, while your answer doesn't.
>>>> 
>>>> @bigdata:
>>>> 
>>>> Basically when it comes to solving two issues: distributing writes and
>>>> having ability to read data sequentially, you have to balance between
>>> being
>>>> good at both of them. Very good presentation by Lars:
>>>> 
>>> http://www.slideshare.net/larsgeorge/hbase-advanced-schema-design-berlin-buzzwords-june-2012
>>> ,
>>>> slide 22. You will see how this is correlated. In short:
>>>> * having md5/other hash prefix of the key does better w.r.t. distributing
>>>> writes, while compromises ability to do range scans efficiently
>>>> * having very limited number of 'salt' prefixes still allows to do range
>>>> scans (less efficiently than normal range scans, of course, but still
>>> good
>>>> enough in many cases) while providing worse distribution of writes
>>>> 
>>>> In the latter case by choosing number of possible 'salt' prefixes (which
>>>> could be derived from hashed values, etc.) you can balance between
>>>> distributing writes efficiency and ability to run fast range scans.
>>>> 
>>>> Hope this helps
>>>> 
>>>> Alex Baranau
>>>> ------
>>>> Sematext :: http://blog.sematext.com/ :: Hadoop - HBase - ElasticSearch
>>> -
>>>> Solr
>>>> 
>>>> On Tue, Dec 18, 2012 at 8:52 AM, Michael Segel <
>>> michael_segel@hotmail.com>wrote:
>>>> 
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> First, the use of a 'Salt' is a very, very bad idea and I would really
>>>>> hope that the author of that blog take it down.
>>>>> While it may solve an initial problem in terms of region hot spotting,
>>> it
>>>>> creates another problem when it comes to fetching data. Fetching data
>>> takes
>>>>> more effort.
>>>>> 
>>>>> With respect to using a hash (MD5 or SHA-1) you are creating a more
>>> random
>>>>> key that is unique to the record.  Some would argue that using MD5 or
>>> SHA-1
>>>>> that mathematically you could have a collision, however you could then
>>>>> append the key to the hash to guarantee uniqueness. You could also do
>>>>> things like take the hash and then truncate it to the first byte and
>>> then
>>>>> append the record key. This should give you enough randomness to avoid
>>> hot
>>>>> spotting after the initial region completion and you could pre-split
out
>>>>> any number of regions. (First byte 0-255 for values, so you can program
>>> the
>>>>> split...
>>>>> 
>>>>> 
>>>>> Having said that... yes, you lose the ability to perform a sequential
>>> scan
>>>>> of the data.  At least to a point.  It depends on your schema.
>>>>> 
>>>>> Note that you need to think about how you are primarily going to access
>>>>> the data.  You can then determine the best way to store the data to
gain
>>>>> the best performance. For some applications... the region hot spotting
>>>>> isn't an important issue.
>>>>> 
>>>>> Note YMMV
>>>>> 
>>>>> HTH
>>>>> 
>>>>> -Mike
>>>>> 
>>>>> On Dec 18, 2012, at 3:33 AM, Damien Hardy <dhardy@viadeoteam.com>
>>> wrote:
>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> There is middle term betwen sequecial keys (hot spoting risk) and
md5
>>>>>> (heavy scan):
>>>>>> * you can use composed keys with a field that can segregate data
>>>>>> (hostname, productname, metric name) like OpenTSDB
>>>>>> * or use Salt with a limited number of values (example
>>>>>> substr(md5(rowid),0,1) = 16 values)
>>>>>>   so that a scan is a combination of 16 filters on on each salt
values
>>>>>>   you can base your code on HBaseWD by sematext
>>>>>> 
>>>>>> 
>>>>> 
>>> http://blog.sematext.com/2012/04/09/hbasewd-avoid-regionserver-hotspotting-despite-writing-records-with-sequential-keys/
>>>>>>    https://github.com/sematext/HBaseWD
>>>>>> 
>>>>>> Cheers,
>>>>>> 
>>>>>> 
>>>>>> 2012/12/18 bigdata <bigdatabase@outlook.com>
>>>>>> 
>>>>>>> Many articles tell me that MD5 rowkey or part of it is good method
to
>>>>>>> balance the records stored in different parts. But If I want
to search
>>>>> some
>>>>>>> sequential rowkey records, such as date as rowkey or partially.
I can
>>>>> not
>>>>>>> use rowkey filter to scan a range of date value one time on the
date
>>> by
>>>>>>> MD5. How to balance this issue?
>>>>>>> Thanks.
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Damien HARDY
>>>>> 
>>>>> 
>>> 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message