Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=e0jv3RrDnc67ltxgSr/RDqN10QV/y12uTX5UJVgrOcEw/VyF6CcB911wjaFvl//Nk5HWrGeJdbYKyv1OhaiF5e7fLxB5Xj/ppSXfn2IPWHKpL6Xk80aRlTKum82BWXWLqy3B8hDRslpqWQlIdHpGVRYpdPIb/KJcVWyJHKIJd60=;
References: <SNT002-W108E99E544990EECA76F137B5310@phx.gbl>
 <CADK=YxsumgHC07J+C1Z5s0isg7O17f8t6rJPbADjHFdSv5T_TA@mail.gmail.com>
 <BLU0-SMTP404EC8B27950638AA25D47A8F310@phx.gbl>
 <CAA7+SiAzR13bU_U-EeWqLhrX36-A8cUKuB59n9cg=68HM9EWmw@mail.gmail.com>
 <BLU0-SMTP100AA4A54A352F4E5C065958F310@phx.gbl>
 <CAA7+SiCuphmga_NS0Ujt+xiycRqeSC7TCYJ=mdQgt6QfGOAxQg@mail.gmail.com>
 <BLU0-SMTP93F19E706CC0ABE2D97068F310@phx.gbl>
 <1355942277.45748.YahooMailNeo@web140602.mail.bf1.yahoo.com>
 <BLU0-SMTP2096DB1DB1D540C1E492C318F300@phx.gbl>
Message-ID: <1355950319.35752.YahooMailNeo@web140606.mail.bf1.yahoo.com>
Date: Wed, 19 Dec 2012 12:51:59 -0800 (PST)
From: lars hofhansl <lhofhansl@yahoo.com>
Reply-To: lars hofhansl <lhofhansl@yahoo.com>
Subject: Re: Is it necessary to set MD5 on rowkey?
To: "user@hbase.apache.org" <user@hbase.apache.org>
In-Reply-To: <BLU0-SMTP2096DB1DB1D540C1E492C318F300@phx.gbl>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="1905101558-832229341-1355950319=:35752"

--1905101558-832229341-1355950319=:35752
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Doesn't Alex' blog post do that?=0A=0A=0A=0A=0A____________________________=
____=0A From: Michael Segel <michael_segel@hotmail.com>=0ATo: user@hbase.ap=
ache.org; lars hofhansl <lhofhansl@yahoo.com> =0ASent: Wednesday, December =
19, 2012 11:46 AM=0ASubject: Re: Is it necessary to set MD5 on rowkey?=0A =
=0AOk, =0A=0AMaybe I'm missing something.=0AWhy don't you walk me through t=
he use of a salt example. =0A=0A=0AOn Dec 19, 2012, at 12:37 PM, lars hofha=
nsl <lhofhansl@yahoo.com> wrote:=0A=0A> I would disagree here.=0A> It depen=
ds on what you are doing and blanket statements about "this is very, very b=
ad" typically do not help.=0A> =0A> Salting (even round robin) is very nice=
 to distribute write load *and* it gives you a natural way to parallelize s=
cans assuming scans are of reasonable size.=0A> =0A> If the typical use cas=
e is point gets then hashing or inverting keys would be preferable. As usua=
l: It depends.=0A> =0A> -- Lars=0A> =0A> =0A> =0A> ________________________=
________=0A> From: Michael Segel <michael_segel@hotmail.com>=0A> To: user@h=
base.apache.org =0A> Sent: Tuesday, December 18, 2012 3:29 PM=0A> Subject: =
Re: Is it necessary to set MD5 on rowkey?=0A> =0A> Alex, =0A> And that's th=
e point. Salt as you explain it conceptually implies that the number you ar=
e adding to the key to ensure a better distribution means that you will hav=
e inefficiencies in terms of scans and gets. =0A> =0A> Using a hash as eith=
er the full key, or taking the hash, truncating it and appending the key ma=
y screw up scans, but your get() is intact. =0A> =0A> There are other optio=
ns like inverting the numeric key ... =0A> =0A> And of course doing nothing=
. =0A> =0A> Using a salt as part of the design pattern is bad. =0A> =0A> Wi=
th respect to the OP, I was discussing the use of hash and some alternative=
s to how to implement the hash of a key. =0A> Again, doing nothing may also=
 make sense too, if you understand the risks and you know how your data is =
going to be used.=0A> =0A> =0A> On Dec 18, 2012, at 11:36 AM, Alex Baranau =
<alex.baranov.v@gmail.com> wrote:=0A> =0A>> Mike,=0A>> =0A>> Please read *f=
ull post* before judge. In particular, "Hash-based=0A>> distribution" secti=
on. You can find the same in HBaseWD small README file=0A>> [1] (not sure i=
f you read it at all before commenting on the lib). Round=0A>> robin is mai=
nly for explaining the concept/idea (though not only for that).=0A>> =0A>> =
Thank you,=0A>> Alex Baranau=0A>> ------=0A>> Sematext :: http://blog.semat=
ext.com/ :: Hadoop - HBase - ElasticSearch -=0A>> Solr=0A>> =0A>> [1] https=
://github.com/sematext/HBaseWD=0A>> =0A>> On Tue, Dec 18, 2012 at 12:24 PM,=
 Michael Segel=0A>> <michael_segel@hotmail.com>wrote:=0A>> =0A>>> Quick ans=
wer...=0A>>> =0A>>> Look at the salt.=0A>>> Its just a number from a round =
robin counter.=0A>>> There is no tie between the salt and row.=0A>>> =0A>>>=
 So when you want to fetch a single row, how do you do it?=0A>>> ...=0A>>> =
;-)=0A>>> =0A>>> On Dec 18, 2012, at 11:12 AM, Alex Baranau <alex.baranov.v=
@gmail.com>=0A>>> wrote:=0A>>> =0A>>>> Hello,=0A>>>> =0A>>>> @Mike:=0A>>>> =
=0A>>>> I'm the author of that post :).=0A>>>> =0A>>>> Quick reply to your =
last comment:=0A>>>> =0A>>>> 1) Could you please describe why "the use of a=
 'Salt' is a very, very bad=0A>>>> idea" in more specific way than "Fetchin=
g data takes more effort". Would=0A>>> be=0A>>>> helpful for anyone who is =
looking into using this approach.=0A>>>> =0A>>>> 2) The approach described =
in the post also says you can prefix with the=0A>>>> hash, you probably mis=
sed that.=0A>>>> =0A>>>> 3) I believe your answer, "use MD5 or SHA-1" doesn=
't help bigdata guy.=0A>>>> Please re-read the question: the intention is t=
o distribute the load=0A>>> while=0A>>>> still being able to do "partial ke=
y scans". The blog post linked above=0A>>>> explains one possible solution =
for that, while your answer doesn't.=0A>>>> =0A>>>> @bigdata:=0A>>>> =0A>>>=
> Basically when it comes to solving two issues: distributing writes and=0A=
>>>> having ability to read data sequentially, you have to balance between=
=0A>>> being=0A>>>> good at both of them. Very good presentation by Lars:=
=0A>>>> =0A>>> http://www.slideshare.net/larsgeorge/hbase-advanced-schema-d=
esign-berlin-buzzwords-june-2012=0A>>> ,=0A>>>> slide 22. You will see how =
this is correlated. In short:=0A>>>> * having md5/other hash prefix of the =
key does better w.r.t. distributing=0A>>>> writes, while compromises abilit=
y to do range scans efficiently=0A>>>> * having very limited number of 'sal=
t' prefixes still allows to do range=0A>>>> scans (less efficiently than no=
rmal range scans, of course, but still=0A>>> good=0A>>>> enough in many cas=
es) while providing worse distribution of writes=0A>>>> =0A>>>> In the latt=
er case by choosing number of possible 'salt' prefixes (which=0A>>>> could =
be derived from hashed values, etc.) you can balance between=0A>>>> distrib=
uting writes efficiency and ability to run fast range scans.=0A>>>> =0A>>>>=
 Hope this helps=0A>>>> =0A>>>> Alex Baranau=0A>>>> ------=0A>>>> Sematext =
:: http://blog.sematext.com/ :: Hadoop - HBase - ElasticSearch=0A>>> -=0A>>=
>> Solr=0A>>>> =0A>>>> On Tue, Dec 18, 2012 at 8:52 AM, Michael Segel <=0A>=
>> michael_segel@hotmail.com>wrote:=0A>>>> =0A>>>>> =0A>>>>> Hi,=0A>>>>> =
=0A>>>>> First, the use of a 'Salt' is a very, very bad idea and I would re=
ally=0A>>>>> hope that the author of that blog take it down.=0A>>>>> While =
it may solve an initial problem in terms of region hot spotting,=0A>>> it=
=0A>>>>> creates another problem when it comes to fetching data. Fetching d=
ata=0A>>> takes=0A>>>>> more effort.=0A>>>>> =0A>>>>> With respect to using=
 a hash (MD5 or SHA-1) you are creating a more=0A>>> random=0A>>>>> key tha=
t is unique to the record.=A0 Some would argue that using MD5 or=0A>>> SHA-=
1=0A>>>>> that mathematically you could have a collision, however you could=
 then=0A>>>>> append the key to the hash to guarantee uniqueness. You could=
 also do=0A>>>>> things like take the hash and then truncate it to the firs=
t byte and=0A>>> then=0A>>>>> append the record key. This should give you e=
nough randomness to avoid=0A>>> hot=0A>>>>> spotting after the initial regi=
on completion and you could pre-split out=0A>>>>> any number of regions. (F=
irst byte 0-255 for values, so you can program=0A>>> the=0A>>>>> split...=
=0A>>>>> =0A>>>>> =0A>>>>> Having said that... yes, you lose the ability to=
 perform a sequential=0A>>> scan=0A>>>>> of the data.=A0 At least to a poin=
t.=A0 It depends on your schema.=0A>>>>> =0A>>>>> Note that you need to thi=
nk about how you are primarily going to access=0A>>>>> the data.=A0 You can=
 then determine the best way to store the data to gain=0A>>>>> the best per=
formance. For some applications... the region hot spotting=0A>>>>> isn't an=
 important issue.=0A>>>>> =0A>>>>> Note YMMV=0A>>>>> =0A>>>>> HTH=0A>>>>> =
=0A>>>>> -Mike=0A>>>>> =0A>>>>> On Dec 18, 2012, at 3:33 AM, Damien Hardy <=
dhardy@viadeoteam.com>=0A>>> wrote:=0A>>>>> =0A>>>>>> Hello,=0A>>>>>> =0A>>=
>>>> There is middle term betwen sequecial keys (hot spoting risk) and md5=
=0A>>>>>> (heavy scan):=0A>>>>>> * you can use composed keys with a field t=
hat can segregate data=0A>>>>>> (hostname, productname, metric name) like O=
penTSDB=0A>>>>>> * or use Salt with a limited number of values (example=0A>=
>>>>> substr(md5(rowid),0,1) =3D 16 values)=0A>>>>>>=A0  so that a scan is =
a combination of 16 filters on on each salt values=0A>>>>>>=A0  you can bas=
e your code on HBaseWD by sematext=0A>>>>>> =0A>>>>>> =0A>>>>> =0A>>> http:=
//blog.sematext.com/2012/04/09/hbasewd-avoid-regionserver-hotspotting-despi=
te-writing-records-with-sequential-keys/=0A>>>>>>=A0 =A0 https://github.com=
/sematext/HBaseWD=0A>>>>>> =0A>>>>>> Cheers,=0A>>>>>> =0A>>>>>> =0A>>>>>> 2=
012/12/18 bigdata <bigdatabase@outlook.com>=0A>>>>>> =0A>>>>>>> Many articl=
es tell me that MD5 rowkey or part of it is good method to=0A>>>>>>> balanc=
e the records stored in different parts. But If I want to search=0A>>>>> so=
me=0A>>>>>>> sequential rowkey records, such as date as rowkey or partially=
. I can=0A>>>>> not=0A>>>>>>> use rowkey filter to scan a range of date val=
ue one time on the date=0A>>> by=0A>>>>>>> MD5. How to balance this issue?=
=0A>>>>>>> Thanks.=0A>>>>>>> =0A>>>>>>> =0A>>>>>> =0A>>>>>> =0A>>>>>> =0A>>=
>>>> =0A>>>>>> --=0A>>>>>> Damien HARDY=0A>>>>> =0A>>>>> =0A>>> 
--1905101558-832229341-1355950319=:35752--