Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-RocketYMMF:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=cRDzZN1U+SxknlOXE/jeq+aWOuFsdP6e90Hu1tBLTASjc0iH5o16QtPU3V6n1pHQarsjybR0xaxCfNMalZlwW5JGaGqGV8eRJ9IjMMIMQms0yYkYvGMZYdEJa9gdRQZScpXX00LYWfZAQOjftGXTPouZLgSq24c0KE9QToj1/QE=;
References: <1361601624.74417.YahooMailNeo@web140602.mail.bf1.yahoo.com>
 <CADcMMgFxHYZsCL6-xBDrB6TcotzszcwgDZWRCZtWxNkNxGB2gQ@mail.gmail.com>
 <CAB5sDNKZv9bvh5hvj0ct4XMWF=MXoFzh+5R6c+y9hU_hR+0vgw@mail.gmail.com>
 <CA+RK=_DKfo3Rfoj2_feEfSkPi_8xpTu7YHfdRuzTrvDJO0PdQQ@mail.gmail.com>
 <CAMUu0w_pmXD2wkcuOUgR_WW8nsXeq8+_D7UDgQC13eb+NiN+cw@mail.gmail.com>
Message-ID: <1361924833.48368.YahooMailNeo@web140604.mail.bf1.yahoo.com>
Date: Tue, 26 Feb 2013 16:27:13 -0800 (PST)
From: lars hofhansl <larsh@apache.org>
Reply-To: lars hofhansl <larsh@apache.org>
Subject: Re: Simple stastics per region
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
In-Reply-To: 
 <CAMUu0w_pmXD2wkcuOUgR_WW8nsXeq8+_D7UDgQC13eb+NiN+cw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="1023258982-1343913261-1361924833=:48368"

--1023258982-1343913261-1361924833=:48368
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Just had a discussion with the Phoenix folks (my cubicle neighbors :) ).=0A=
Turns out that the types of problem we're trying to solve for Phoenix would=
 need equal-depth histograms, whereas for decisions such as picking a 2ndar=
y index equal-width histograms are often used.=0ASo a key in this is a prop=
er framework through, which, stats can hooked up and calculated. OSGi for c=
oprocessors would be nice, but may also be overkill for this.=0AMaybe somet=
hing like the chores framework would work.=0A=0AIn either case, there will =
be core stats (that would allow HBase to decide between a scan and a multi =
get), and user defined stats to help higher layers such as Phoenix, or an i=
ndexing library.=0A=0A=0A-- Lars=0A=0A=0A=0A_______________________________=
_=0A From: Enis S=F6ztutar <enis.soz@gmail.com>=0ATo: "dev@hbase.apache.org=
" <dev@hbase.apache.org> =0ASent: Tuesday, February 26, 2013 4:15 PM=0ASubj=
ect: Re: Simple stastics per region=0A =0A+1 for core. I can see that histo=
grams might help us in automatic splits=0Aand merges as well.=0A=0A=0AOn Tu=
e, Feb 26, 2013 at 3:27 PM, Andrew Purtell <apurtell@apache.org> wrote:=0A=
=0A> If this is going to be a CP then other CPs need an easy way to use the=
=0A> output stats. If a subsequent proposal from core requires statistics f=
rom=0A> this CP does that then mandate it itself must be a CP? What if that=
 can't=0A> work?=0A>=0A> Putting the stats into a table addresses the first=
 concern.=0A>=0A> For the second, it is an issue that comes up I think when=
 building a=0A> generally useful shared function as a CP. Please consider i=
nserting my=0A> earlier comments about OSGi here, in that we trend toward a=
 real module=0A> system if we're not careful (unless that is the aim).=0A>=
=0A>=0A> On Tue, Feb 26, 2013 at 2:31 PM, Jesse Yates <jesse.k.yates@gmail.=
com=0A> >wrote:=0A>=0A> > TL;DR Making it part of the UI and ensuring that =
you don't load things=0A> the=0A> > wrong way seem to be the only reasons f=
or making this part of core -=0A> > certainly not bad reasons. They are fai=
rly easy to handle as a CP though,=0A> > so maybe its not necessary immedia=
tely.=0A> >=0A> > I ended up writing a simple stats framework last week (ok=
, its like 6=0A> > classes) that makes it easy to create your own stats for=
 a table. Its all=0A> > coprocessor based, and as Lars suggested, hooks up =
to the major=0A> compactions=0A> > to let you build per-column-per-region s=
tats and writes it to a 'system'=0A> > table =3D "_stats_".=0A> >=0A> > Wit=
h the framework you could easily write your own custom stats, from=0A> > si=
mple things like min/max keys to things like fixed width or fixed depth=0A>=
 > histograms, or even more complicated. There has been some internal=0A> >=
 discussion around how to make this available to the community (as part of=
=0A> > Phoenix, core in HBase, an independent github project, ...?).=0A> >=
=0A> > The biggest isssue around having it all CP based is that you need to=
 be=0A> > really careful to ensure that it comes _after_ all the other comp=
action=0A> > coprocessors. This way you know exactly what keys come out and=
 have=0A> correct=0A> > statistics (for that point in time). Not a huge iss=
ue - you just need to=0A> be=0A> > careful. Baking the stats framework into=
 HBase is really nice in that we=0A> > can be sure we never mess this up.=
=0A> >=0A> > Building it into the core of HBase isn't going to get us per-r=
egion=0A> > statistics without a whole bunch of pain - compactions per stor=
e make=0A> this=0A> > a pain to actualize; there isn't a real advantage her=
e, as I'd like to=0A> keep=0A> > it per CF, if only not to change all the t=
hings.=0A> >=0A> > Further, this would be a great first use-case for real s=
ystem tables.=0A> > Mixing this data with .META. is going to be a bit of a =
mess, especially=0A> for=0A> > doing clean scans, etc. to read the stats. A=
lso, I'd be gravely concerned=0A> > to muck with such important state, espe=
cially if we make a 'statistic' a=0A> > pluggable element (so people can ea=
sily expand their own).=0A> >=0A> > And sure, we could make it make pretty =
graphs on the UI, no harm in it=0A> and=0A> > very little overhead :)=0A> >=
=0A> > -------------------=0A> > Jesse Yates=0A> > @jesse_yates=0A> > jyate=
s.github.com=0A> >=0A> >=0A> > On Tue, Feb 26, 2013 at 2:08 PM, Stack <stac=
k@duboce.net> wrote:=0A> >=0A> > > On Fri, Feb 22, 2013 at 10:40 PM, lars h=
ofhansl <larsh@apache.org>=0A> > wrote:=0A> > >=0A> > > > This topic comes =
up now and then (see recent discussion about=0A> > translating=0A> > > > mu=
lti Gets into Scan+Filter).=0A> > > >=0A> > > > It's not that hard to keep =
statistics as part of compactions.=0A> > > > I envision two knobs:=0A> > > =
> 1. Max number of distinct values to track directly. If a column has=0A> >=
 less=0A> > > > this # of values, keep track of their occurrences explicitl=
y.=0A> > > > 2. Number of (equal width) histogram partitions to maintain.=
=0A> > > >=0A> > > > Statistics would be kept per store (i.e. per region pe=
r column=0A> family)=0A> > > and=0A> > > > stored into an HBase table (one =
row per store).Initially we could=0A> just=0A> > > > support major compacti=
ons that atomically insert a new version of=0A> that=0A> > > > statistics f=
or the store.=0A> > > >=0A> > > >=0A> > > Sounds great.=0A> > >=0A> > > In =
.META. add columns for each each cf on each region row?=A0 Or another=0A> >=
 > table?=0A> > >=0A> > > What kind of stats would you keep?=A0 Would they =
be useful for operators?=0A> >=A0 Or=0A> > > just for stuff like say Phoeni=
x making decisions?=0A> > >=0A> > >=0A> > >=0A> > > > An simple implementat=
ion (not knowing ahead of time how many values=0A> it=0A> > > > will see du=
ring the compaction) could start by keeping track of=0A> > > individual=0A>=
 > > > values for columns. If it gets past the max # of distinct values to=
=0A> > > track,=0A> > > > start with equal width histograms (using the dist=
inct values picket=0A> up=0A> > so=0A> > > > far to estimate an initial par=
tition width).=0A> > > > If the number of partition gets larger than what w=
as configured it=0A> > would=0A> > > > increase the width and merge the pre=
vious counts into the new width=0A> > > (which=0A> > > > means the new part=
ition width must be a multiple of the previous=0A> size).=0A> > > > There's=
 probably a lot of other fanciness that could be used here=0A> > > (haven't=
=0A> > > > spent a lot of time thinking about details).=0A> > > >=0A> > > >=
=0A> > > > Is this something that should be in core HBase or rather be=0A> =
implemented=0A> > > as=0A> > > > coprocessor?=0A> > > >=0A> > >=0A> > >=0A>=
 > > I think it could go in core if it generated pretty pictures.=0A> > >=
=0A> > > St.Ack=0A> > >=0A> >=0A>=0A>=0A>=0A> --=0A> Best regards,=0A>=0A>=
=A0 =A0 - Andy=0A>=0A> Problems worthy of attack prove their worth by hitti=
ng back. - Piet Hein=0A> (via Tom White)=0A>
--1023258982-1343913261-1361924833=:48368--