Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=ODpKFmXxfj1YYGjWDQpFbi927D6dwPWHGxqbZ9VA0I15Ma3riCrND/XXMCQAJ9cTaKVMPqcZS6sQ4MhyJEUjnzBSx1Y9n+3Dc7wq7IkXgNWezbo/XWi4/7K17xNkqkD/BdczhsPi18T9iLX0X6UmiGYJ55C7Jf6zaMxlej+jTX0=;
References: 
 <CAHtRq2dqQKBROEBt+uGCB+2i6uwyzes8wYoF_g=b_gw4yR065w@mail.gmail.com>
 <2ABF8C7192CD4F68BDDEE0004DA15A01@gmail.com>
 <AF1B2410-E863-4F2A-AED4-FB0A1EBA37AF@gmail.com>
 <CAHtRq2de6Tc6Ssxa6yOUZUCF5hy5p1z1OhfPBT0PiZv1OO5VrA@mail.gmail.com>
Message-ID: <1338973570.11638.YahooMailNeo@web121705.mail.ne1.yahoo.com>
Date: Wed, 6 Jun 2012 02:06:10 -0700 (PDT)
From: lars hofhansl <lhofhansl@yahoo.com>
Reply-To: lars hofhansl <lhofhansl@yahoo.com>
Subject: Re: When does compaction actually occur?
To: "user@hbase.apache.org" <user@hbase.apache.org>
In-Reply-To: 
 <CAHtRq2de6Tc6Ssxa6yOUZUCF5hy5p1z1OhfPBT0PiZv1OO5VrA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi Tom,=0A=0AYou have set MIN_VERSIONS to 1. That tells HBase that for this=
 column family you want to keep=0Aat least 1 version of a cell around regar=
dless of whether it expired (due to TTL) or not.=0AI think if you remove th=
at it will behave as you expect.=0A=0A=0AAs a general rule a compaction wil=
l never influence visibility of data that was inserted before the compactio=
n (except for RAW scans), and hence you should never need to ask when a com=
paction happens - unless you are running out of disk space.=0A=0A=0A-- Lars=
=0A________________________________=0AFrom: Tom Brown <tombrown52@gmail.com=
>=0ATo: user@hbase.apache.org =0ASent: Tuesday, June 5, 2012 2:37 PM=0ASubj=
ect: Re: When does compaction actually occur?=0A=0ALars,=0A=0AIn response t=
o your earlier email, I'm not completely sure whether or=0Anot I'm using a =
raw scan. The scan is performed in a region server=0Acoprocessor initialize=
d as such:=0A=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 Scan scan =3D new Sca=
n()=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 .setMaxVersi=
ons(1)=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 .setTimeR=
ange(myMinTimeStamp, myMaxTimeStamp)=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=
=A0 =C2=A0=C2=A0=C2=A0 .setStartRow(myStartRow)=0A=C2=A0=C2=A0=C2=A0 =C2=A0=
=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 .setStopRow(myStopRow);=0A=C2=A0=C2=A0=C2=
=A0 =C2=A0=C2=A0=C2=A0 scan.setCaching(1000);=0A=0A=C2=A0=C2=A0=C2=A0 =C2=
=A0=C2=A0=C2=A0 InternalScanner scanner =3D ((RegionCoprocessorEnvironment)=
 getEnvironment())=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=
=A0 =C2=A0=C2=A0=C2=A0 .getRegion().getScanner(scan);=0A=0AThe scan is inde=
ed being filtered to the range I provide (using=0AsetTimeRange), but it wil=
l retrieve records much older than should be=0Aallowed given the TTL.=0A=0A=
I have multiple tables setup in a similar fashion, but here a=0Adescription=
 of one of them:=0A=0A{NAME =3D> 'facts', FAMILIES =3D> [{NAME =3D> 'd', BL=
OOMFILTER =3D> 'ROW',=0ACOMPRESSION =3D> 'SNAPPY', VERSIONS =3D> '1', TTL =
=3D> '3600', MIN_VERSIONS=0A=3D> '1'}]}=0A=0A=0AI'm building an OLAP cube f=
or this project and want to make sure the=0Adata size doesn't grow through =
the roof. Whether or not data expires=0Aafter exactly one hour is not an ab=
solute requirement for this use=0Acase. But I want to know why the system i=
s not behaving as I think I=0Aconfigured it to behave.=0A=0AThanks!=0A=0A--=
Tom=0A=0AOn Sun, Jun 3, 2012 at 2:57 AM, Lars George <lars.george@gmail.com=
> wrote:=0A> What Amandeep says and also keep in mind that with the current=
 selection process HBase holds O(log N) files for N data. So say for 2GB re=
gion sizes you get 2-3 files. This means it very "aggressively" is compacti=
ng files, and most of these are "all files included" once... which are the =
promoted to major compactions implicitly. That way your predicate deletes s=
hould be in effect and you will only need scheduled major compactions only =
ever so often.=0A>=0A> Lars=0A>=0A> On Jun 2, 2012, at 1:04 AM, Amandeep Kh=
urana wrote:=0A>=0A>> Tom,=0A>>=0A>> Old cells will get deleted as a part o=
f the next major compaction, which is typically recommended to be done once=
 a day, when the load on the system is at its lowest.=0A>>=0A>> FWIW=E2=80=
=A6 To have a TTL of 3600 take effect, you'll have to do a major compaction=
 every hour, which is an expensive operation specially at scale. Chances ar=
e that your I/O loads will shoot up and latencies will spike for operations=
 to HBase. Can you tell us why a TTL of 3600s is of interest? What are your=
 access patterns?=0A>>=0A>> -Amandeep=0A>>=0A>>=0A>> On Friday, June 1, 201=
2 at 3:59 PM, Tom Brown wrote:=0A>>=0A>>> I have a table that holds rotatin=
g data. It has a TTL of 3600. For=0A>>> some reason, when I scan the table =
I still get old cells that are much=0A>>> older than that TTL.=0A>>>=0A>>> =
I have tried issuing a compaction request via the web UI, but that=0A>>> di=
dn't seem to do anything.=0A>>>=0A>>> Am I misunderstanding the data model =
used by HBase? Is there anything=0A>>> else I can check to verify the funct=
ionality of my integration?=0A>>>=0A>>> I am using HBase 0.92 with Hadoop 1=
.0.2.=0A>>>=0A>>> Thanks in advance!=0A>>>=0A>>> --Tom=0A>>=0A>