Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of anoopsj@huawei.com designates
 119.145.14.65 as permitted sender)
From: Anoop Sam John <anoopsj@huawei.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Subject: RE: Where is code in hbase that physically delete a record?
Thread-Topic: Where is code in hbase that physically delete a record?
Thread-Index: AQHNrHzjJeSvrUeyDUeoDagqjRtSRZe+Y3D8
Date: Thu, 18 Oct 2012 03:32:20 +0000
Message-ID: 
 <0CE69E9126D0344088798A3B7F7F80863A4E45AF@szxeml531-mbx.china.huawei.com>
References: 
 <CALxHxa0-Y5Qre+Cu-Hr1oERvPP_Q9rqa3mhz7=dVcK6UfqekbA@mail.gmail.com>
	<507ea940.8559420a.4715.ffffb94cSMTPIN_ADDED@mx.google.com>,<CALxHxa3_v17E2jw5qaKq-pXesXZDZ3NgS1985j+ZBrdPcHmwcA@mail.gmail.com>
In-Reply-To: 
 <CALxHxa3_v17E2jw5qaKq-pXesXZDZ3NgS1985j+ZBrdPcHmwcA@mail.gmail.com>
Accept-Language: en-US, zh-CN
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Yun,
             We have preCompactScannerOpen() and preCompact() hooks..=20
As we said, for compaction, a scanner for reading all corresponding HFiles =
( all HFiles in major compaction) will be created and scan via that scanner=
.. ( calling next() methods).. The kernel will do this way..=20
Now using these hooks you can create a wrapper over the actual scanner... I=
n fact you can use preCompact() hook(I think that is fine for you).. By the=
 time this is being called,  the actual scanner is made and will get that o=
bject passed to your hook... You can create  a custom scanner impl and wrap=
 the actual scanner within that and return the new wrapper scanner from you=
r post hook.. [Yes its return type is InternalScanner]  The actual scanner =
you can use as a delegator to do the actual scanning purpose... Now all the=
 KVs ( which the underlying scanner passed) will flow via ur new wrapper sc=
anner where you can avoid certain KVs based on your condition or logic

Core                 WrapperScannerImpl                      Actual Scanner=
 (created by core)
        ->                 next(List<KeyValue>)     ->         next(List<Ke=
yValue>) =20
                                                                    <-     =
    Do the real scan from HFiles
                          See List KVs and remove=20
                           those u dont want
                                 <-
Only the passed
KVs come in final
merged file
          =20
Hope I make it clear for you :)

Note : - preCompactScannerOpen() will be called before even creating the ac=
tual scanner while preCompact() after this scanner creation.. You can see t=
he code in Store#compactStore()

-Anoop-
________________________________________
From: yun peng [pengyunmomo@gmail.com]
Sent: Wednesday, October 17, 2012 9:04 PM
To: user@hbase.apache.org
Subject: Re: Where is code in hbase that physically delete a record?

Hi, Ram and Anoop, Thanks for the nice reference on the java file, which I
will check through.

It is interesting to know about the recent feature on
preCompactScannerOpen() hook. Ram, it would be nice if I can know how to
specify conditions like c1 =3D 'a'.  I have also checked the example code i=
n
hbase 6496 link <https://issues.apache.org/jira/browse/HBASE-6496>. which
show how to delete data before time as in a on-demand specification...
Cheers,
Yun

On Wed, Oct 17, 2012 at 8:46 AM, Ramkrishna.S.Vasudevan <
ramkrishna.vasudevan@huawei.com> wrote:

> Also to see the code how the delete happens pls refer to StoreScanner.jav=
a
> and how the ScanQueryMatcher.match() works.
>
> That is where we decide if any kv has to be avoided due to already delete=
d
> tombstone marker.
>
> Forgot to tell you about this.
>
> Regards
> Ram
>
> > -----Original Message-----
> > From: yun peng [mailto:pengyunmomo@gmail.com]
> > Sent: Wednesday, October 17, 2012 5:54 PM
> > To: user@hbase.apache.org
> > Subject: Where is code in hbase that physically delete a record?
> >
> > Hi, All,
> > I want to find internal code in hbase where physical deleting a record
> > occurs.
> >
> > -some of my understanding.
> > Correct me if I am wrong. (It is largely based on my experience and
> > even
> > speculation.) Logically deleting a KeyValue data in hbase is performed
> > by
> > marking tombmarker (by Delete() per records) or setting TTL/max_version
> > (per Store). After these actions, however, the physical data are still
> > there, somewhere in the system. Physically deleting a record in hbase
> > is
> > realised by *a scanner to discard a keyvalue data record* during the
> > major_compact.
> >
> > -what I need
> > I want to extend hbase to associate some actions with physically
> > deleting a
> > record. Does hbase provide such hook (or coprocessor API) to inject
> > code
> > for each KV record that is skipped by hbase storescanner in
> > major_compact.
> > If not, anyone knows where should I look into in hbase (-0.94.2) for
> > such
> > code modification?
> >
> > Thanks.
> > Yun
>
>=