Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of anoop.hbase@gmail.com
 designates 209.85.216.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALte62zPByzukmcvKoX_CSb3xYeoi=K4Eqe+n=MMPtUR=f=r5Q@mail.gmail.com>
References: 
 <CAAT7MkrhP84FjY5i2TSv9hBpuFJZcrncvGYUq+e59mMeD3bKow@mail.gmail.com>
	<CADcMMgEH=2-7Ps8dF54ybKFtRdx4sOR8njegOmh4C0xzYzd55A@mail.gmail.com>
	<CAPQV63XnS3dWtPKdyixG8qKRGg12Mi-kiUi4rTPONPnVTKTd4w@mail.gmail.com>
	<CAAT7MkqC7Fr2HEmStojaWg=4kV=1MauZ5HtA-xRnsr1ek2K5Sw@mail.gmail.com>
	<CAPQV63Ue2ytozQnO-tB-kXxWG0pXvnYgpRYgxP5RV9cMt2myuQ@mail.gmail.com>
	<CAOtJ30rOpn8WrdtzmXAo60-VuaA_nX+smupA0_FiAtFYe9MAWA@mail.gmail.com>
	<CAAT7MkpAWGgKApMtysCK80AcoHfWX7OiFncF5ZFbXy8VxLvjzg@mail.gmail.com>
	<CAPQV63U1SMfKPKsno=AzfTguS2epwe58PoMjVohBqp6CgADQ1w@mail.gmail.com>
	<CALte62zPByzukmcvKoX_CSb3xYeoi=K4Eqe+n=MMPtUR=f=r5Q@mail.gmail.com>
Date: Fri, 19 Jul 2013 20:42:21 +0530
Message-ID: 
 <CAOtJ30oK7sgP0WADBMZoonzH9g1LWk-_uVrS8evBxjxV-0cDjw@mail.gmail.com>
Subject: Re: DISCUSS : HFile V3 proposal for tags in 0.96
From: Anoop John <anoop.hbase@gmail.com>
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=90e6ba3098a8e1967a04e1dec0e3

--90e6ba3098a8e1967a04e1dec0e3
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

It should be Ted. Tags will be present in KV(Cell).  So whichever part
deals with KVs (Cells) can use the tags and do some thing with that.  Do
some checks in Filter and filter out KVs, or access in CP etc etc.

-Anoop-

On Fri, Jul 19, 2013 at 7:48 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Would tags be visible to methods of BaseRegionObserver, other than
> AccessController ?
>
> Meaning, would other (non-secure) components of HBase be able to use cell
> tagging to store certain information ?
>
> Please clarify.
>
> Thanks
>
> On Fri, Jul 19, 2013 at 6:09 AM, Jean-Marc Spaggiari <
>  jean-marc@spaggiari.org> wrote:
>
> > Thanks Ram and Anoop for those details again. I don't think there is a
> need
> > to be able to revert from V3 to V2. And 1 byte overhead on an HFile is
> not
> > really an overhead. As Anoop proposed, if there is a way to de-activate
> the
> > tags feature when all the KVs in a file are having tag length as zero,
> then
> > it's all good!
> >
> > Looking forward to test that!
> >
> > JM
> >
> > 2013/7/19 ramkrishna vasudevan <ramkrishna.s.vasudevan@gmail.com>
> >
> > > But am afraid that once the user switches to V3 with tags he cannot
> come
> > > back to V2.  If this scenario is possible then we need to see a work
> > around
> > > for that?
> > > Particularly in the case if the user has written the tags and tries t=
o
> > read
> > > it back with V2 then it would not work.
> > >
> > > If user switches to V3 but does not write any tags then if we go with
> the
> > > option of making tags optional using the Fileinfo then atleast after
> the
> > > compaction is done the Hfile could be read with the V2 reader also.
>  But
> > i
> > > don't think the user would intend to do this given the fact that he
> needs
> > > tags for his usecase.
> > >
> > > Regards
> > > Ram
> > >
> > >
> > > On Fri, Jul 19, 2013 at 5:21 PM, Anoop John <anoop.hbase@gmail.com>
> > wrote:
> > >
> > > > Jean
> > > >         When V2 will be used there wont any extra bytes and so no
> > > overhead
> > > > in write or read paths.
> > > > When V3 is used, and there are no tags present at all, we will have
> > extra
> > > > bytes for writing tag length.  Trying to put tag length as VInt so
> that
> > > > this will be 1 byte only.  Then using File infos we can avoid
> overhead.
> > > >
> > > > Say when all the KVs in a file are having tag length as zero( a fil=
er
> > > > trailer indicate this) , during read we can avoid the read and deco=
de
> > of
> > > > teh tag length. Just skip one byte of tag length.
> > > >
> > > > Regarding avoiding the tag length (even the 1 byte fully)  maybe
> during
> > > > compaction it should be possible. But whether really needed I am
> > > thinikng.
> > > > User can select V3 when there is a need for Tags.
> > > >
> > > > -Anoop-
> > > >
> > > > On Fri, Jul 19, 2013 at 4:53 PM, Jean-Marc Spaggiari <
> > > > jean-marc@spaggiari.org> wrote:
> > > >
> > > > > Thanks Ram.
> > > > >
> > > > > One last. Space wise. If I understand correctly, between V2 and V=
3,
> > > when
> > > > > tags are de-activated, there will be only a 1 bit difference, so
> same
> > > > > storage space used. If tags are activated but empty, is it going =
to
> > be
> > > > the
> > > > > same thing? Or are we going to have all the tags overhead? Like c=
an
> > we
> > > > have
> > > > > a byte to say "no tags in that file" in addition to "tags are
> > activated
> > > > for
> > > > > that file"?
> > > > >
> > > > > So 2 questions.
> > > > >
> > > > > 1) what the overhead on disk space from the tags.
> > > > > 2) should we have a flag(bit) per file to say no tags even if
> > activated
> > > > to
> > > > > limit this overhead and ket people activate it for futur uses?
> > > > >
> > > > > JMS
> > > > > Le 2013-07-19 07:11, "ramkrishna vasudevan" <
> > > > > ramkrishna.s.vasudevan@gmail.com> a =E9crit :
> > > > >
> > > > > > >>Based on your details, I think it will be, but very minimal, =
or
> > > > > > almost invisible, correct?
> > > > > > Yes of course.
> > > > > > Regarding migration, any file written with V2 would still be re=
ad
> > > with
> > > > > > HFileReaderV2 and the new files will be written with V3.  So
> there
> > > > should
> > > > > > not be any problem here.  We are anyway testing these things to
> >  make
> > > > > sure
> > > > > > we don't break anywhere.  Thanks Jean for the interest.
> > > > > >
> > > > > > @Stack
> > > > > > I would write up on the changes foreseen for the Codec changes =
to
> > > > support
> > > > > > RPC and HFileV3.
> > > > > > Discussing with Anoop, we have some benefits when the Tags are
> > > written
> > > > as
> > > > > > the byte array and when tags are in memory.  Anyway that i woul=
d
> > > write
> > > > up
> > > > > > in a seperate thread also considering the inputs on the current
> way
> > > the
> > > > > > patch has been made.
> > > > > >
> > > > > > Regards
> > > > > > Ram
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 19, 2013 at 4:32 PM, Jean-Marc Spaggiari <
> > > > > > jean-marc@spaggiari.org> wrote:
> > > > > >
> > > > > > > Like Ted and St.Ack, I read all of this with a great interest
> and
> > > > > > > everything looked good to me.
> > > > > > >
> > > > > > > My only concern will be performance wise.  Even if tags are
> > > disabled,
> > > > > di
> > > > > > > you forsee some performances impacts because everything will
> now
> > > need
> > > > > to
> > > > > > be
> > > > > > > tag aware? Based on your details, I think it will be, but ver=
y
> > > > minimal,
> > > > > > or
> > > > > > > almost invisible, correct?
> > > > > > >
> > > > > > > Also, for migrations from v2 to v3, if v3 is activated, that
> will
> > > be
> > > > > > simply
> > > > > > > done when HFilea will be written, correct? So not really any
> > > > migration
> > > > > > > process required?
> > > > > > >
> > > > > > > JM
> > > > > > > Le 2013-07-19 01:13, "Stack" <stack@duboce.net> a =E9crit :
> > > > > > >
> > > > > > > > On Thu, Jul 18, 2013 at 10:14 AM, ramkrishna vasudevan <
> > > > > > > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > > > > > > > ...
> > > > > > > >
> > > > > > > > >  We can avoid several problems with HFile V2 internals, a=
nd
> > > > > backwards
> > > > > > > > > compatibility concerns, and allow for working tags suppor=
t
> > with
> > > > no
> > > > > > > > > performance impact and low risk to all HBase users who do
> not
> > > > want
> > > > > > tag
> > > > > > > > > support, while still allowing for inline tags capabilitie=
s
> > in a
> > > > > > > shipping
> > > > > > > > > version of HBase, by introducing this in a new V3 version
> for
> > > > > HFile.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > This seems like a good tactic to me.  HFileV2 has the curre=
nt
> > KV
> > > > > format
> > > > > > > > hard-coded all over and trying to 'fix' this would probably
> > take
> > > a
> > > > > > bunch
> > > > > > > of
> > > > > > > > effort and would jeopardize current workings.
> > > > > > > >
> > > > > > > > ....
> > > > > > > >
> > > > > > > > >
> > > > > > > > >  We have been working on this and will have a clean patch
> > with
> > > > good
> > > > > > > > amount
> > > > > > > > > of testing in time for 0.96.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > I'd think that your moving into a green field by doing an
> > hfilev3
> > > > > would
> > > > > > > > make it so your work could run independent of 0.96 timeline=
;
> > i.e.
> > > > it
> > > > > > > could
> > > > > > > > come in post 0.96?
> > > > > > > >
> > > > > > > > What sort of changes do you foresee necessary in core to
> > support
> > > > cell
> > > > > > > > codecs?  Between rpc and hfilev3?
> > > > > > > >
> > > > > > > > Thanks Ram,
> > > > > > > > St.Ack
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--90e6ba3098a8e1967a04e1dec0e3--