Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <pony-51ef25fd0a3611d18ffb478ae2ae610bbbd2432c-837448e76e4b0889484fa783ca7225d999a6f25b@dev.hbase.apache.org>
References: <pony-51ef25fd0a3611d18ffb478ae2ae610bbbd2432c-22a75ff782d230860eccca242e93cd918fcdf2df@dev.hbase.apache.org>
 <CADcMMgFYGAFYnP10-iO_8HVZyXTGk-3P53fnkN6FNd5+ezzA6g@mail.gmail.com>
 <CA+RK=_Ae5wCr5Q+da174beJHcQ=OH2OQrPn-G=L=xZ8aYfHVjA@mail.gmail.com>
 <pony-51ef25fd0a3611d18ffb478ae2ae610bbbd2432c-e6cb0bdffcf3c766389ac31d0201c0ca2113a5c7@dev.hbase.apache.org>
 <CA+RK=_CNCYs+QCwLU3pcPLqKHwJCovhhmfiVht0B6=r3KP7zZQ@mail.gmail.com>
 <pony-51ef25fd0a3611d18ffb478ae2ae610bbbd2432c-0e7b8c2715c687d60717c3a14515dc4110190f41@dev.hbase.apache.org>
 <CA+RK=_CYMzYTU87mv+jHdv2UAB+QaEWRx9P48LwMkpk2Hh0xtQ@mail.gmail.com> <pony-51ef25fd0a3611d18ffb478ae2ae610bbbd2432c-837448e76e4b0889484fa783ca7225d999a6f25b@dev.hbase.apache.org>
From: Andrew Purtell <apurtell@apache.org>
Date: Fri, 29 Sep 2017 15:05:42 -0700
Message-ID: <CA+RK=_B5=pgVne1_2pF6Uwti2eMoid6d3Rusoh95SPGp8M2Y+A@mail.gmail.com>
Subject: Re: [DISCUSS] Move Type out of KeyValue
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Content-Type: multipart/alternative; boundary="f4030437aa24424062055a5b3cdb"
archived-at: Fri, 29 Sep 2017 22:06:27 -0000

--f4030437aa24424062055a5b3cdb
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=E2=80=8BConstruct a normal put or delete or batch mutation, add whatever e=
xtra
state you need in one or more operation attributes, and use a
regionobserver to extend normal processing to handle the extra state. I'm
curious what dispatching to extension code because of a custom cell type
buys you over dispatching to extension code because of the presence of an
attribute (or cell tag). For example, in security coprocessors we take
attribute data and attach it to the cell using cell tags. Later we check
for cell tag(s) to determine if we have to take special action when the
cell is accessed by a scanner, or during some operations (e.g. appends or
increments have to do extra handling for cell security tags).


On Fri, Sep 29, 2017 at 2:43 PM, Chia-Ping Tsai <chia7712@apache.org> wrote=
:

> > Instead of a custom cell, could you use a regular cell with a custom
> > operation attribute (see OperationWithAttributes).
> Pardon me, I didn't get what you said.
>
>
>
> On 2017-09-30 04:31, Andrew Purtell <apurtell@apache.org> wrote:
> > Instead of a custom cell, could you use a regular cell with a custom
> > operation attribute (see OperationWithAttributes).
> >
> > On Fri, Sep 29, 2017 at 1:28 PM, Chia-Ping Tsai <chia7712@apache.org>
> wrote:
> >
> > > The custom cell help us to save memory consumption. We don't have own
> > > serialization/deserialization mechanism, hence to transform data from
> > > client to server needs many conversion phase (user data -> Put/Cell -=
>
> pb
> > > object). The cost of conversion is large in transferring bulk data. I=
n
> > > fact, we also have custom mutation to manage the memory usage of inne=
r
> cell
> > > collection.
> > >
> > > On 2017-09-30 02:43, Andrew Purtell <apurtell@apache.org> wrote:
> > > > What are the use cases for a custom cell? It seems a dangerously lo=
w
> > > level
> > > > thing to attempt and perhaps we should unwind support for it. But
> perhaps
> > > > there is a compelling justification.
> > > >
> > > >
> > > > On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <
> chia7712@apache.org>
> > > > wrote:
> > > >
> > > > > Thanks for all comment.
> > > > >
> > > > > The problem i want to resolve is the valid code should be exposed
> as
> > > > > IA.Public. Otherwise, end user have to access the IA.Private clas=
s
> to
> > > build
> > > > > the custom cell.
> > > > >
> > > > > For example, I have a use case which plays a streaming role in ou=
r
> > > > > appliaction. It
> > > > > applies the CellBuilder(HBASE-18519) to build custom cells. These
> cells
> > > > > have many same fields so they are put in shared-memory for
> avoiding GC
> > > > > pause. Everything is wonderful. However, we have to access the
> > > IA.Private
> > > > > class - KeyValue#Type - to get the valid code of Put.
> > > > >
> > > > > I believe there are many use cases of custom cell, and
> consequently it
> > > is
> > > > > worth adding a way to get the valid type via IA.Public class.
> > > Otherwise, it
> > > > > may imply that the custom cell is based on a unstable way, becaus=
e
> the
> > > > > related code can be changed at any time.
> > > > > --
> > > > > Chia-Ping
> > > > >
> > > > > On 2017-09-29 00:49, Andrew Purtell <apurtell@apache.org> wrote:
> > > > > > I agree with Stack. Was typing up a reply to Anoop but let me
> move it
> > > > > down
> > > > > > here.
> > > > > >
> > > > > > The type code exposes some low level details of how our current
> > > stores
> > > > > are
> > > > > > architected. But what if in the future you could swap out HStor=
e
> > > > > implements
> > > > > > Store with PStore implements Store, where HStore is backed by
> HFiles
> > > and
> > > > > > PStore is backed by Parquet? Just as a hypothetical example. I
> know
> > > there
> > > > > > would be larger issues if this were actually attempted. Bear wi=
th
> > > me. You
> > > > > > can imagine some different new Store implementation that has so=
me
> > > > > > advantages but is not a design derived from the log structured
> merge
> > > tree
> > > > > > if you like. Most values from a new Cell.Type based on
> KeyValue.Type
> > > > > > wouldn't apply to cells from such a thing because they are
> > > particular to
> > > > > > how LSMs work. I'm sure such a project if attempted would make =
a
> > > number
> > > > > of
> > > > > > changes requiring a major version increment and low level detai=
ls
> > > could
> > > > > be
> > > > > > unwound from Cell then, but if we could avoid doing it in the
> first
> > > > > place,
> > > > > > I think it would better for maintainability.
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 28, 2017 at 9:39 AM, Stack <stack@duboce.net> wrote=
:
> > > > > >
> > > > > > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> > > chia7712@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > hi folks,
> > > > > > > >
> > > > > > > > User is allowed to create custom cell but the valid code of
> type
> > > -
> > > > > > > > KeyValue#Type - is declared as IA.Private. As i see it, we
> should
> > > > > expose
> > > > > > > > KeyValue#Type as Public Client. Three possible ways are sho=
wn
> > > below:
> > > > > > > > 1) Change declaration of KeyValue#Type from IA.Private to
> > > IA.Public
> > > > > > > > 2) Move KeyValue#Type into Cell.
> > > > > > > > 3) Move KeyValue#Type to upper level
> > > > > > > >
> > > > > > > > Any suggestions?
> > > > > > > >
> > > > > > > >
> > > > > > > What is the problem that we are trying to solve Chia-Ping? Yo=
u
> > > want to
> > > > > make
> > > > > > > Cells of a new Type?
> > > > > > >
> > > > > > > My first reaction is that KV#Type is particular to the KV
> > > > > implementation.
> > > > > > > Any new Cell implementation should not have to adopt the
> KeyValue
> > > > > typing
> > > > > > > mechanism.
> > > > > > >
> > > > > > > S
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > --
> > > > > > > > Chia-Ping
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Andrew
> > > > > >
> > > > > > Words like orphans lost among the crosstalk, meaning torn from
> > > truth's
> > > > > > decrepit hands
> > > > > >    - A23, Crosstalk
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > >    - A23, Crosstalk
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >    - A23, Crosstalk
> >
>


--=20
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

--f4030437aa24424062055a5b3cdb--