hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: [DISCUSS] Move Type out of KeyValue
Date Fri, 29 Sep 2017 20:31:34 GMT
Instead of a custom cell, could you use a regular cell with a custom
operation attribute (see OperationWithAttributes).

On Fri, Sep 29, 2017 at 1:28 PM, Chia-Ping Tsai <chia7712@apache.org> wrote:

> The custom cell help us to save memory consumption. We don't have own
> serialization/deserialization mechanism, hence to transform data from
> client to server needs many conversion phase (user data -> Put/Cell -> pb
> object). The cost of conversion is large in transferring bulk data. In
> fact, we also have custom mutation to manage the memory usage of inner cell
> collection.
>
> On 2017-09-30 02:43, Andrew Purtell <apurtell@apache.org> wrote:
> > What are the use cases for a custom cell? It seems a dangerously low
> level
> > thing to attempt and perhaps we should unwind support for it. But perhaps
> > there is a compelling justification.
> >
> >
> > On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <chia7712@apache.org>
> > wrote:
> >
> > > Thanks for all comment.
> > >
> > > The problem i want to resolve is the valid code should be exposed as
> > > IA.Public. Otherwise, end user have to access the IA.Private class to
> build
> > > the custom cell.
> > >
> > > For example, I have a use case which plays a streaming role in our
> > > appliaction. It
> > > applies the CellBuilder(HBASE-18519) to build custom cells. These cells
> > > have many same fields so they are put in shared-memory for avoiding GC
> > > pause. Everything is wonderful. However, we have to access the
> IA.Private
> > > class - KeyValue#Type - to get the valid code of Put.
> > >
> > > I believe there are many use cases of custom cell, and consequently it
> is
> > > worth adding a way to get the valid type via IA.Public class.
> Otherwise, it
> > > may imply that the custom cell is based on a unstable way, because the
> > > related code can be changed at any time.
> > > --
> > > Chia-Ping
> > >
> > > On 2017-09-29 00:49, Andrew Purtell <apurtell@apache.org> wrote:
> > > > I agree with Stack. Was typing up a reply to Anoop but let me move it
> > > down
> > > > here.
> > > >
> > > > The type code exposes some low level details of how our current
> stores
> > > are
> > > > architected. But what if in the future you could swap out HStore
> > > implements
> > > > Store with PStore implements Store, where HStore is backed by HFiles
> and
> > > > PStore is backed by Parquet? Just as a hypothetical example. I know
> there
> > > > would be larger issues if this were actually attempted. Bear with
> me. You
> > > > can imagine some different new Store implementation that has some
> > > > advantages but is not a design derived from the log structured merge
> tree
> > > > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > > > wouldn't apply to cells from such a thing because they are
> particular to
> > > > how LSMs work. I'm sure such a project if attempted would make a
> number
> > > of
> > > > changes requiring a major version increment and low level details
> could
> > > be
> > > > unwound from Cell then, but if we could avoid doing it in the first
> > > place,
> > > > I think it would better for maintainability.
> > > >
> > > >
> > > > On Thu, Sep 28, 2017 at 9:39 AM, Stack <stack@duboce.net> wrote:
> > > >
> > > > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> chia7712@apache.org>
> > > > > wrote:
> > > > >
> > > > > > hi folks,
> > > > > >
> > > > > > User is allowed to create custom cell but the valid code of
type
> -
> > > > > > KeyValue#Type - is declared as IA.Private. As i see it, we should
> > > expose
> > > > > > KeyValue#Type as Public Client. Three possible ways are shown
> below:
> > > > > > 1) Change declaration of KeyValue#Type from IA.Private to
> IA.Public
> > > > > > 2) Move KeyValue#Type into Cell.
> > > > > > 3) Move KeyValue#Type to upper level
> > > > > >
> > > > > > Any suggestions?
> > > > > >
> > > > > >
> > > > > What is the problem that we are trying to solve Chia-Ping? You
> want to
> > > make
> > > > > Cells of a new Type?
> > > > >
> > > > > My first reaction is that KV#Type is particular to the KV
> > > implementation.
> > > > > Any new Cell implementation should not have to adopt the KeyValue
> > > typing
> > > > > mechanism.
> > > > >
> > > > > S
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > --
> > > > > > Chia-Ping
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > >    - A23, Crosstalk
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >    - A23, Crosstalk
> >
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message