hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chia-Ping Tsai"<chia7...@apache.org>
Subject Re: [DISCUSS] Move Type out of KeyValue
Date Fri, 29 Sep 2017 20:28:33 GMT
The custom cell help us to save memory consumption. We don't have own serialization/deserialization
mechanism, hence to transform data from client to server needs many conversion phase (user
data -> Put/Cell -> pb object). The cost of conversion is large in transferring bulk
data. In fact, we also have custom mutation to manage the memory usage of inner cell collection.

On 2017-09-30 02:43, Andrew Purtell <apurtell@apache.org> wrote: 
> What are the use cases for a custom cell? It seems a dangerously low level
> thing to attempt and perhaps we should unwind support for it. But perhaps
> there is a compelling justification.
> 
> 
> On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <chia7712@apache.org>
> wrote:
> 
> > Thanks for all comment.
> >
> > The problem i want to resolve is the valid code should be exposed as
> > IA.Public. Otherwise, end user have to access the IA.Private class to build
> > the custom cell.
> >
> > For example, I have a use case which plays a streaming role in our
> > appliaction. It
> > applies the CellBuilder(HBASE-18519) to build custom cells. These cells
> > have many same fields so they are put in shared-memory for avoiding GC
> > pause. Everything is wonderful. However, we have to access the IA.Private
> > class - KeyValue#Type - to get the valid code of Put.
> >
> > I believe there are many use cases of custom cell, and consequently it is
> > worth adding a way to get the valid type via IA.Public class. Otherwise, it
> > may imply that the custom cell is based on a unstable way, because the
> > related code can be changed at any time.
> > --
> > Chia-Ping
> >
> > On 2017-09-29 00:49, Andrew Purtell <apurtell@apache.org> wrote:
> > > I agree with Stack. Was typing up a reply to Anoop but let me move it
> > down
> > > here.
> > >
> > > The type code exposes some low level details of how our current stores
> > are
> > > architected. But what if in the future you could swap out HStore
> > implements
> > > Store with PStore implements Store, where HStore is backed by HFiles and
> > > PStore is backed by Parquet? Just as a hypothetical example. I know there
> > > would be larger issues if this were actually attempted. Bear with me. You
> > > can imagine some different new Store implementation that has some
> > > advantages but is not a design derived from the log structured merge tree
> > > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > > wouldn't apply to cells from such a thing because they are particular to
> > > how LSMs work. I'm sure such a project if attempted would make a number
> > of
> > > changes requiring a major version increment and low level details could
> > be
> > > unwound from Cell then, but if we could avoid doing it in the first
> > place,
> > > I think it would better for maintainability.
> > >
> > >
> > > On Thu, Sep 28, 2017 at 9:39 AM, Stack <stack@duboce.net> wrote:
> > >
> > > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <chia7712@apache.org>
> > > > wrote:
> > > >
> > > > > hi folks,
> > > > >
> > > > > User is allowed to create custom cell but the valid code of type
-
> > > > > KeyValue#Type - is declared as IA.Private. As i see it, we should
> > expose
> > > > > KeyValue#Type as Public Client. Three possible ways are shown below:
> > > > > 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
> > > > > 2) Move KeyValue#Type into Cell.
> > > > > 3) Move KeyValue#Type to upper level
> > > > >
> > > > > Any suggestions?
> > > > >
> > > > >
> > > > What is the problem that we are trying to solve Chia-Ping? You want to
> > make
> > > > Cells of a new Type?
> > > >
> > > > My first reaction is that KV#Type is particular to the KV
> > implementation.
> > > > Any new Cell implementation should not have to adopt the KeyValue
> > typing
> > > > mechanism.
> > > >
> > > > S
> > > >
> > > >
> > > >
> > > >
> > > > > --
> > > > > Chia-Ping
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >    - A23, Crosstalk
> > >
> >
> 
> 
> 
> -- 
> Best regards,
> Andrew
> 
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>    - A23, Crosstalk
> 

Mime
View raw message