hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Cell Encoders and usage of Cell
Date Mon, 22 Apr 2013 22:16:24 GMT
On HBASE-7544 I added encryption to HFile starting with the encoder and
decoder contexts and working outward, so I can attest it is possible to add
features that wrap *around* KV serialization without changing the KV
serialization itself. On trunk the block encoder stuff has been at least
partially factored out so it is not so bad. The HFileV2 code in 0.94 is
more of a hairball.

To pursue an incremental development approach, one option to consider is
making it possible to stack block encoders. Then we might be able to
implement tag support as a block encoder without losing the ability to do
real block encoding. Something else to undo is the inflexible enum that
specifies block encoder types. We should use a short int signature or
something instead. Think of treating block encoders like filters. Then
encoder/decoder stacking could be assembled at runtime from metadata read
out of hfile.

On Mon, Apr 22, 2013 at 12:28 PM, Stack <stack@duboce.net> wrote:

> On Thu, Apr 18, 2013 at 8:19 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
> > My questions were mainly because, if i have the current code  and i would
> > want to introduce tags in it, where would i do it?
> >
> Good question.  If the read/write path currently does not support Cell-only
> access, you are in a bit of a bind.
> It would take some effort converting all to pure Cell.  Most would be just
> grunt work but there are a few tricky spots (according to a quick survey
> done by our LarsH).  If all was Cell on the read/write path, I'd imagine
> it'd be easy enough getting your tags in the mix.
> Until then, you are left w/ some sort of hack on KV.
> > So if i need tags to be introduced should i start changing the HFile
> > formats also and only then i would be getting the tags to work?
> > What do you think here?
> >
> >
> hfiles are versioned so we can up the version when we put a Cell API on it.
>  Might be a bit of work though since will have to bring along the
> compressors and block encoders and currently these are hard-coded to expect
> KV and in particular KVs current serialization.
> St.Ack

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message