Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Received-SPF: pass (athena.apache.org: domain of jwills@cloudera.com
 designates 209.85.216.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAA5C_ptLiDbvah--ffjmG5Tm_e+HMsuNOJ2SiAcSJJqJQNrcbQ@mail.gmail.com>
References: 
 <CAH29n6Met+KGNL2U8smEL20GzTH1QpyHbiPS7yJSJsr8xGfiXw@mail.gmail.com>
 <BLU0-SMTP19933C1DC5EB00A68F765B6AD040@phx.gbl>
 <CAH29n6ML+967w3MpO_z6NsQV_md8KV1iOZCJEpAzCeXfhkzA1A@mail.gmail.com>
 <CAA5C_ptLiDbvah--ffjmG5Tm_e+HMsuNOJ2SiAcSJJqJQNrcbQ@mail.gmail.com>
From: Josh Wills <jwills@cloudera.com>
Date: Wed, 16 Oct 2013 07:34:18 -0700
Message-ID: 
 <CAH29n6PH4BAT0U6eOvmdpvTyfqgsGmVtVo3ym_XA6hQUof3oiQ@mail.gmail.com>
Subject: Re: Thoughts on supporting HBase 0.96
To: dev <dev@crunch.apache.org>
Content-Type: multipart/alternative; boundary=089e0158b094d92baa04e8dc99fb

--089e0158b094d92baa04e8dc99fb
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Oct 16, 2013 at 12:15 AM, Gabriel Reid <gabriel.reid@gmail.com>wrote:

> On Wed, Oct 16, 2013 at 8:46 AM, Josh Wills <jwills@cloudera.com> wrote:
>
> > On Tue, Oct 15, 2013 at 11:42 PM, Chao Shi <stepinto@live.com> wrote:
> >
> > > I don't understand why needs another PTypeFamily here. I think we can
> > > simply provide some pre-defined PTypes.
> > >
> > > interface HFilePTypes {
> > >   static KEY_VALUE_PTYPE = xxx
> > >   static PUT_PTYPE = xxx
> > > }
> > >
> >
> > Technically, every PType has to provide an implementation of the
> > PTypeFamily getFamily() method-- even if it's just returning a dummy
> > object.
> >
> >
> Wouldn't a derived PType (like in o.a.c.types.PTypes) be a better fit here?
>

That was my initial attempt, and in an ideal world, my preferred solution--
but I haven't figured out how to make it work. The question here is: what
do I derive a KeyValue object to? What I really want, for purposes of
reading it/writing it to one of our HBase IO formats, is to map it to
itself, and not some subclass of Writable. Another option might be an
extension of WritableType to handle these special case formats-- I'll take
a crack at getting that to work.


> A whole new PTypeFamily sounds like a lot of work (unless maybe if it was a
> subclass of one of the existing ones), and I think there's still a fair bit
> of code
> that assumes that Avro & Writable are the only two possible PTypeFamily
> implementations.
>

For any kind of intermediate processing, that is still true. The
HBaseTypeFamily would only ever really appear at the input or output for a
job.


>
> - Gabriel
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

--089e0158b094d92baa04e8dc99fb--