hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zheng Shao <zsh...@gmail.com>
Subject Re: can RCFile* be also exported as a hadoop contrib project?
Date Mon, 15 Jun 2009 08:23:34 GMT
Yes we did take a look at TFile before we started. The conclusion at the
time was that TFile was not ready yet, and as a result the RCFile is
implemented on top of SequenceFile (most of the SequenceFile code got
reused).
Once the TFile is ready, we should think about adding Columnar Support to
TFile just as the same way we do to SequenceFile.


Another thing is that I think as soon as we get time, we should publish the
current RCFile work to hadoop user mailing lists to see the feedback from
there. We should include some introduction of the design as well as
performance numbers. What do you think Yongqiang?

Zheng

2009/6/14 Jeff Hammerbacher <hammer@cloudera.com>

> Hey,
>
> The folks from Yahoo say that their work on TFile (see
> http://issues.apache.org/jira/browse/HADOOP-3315) is quite similar to the
> Hive work on RCFile. Might be worth checking out what's in TFile also and
> see if you can merge one into the other.
>
> Later,
> Jeff
>
> 2009/6/12 He Yongqiang <heyongqiang@software.ict.ac.cn>
>
> > I think this would be great. And we should avoid duplicate work and
> should
> > put together the develop forces instead of isolating them.  I completely
> > support this :).
> >
> >
> > On 09-6-13 上午5:42, "Joydeep Sen Sarma" <jssarma@facebook.com> wrote:
> >
> > > Is this columnar format reusable by other folks?
> > >
> > > Is it possible to demonstrate the use (in case of simple delimited text
> > data)
> > > by writing a configurable inputformat on top of this that can project
> out
> > the
> > > configured columns?
> > >
> > > The reason I am asking is that there is some parallel work going on in
> > > yahoo/pig on columnar format and no one's aware of this format -
> whereas
> > it
> > > should be usable by other people (and not just Hive).
> > >
> > > Comments?
> > >
> > >
> >
> >
> >
>



-- 
Yours,
Zheng

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message