Mailing-List: contact pig-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pig-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of dvryaboy@gmail.com designates
 74.125.78.24 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=al6K62h/pLia97yc17CPdTk4NvTI5sN+JKyoRy9C8Gk22KXFHI8kQLfICz9FCCIxHR
         JEER48tl3DMZLYSeulkXxD0Ce92U0cUbP1l4BHoC6B/wdkwyNoyUdLyNifuNdFvlQpo7
         7un7lBh63+Hkqa76zQpkf05TQ9o5wH13+uge0=
MIME-Version: 1.0
In-Reply-To: 
 <236D5828CF7F5749AF1BCB7F87596DE1047F5963@SNV-EXVS09.ds.corp.yahoo.com>
References: 
 <B0CADC41A6E8764D906EC1F9B54449400B19C091@WESTMINSTER.global.adviva>
	 <236D5828CF7F5749AF1BCB7F87596DE1047F590E@SNV-EXVS09.ds.corp.yahoo.com>
	 <357a70950911301218k171c014ave8e9e0238012a44a@mail.gmail.com>
	 <ABDFAEBA-175B-46CE-95CF-1BF98D7A9036@yahoo-inc.com>
	 <236D5828CF7F5749AF1BCB7F87596DE1047F5963@SNV-EXVS09.ds.corp.yahoo.com>
Date: Mon, 30 Nov 2009 18:39:23 -0500
Message-ID: <357a70950911301539u1258fee1gc5f288b58d69f437@mail.gmail.com>
Subject: Re: Pig reading hive columnar rc tables
From: Dmitriy Ryaboy <dvryaboy@gmail.com>
To: pig-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d99c4bc20b9004799f2631

--0016e6d99c4bc20b9004799f2631
Content-Type: text/plain; charset=ISO-8859-1

I retract the suggestion :).
How would we do testing/building for it in piggybank? Not include it in the
compile and test targets, and set up a separate compile-rcstore,
test-rcstore targets?

-D


On Mon, Nov 30, 2009 at 6:31 PM, Olga Natkovich <olgan@yahoo-inc.com> wrote:

> +1 on what Alan is saying. I think it would be an overkill to have
> another contrib. for this.
>
> Olga
>
> -----Original Message-----
> From: Alan Gates [mailto:gates@yahoo-inc.com]
> Sent: Monday, November 30, 2009 2:42 PM
> To: pig-dev@hadoop.apache.org
> Subject: Re: Pig reading hive columnar rc tables
>
>
> On Nov 30, 2009, at 12:18 PM, Dmitriy Ryaboy wrote:
>
> > That's awesome, I've been itching to do that but never got around to
> > it..
> > Garrit, do you have any benchmarks on read speeds?
> >
> > I don't know about putting this in piggybank, as it carries with it
> > pretty
> > significant dependencies, increasing the size of the jar and making it
> > difficult for users to don't need it to build piggybank in the first
> > place.
> > We might want to consider some other contrib for it -- maybe a "misc"
> > contrib that would have indivudual ant targets for these kinds of
> > compatibility submissions?
>
> Does it have to increase the size of the piggybank jar?  Instead of
> including hive in our piggybank jar, which I agree would be bad, can
> we just say that if you want to use this function you need to provide
> the appropriate hive jar yourself?  This way we could use ivy to pull
> the jars and build piggybank.
>
> I'm not really wild about creating a new section of contrib just for
> functions that have heavier weight requirements.
>
> Alan.
>
> >
> > -D
> >
> >
> > On Mon, Nov 30, 2009 at 3:09 PM, Olga Natkovich <olgan@yahoo-
> > inc.com> wrote:
> >
> >> Hi Garrit,
> >>
> >> It would be great if you could contribute the code. The process is
> >> pretty simple:
> >>
> >> - Open a JIRA that describes what the loader does and that you would
> >> like to contribute it to the Piggybank.
> >> - Submit the patch that contains the loader. Make sure it has unit
> >> tests
> >> and javadoc.
> >>
> >> On this is done, one of the committers will review and commit the
> >> patch.
> >>
> >> More details on how to contribute are in
> >> http://wiki.apache.org/pig/PiggyBank.
> >>
> >> Olga
> >>
> >> -----Original Message-----
> >> From: Gerrit van Vuuren [mailto:gvanvuuren@specificmedia.com]
> >> Sent: Friday, November 27, 2009 2:42 AM
> >> To: pig-dev@hadoop.apache.org
> >> Subject: Pig reading hive columnar rc tables
> >>
> >> Hi,
> >>
> >>
> >>
> >> I've coded a LoadFunc implementation that can read from Hive
> >> Columnar RC
> >> tables, this is needed for a project that I'm working on because
> >> all our
> >> data is stored using the Hive thrift serialized Columnar RC format. I
> >> have looked at the piggy bank but did not find any implementation
> >> that
> >> could do this. We've been running it on our cluster for the last week
> >> and have worked out most bugs.
> >>
> >>
> >>
> >> There are still some improvements to be done but I would need  like
> >> setting the amount of mappers based on date partitioning. Its been
> >> optimized so as to read only specific columns and can churn through a
> >> data set almost 8 times faster with this improvement because not all
> >> column data is read.
> >>
> >>
> >>
> >> I would like to contribute the class to the piggybank can you guide
> >> me
> >> in what I need to do?
> >>
> >> I've used hive specific classes to implement this, is it possible
> >> to add
> >> this to the piggy bank build ivy for automatic download of the
> >> dependencies?
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Gerrit Jansen van Vuuren
> >>
> >>
>
>

--0016e6d99c4bc20b9004799f2631--