Mailing-List: contact dev-help@pig.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@pig.apache.org
Received-SPF: pass (nike.apache.org: domain of jieli@cs.duke.edu designates
 152.3.140.1 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKne9Z6tgmmWj6OqOp3sS7hT=NnNYTwTf1EsEgYCMNYEmQT--Q@mail.gmail.com>
References: 
 <CALY2=u4uZhudU5JZxbb_ydbB143YBTRbk0Nuafd7t+65RDWKoA@mail.gmail.com>
	<CAKne9Z6tgmmWj6OqOp3sS7hT=NnNYTwTf1EsEgYCMNYEmQT--Q@mail.gmail.com>
Date: Fri, 2 Dec 2011 19:45:57 -0500
Message-ID: 
 <CALY2=u6aTzTsXTze4umf4KpBuZ309nF92K+tpyRRWFZ6FxB7Vw@mail.gmail.com>
Subject: Re: Early projection and lazy casting
From: Jie Li <jieli@cs.duke.edu>
To: dev@pig.apache.org
Content-Type: multipart/alternative; boundary=001517740942a7406a04b3256890

--001517740942a7406a04b3256890
Content-Type: text/plain; charset=ISO-8859-1

Why do joins prevent the early projection? Actually join has the greatest
need for it.

Jie

On Fri, Dec 2, 2011 at 7:33 PM, Jonathan Coveney <jcoveney@gmail.com> wrote:

> In what context? I always thought that it generally could, but that if you
> do joins it doesn't. Would be curious to know more from someone who
> knows...
>
> 2011/12/2 Jie Li <jieli@cs.duke.edu>
>
> > Hi all,
> >
> > We just figured out Pig 0.9.1 doesn't drop those non-necessary fields
> asap,
> > which really affects the performance. Though
> >
> >
> http://ofps.oreilly.com/titles/9781449302641/load_and_store_funcs.html#loadfunc_loaderpushdownsaid
> > that "As part of its optimizations Pig analyzes Pig Latin scripts and
> > determines what fields in an input it needs at each step in the script.
> It
> > uses this information to aggressively drop fields it no longer needs."
> >
> > We also found that Pig casts the data into the types defined in the
> schema,
> > which is usually unnecessary, as most of them will be soon dropped.
> >
> > To work around these, we have to manually drop those fields and remove
> the
> > types in the schema, which are really not interesting.
> >
> > Jie
> >
>

--001517740942a7406a04b3256890--