Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Received-SPF: pass (nike.apache.org: domain of jinalshah2007@gmail.com
 designates 74.125.82.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAH29n6NZq3o3u64koZADqmzuqJ1R4fHozrkienL+Ao75GBkmjg@mail.gmail.com>
References: 
 <CAAZ8DxrzW_WRQvQDsJ5-q5Q+FF-KS=xKd4fgpAGAmUaLxJ1ycQ@mail.gmail.com>
	<CAH29n6NGQq3f3n_O7nM24jHj99CPHB8R6=7OBhK4gGQCNN_S9A@mail.gmail.com>
	<CAAZ8DxpDR+Rf-EU93P6ufYiAas3sdXzwV21EKMBTXNkoqxc5ZQ@mail.gmail.com>
	<CAH29n6NZq3o3u64koZADqmzuqJ1R4fHozrkienL+Ao75GBkmjg@mail.gmail.com>
Date: Mon, 24 Feb 2014 13:25:03 -0600
Message-ID: 
 <CAAZ8DxqfLS+vQTy7wLa5PwyfbqPm4aqZuo7+geH_JSs0TbcTPg@mail.gmail.com>
Subject: Re: Object size
From: Jinal Shah <jinalshah2007@gmail.com>
To: dev@crunch.apache.org
Content-Type: multipart/alternative; boundary=00248c11e82db2dc9804f32bedd4

--00248c11e82db2dc9804f32bedd4
Content-Type: text/plain; charset=ISO-8859-1

Thanks Josh, I have a few following questions
so let's say with the default scaleFactor how much approximation should we
assume like +/- 1%?
How does scaleFactor affect the size of the object?
Can this be a part of Crunch as an enhancement to the current Join strategy?

Thanks
Jinal


On Mon, Feb 24, 2014 at 1:01 PM, Josh Wills <jwills@cloudera.com> wrote:

> Ah, cool. the long getSize() method will return Crunch's estimate of the
> size of the object in bytes, but it's good to keep in mind that it's a very
> rough approximation based on the size of the file on disk and any info we
> have about the behavior of any DoFns that are applied to the PTable when it
> is processed, which is communicated via the scaleFactor() function on each
> DoFn.
>
>
> On Mon, Feb 24, 2014 at 10:57 AM, Jinal Shah <jinalshah2007@gmail.com
> >wrote:
>
> > By size I meant the memory size sorry for the confusion. Like how much
> > memory will a PTable object require. Basically what I'm trying to do is
> if
> > the object is not that large and if it could fit in memory I wanted to
> > apply map-side join to optimize the join and depending on that I also
> > wanted to determine which one is smaller to use the Left join.
> >
> >
> > On Mon, Feb 24, 2014 at 12:45 PM, Josh Wills <jwills@cloudera.com>
> wrote:
> >
> > > There is the length() method, which will return a PObject<Long> with
> the
> > > number of elements in the PCollection. It requires running an MR job
> > > though.
> > >
> > > J
> > >
> > >
> > > On Mon, Feb 24, 2014 at 10:03 AM, Jinal Shah <jinalshah2007@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > Is there a way possible in crunch to find the size of a particular
> > > > PCollection or PTable in whole.
> > > >
> > > > Thanks
> > > > Jinal
> > > >
> > >
> > >
> > >
> > > --
> > > Director of Data Science
> > > Cloudera <http://www.cloudera.com>
> > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> > >
> >
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>

--00248c11e82db2dc9804f32bedd4--