Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
MIME-Version: 1.0
In-Reply-To: <D25E4D19.49BE%landon.t.robinson@lowes.com>
References: <D25E4D19.49BE%landon.t.robinson@lowes.com>
Date: Tue, 3 Nov 2015 10:54:47 -0600
Message-ID: 
 <CADH-_NK=FZb4uKQOFQw_z3m+L+5dM_uE47EFFVQv0i_BpL3bgQ@mail.gmail.com>
Subject: Re: Understanding ScaleFactor in Crunch
From: Micah Whitacre <mkwhitacre@gmail.com>
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=001a1135f99e6025f60523a5c046

--001a1135f99e6025f60523a5c046
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

There are some details on scaleFactor here.[1]  Essentially crunch would
use it for a couple of options:

1. Calculating the number of reducers to use when grouping and nothing is
specified
2. Optimizing to decrease how much I/O it has to do if possible.

In the last situation if your pipeline might will require state to persist
Crunch will try to optimize to do the least amount of I/O at the cost of
doing recalculations.  So it might persist to disk right before your DoFn
if it creates a significant amount more of data.  If you know the data
increase is that significant it would definitely be advisable to override
the method and give a more reasonable factor value.

>> Can I tell it to leverage more mappers/reducers in the DoFn?

Scale factors will be applicable for the number of reducers but shouldn't
affect mappers as that would be controlled by the input splits.

[1] - http://crunch.apache.org/user-guide.html#doplan
[2] -
https://github.com/apache/crunch/blob/188360048a7f2d3cedf5fc915b48d7671f1d8=
d46/crunch-core/src/main/java/org/apache/crunch/DoFn.java#L132

On Tue, Nov 3, 2015 at 10:38 AM, Robinson, Landon - Landon <
landon.t.robinson@lowes.com> wrote:

> All,
>
> I=E2=80=99m trying to understand how I might use scaleFactor() in my Crun=
ch code.
> My use case is this: I have data that I read into a Pcollection that is
> smaller than my system=E2=80=99s block size, but when processed in a DoFn=
, *grows*
> pretty exponentially.
>
> So what started as a 10mb file might become 10 times larger.
>
> To prevent spills and memory issues, how could I leverage something like
> scaleFactor() (or whatever is needed) to indicate to the Crunch Planner
> that my resulting Pcollection will grow exponentially?
> Can I tell it to leverage more mappers/reducers in the DoFn?
>
> Guidance, if you could!
>
> Thanks,
> Landon
> -------------------------------------------------------------------------=
--
>
> Landon Robinson
> -------------------------------------------------------------------------=
--
> NOTICE: All information in and attached to the e-mails below may be
> proprietary, confidential, privileged and otherwise protected from improp=
er
> or erroneous disclosure. If you are not the sender's intended recipient,
> you are not authorized to intercept, read, print, retain, copy, forward, =
or
> disseminate this message. If you have erroneously received this
> communication, please notify the sender immediately by phone (704-758-100=
0)
> or by e-mail and destroy all copies of this message electronic, paper, or
> otherwise.
>
> *By transmitting documents via this email: Users, Customers, Suppliers an=
d
> Vendors collectively acknowledge and agree the transmittal of information
> via email is voluntary, is offered as a convenience, and is not a secured
> method of communication; Not to transmit any payment information E.G.
> credit card, debit card, checking account, wire transfer information,
> passwords, or sensitive and personal information E.G. Driver's license,
> DOB, social security, or any other information the user wishes to remain
> confidential; To transmit only non-confidential information such as plans=
,
> pictures and drawings and to assume all risk and liability for and
> indemnify Lowe's from any claims, losses or damages that may arise from t=
he
> transmittal of documents or including non-confidential information in the
> body of an email transmittal. Thank you. *
>

--001a1135f99e6025f60523a5c046
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">There are some details on scaleFactor here.[1] =C2=A0Essen=
tially crunch would use it for a couple of options:<div><br></div><div>1. C=
alculating the number of reducers to use when grouping and nothing is speci=
fied</div><div>2. Optimizing to decrease how much I/O it has to do if possi=
ble.</div><div><br></div><div>In the last situation if your pipeline might =
will require state to persist Crunch will try to optimize to do the least a=
mount of I/O at the cost of doing recalculations.=C2=A0 So it might persist=
 to disk right before your DoFn if it creates a significant amount more of =
data.=C2=A0 If you know the data increase is that significant it would defi=
nitely be advisable to override the method and give a more reasonable facto=
r value.</div><div><br></div><div>&gt;&gt;=C2=A0<span style=3D"color:rgb(0,=
0,0);font-family:Calibri,sans-serif;font-size:14px">Can I tell it to levera=
ge more mappers/reducers in the DoFn?</span></div><div><span style=3D"color=
:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br></span></div=
><div><span style=3D"color:rgb(0,0,0);font-family:Calibri,sans-serif;font-s=
ize:14px">Scale factors will be applicable for the number of reducers but s=
houldn&#39;t affect mappers as that would be controlled by the input splits=
.</span></div><div><br></div><div><div>[1] -=C2=A0<a href=3D"http://crunch.=
apache.org/user-guide.html#doplan">http://crunch.apache.org/user-guide.html=
#doplan</a></div></div><div>[2] -=C2=A0<a href=3D"https://github.com/apache=
/crunch/blob/188360048a7f2d3cedf5fc915b48d7671f1d8d46/crunch-core/src/main/=
java/org/apache/crunch/DoFn.java#L132">https://github.com/apache/crunch/blo=
b/188360048a7f2d3cedf5fc915b48d7671f1d8d46/crunch-core/src/main/java/org/ap=
ache/crunch/DoFn.java#L132</a></div></div><div class=3D"gmail_extra"><br><d=
iv class=3D"gmail_quote">On Tue, Nov 3, 2015 at 10:38 AM, Robinson, Landon =
- Landon <span dir=3D"ltr">&lt;<a href=3D"mailto:landon.t.robinson@lowes.co=
m" target=3D"_blank">landon.t.robinson@lowes.com</a>&gt;</span> wrote:<br><=
blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px=
 #ccc solid;padding-left:1ex">


<div style=3D"word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-fam=
ily:Calibri,sans-serif">
<div>All,</div>
<div><br>
</div>
<div>I=E2=80=99m trying to understand how I might use scaleFactor() in my C=
runch code.</div>
<div>My use case is this: I have data that I read into a Pcollection that i=
s smaller than my system=E2=80=99s block size, but when processed in a DoFn=
,
<b>grows</b> pretty exponentially.</div>
<div><br>
</div>
<div>So what started as a 10mb file might become 10 times larger.</div>
<div><br>
</div>
<div>To prevent spills and memory issues, how could I leverage something li=
ke scaleFactor() (or whatever is needed) to indicate to the Crunch Planner =
that my resulting Pcollection will grow exponentially?</div>
<div>Can I tell it to leverage more mappers/reducers in the DoFn?</div>
<div><br>
</div>
<div>Guidance, if you could!</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Landon</div>
<div>
<div>----------------------------------------------------------------------=
-----</div>
<div>
<div><br>
</div>
<div>Landon Robinson</div>
</div>
---------------------------------------------------------------------------=
</div>

<div>
NOTICE: All information in and attached to the e-mails below may be proprie=
tary, confidential, privileged and otherwise protected from improper or err=
oneous disclosure. If you are not the sender&#39;s intended recipient, you =
are not authorized to intercept, read, print, retain, copy, forward, or dis=
seminate this message. If you have erroneously received this communication,=
 please notify the sender immediately by phone <a href=3D"tel:%28704-758-10=
00" value=3D"+17047581000" target=3D"_blank">(704-758-1000</a>) or by e-mai=
l and destroy all copies of this message electronic, paper, or otherwise.
<br>
<br>
<b>By transmitting documents via this email: Users, Customers, Suppliers an=
d Vendors collectively acknowledge and agree the transmittal of information=
 via email is voluntary, is offered as a convenience, and is not a secured =
method of communication; Not to transmit any payment information E.G. credi=
t card, debit card, checking account, wire transfer information, passwords,=
 or sensitive and personal information E.G. Driver&#39;s license, DOB, soci=
al security, or any other information the user wishes to remain confidentia=
l; To transmit only non-confidential information such as plans, pictures an=
d drawings and to assume all risk and liability for and indemnify Lowe&#39;=
s from any claims, losses or damages that may arise from the transmittal of=
 documents or including non-confidential information in the body of an emai=
l transmittal. Thank you.
</b>
</div>

</div>

</blockquote></div><br></div>

--001a1135f99e6025f60523a5c046--