Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAMQQebCN_Z+DFa4R5hhoJjsORhsVcAdUcwmAy2v=2F4q2-Mq6g@mail.gmail.com>
References: 
 <CAMQQebCN_Z+DFa4R5hhoJjsORhsVcAdUcwmAy2v=2F4q2-Mq6g@mail.gmail.com>
Date: Thu, 1 Oct 2015 15:28:46 -0700
Message-ID: 
 <CABc3QxFyvkcLdQ8k9J4-oUahP7EuxbyPFDYR2DRR3PdoaJ83OQ@mail.gmail.com>
Subject: Re: Preventing Cleanup of PCollections
From: Everett Anderson <everett@nuna.com>
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=047d7beb9f6e021307052112923c

--047d7beb9f6e021307052112923c
Content-Type: text/plain; charset=UTF-8

(Context: This is related to the 'LeaseExpiredExceptions and temp side
effect files' thread.)

In particular, the workaround would mean that we'd keep using the same
PCollection/PTable references after a call to run()/cleanup(), which feels
weird.

Example:

PTable liveTable = ...
liveTable = liveTable.parallelDo(...)

// Write the table somewhere we know won't get cleaned up,
// which changes its internal Target.
liveTable.write(To.sequenceFile(tempPath),
                Target.WriteMode.CHECKPOINT);

// Call run() and cleanup() to flush old temporary data.
pipeline.run();
pipeline.cleanup(false);

// Keep using liveTable since we know it'll work under the
// covers because its Target is a sequence file that wasn't
// cleaned up.
liveTable = liveTable.parallelDo(...)


On Thu, Oct 1, 2015 at 10:54 AM, Jeff Quinn <jeff@nuna.com> wrote:

> Hello,
>
> Our crunch pipeline has suffered from ballooning HDFS usage which spikes
> during the course of the job. Our solution has been to call Pipeline.run()
> and Pipeline.cleanup() between the major operations, hoping to achieve
> periodic "garbage collection" of the temporary outputs that are produced
> during the course of the pipeline.
>
> The problem is some PCollections from one operation will need to be used
> as input to subsequent operations, and cleanup() seems to blow away ALL
> PCollections that have not been explicitly written to a target (from
> reading the source, it seems to just blow away the pipeline temp directory).
>
> Our workaround has been to explicitly call .write on the PCollections we
> know we will need across calls to run()/cleanup(). This seems to work as
> far as I can tell, but it feels hacky. Is there a better or more supported
> way to handle this, and is this approach likely to fail in future crunch
> versions?
>
> Thanks!
>
> Jeff
>
> *DISCLAIMER:* The contents of this email, including any attachments, may
> contain information that is confidential, proprietary in nature, protected
> health information (PHI), or otherwise protected by law from disclosure,
> and is solely for the use of the intended recipient(s). If you are not the
> intended recipient, you are hereby notified that any use, disclosure or
> copying of this email, including any attachments, is unauthorized and
> strictly prohibited. If you have received this email in error, please
> notify the sender of this email. Please delete this and all copies of this
> email from your system. Any opinions either expressed or implied in this
> email and all attachments, are those of its author only, and do not
> necessarily reflect those of Nuna Health, Inc.

-- 
*DISCLAIMER:* The contents of this email, including any attachments, may 
contain information that is confidential, proprietary in nature, protected 
health information (PHI), or otherwise protected by law from disclosure, 
and is solely for the use of the intended recipient(s). If you are not the 
intended recipient, you are hereby notified that any use, disclosure or 
copying of this email, including any attachments, is unauthorized and 
strictly prohibited. If you have received this email in error, please 
notify the sender of this email. Please delete this and all copies of this 
email from your system. Any opinions either expressed or implied in this 
email and all attachments, are those of its author only, and do not 
necessarily reflect those of Nuna Health, Inc.

--047d7beb9f6e021307052112923c
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">(Context: This is related to the &#39;LeaseExpiredExceptio=
ns and temp side effect files&#39; thread.)<div><br></div><div>In particula=
r, the workaround would mean that we&#39;d keep using the same PCollection/=
PTable references after a call to run()/cleanup(), which feels weird.</div>=
<div><br></div><div>Example:</div><div><br></div><div><font face=3D"monospa=
ce, monospace">PTable liveTable =3D ...</font></div><div><font face=3D"mono=
space, monospace">liveTable =3D liveTable.parallelDo(...)</font></div><div>=
<font face=3D"monospace, monospace"><br></font></div><div><font face=3D"mon=
ospace, monospace">// Write the table somewhere we know won&#39;t get clean=
ed up,</font></div><div><font face=3D"monospace, monospace">// which change=
s its internal Target.</font></div><div><font face=3D"monospace, monospace"=
>liveTable.write(To.sequenceFile(tempPath),=C2=A0</font></div><div><font fa=
ce=3D"monospace, monospace">=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 Target.WriteMode.CHECKPOINT);</font></div><div><font face=3D"mon=
ospace, monospace"><br></font></div><div><font face=3D"monospace, monospace=
">// Call run() and cleanup() to flush old temporary data.</font></div><div=
><font face=3D"monospace, monospace">pipeline.run();</font></div><div><font=
 face=3D"monospace, monospace">pipeline.cleanup(false);</font></div><div><f=
ont face=3D"monospace, monospace"><br></font></div><div><font face=3D"monos=
pace, monospace">// Keep using liveTable since we know it&#39;ll work under=
 the</font></div><div><font face=3D"monospace, monospace">// covers because=
 its Target is a sequence file that wasn&#39;t</font></div><div><font face=
=3D"monospace, monospace">// cleaned up.</font></div><div><font face=3D"mon=
ospace, monospace">liveTable =3D liveTable.parallelDo(...)</font></div><div=
><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div=
><br></div><div><br></div><div><br></div></div><div class=3D"gmail_extra"><=
br><div class=3D"gmail_quote">On Thu, Oct 1, 2015 at 10:54 AM, Jeff Quinn <=
span dir=3D"ltr">&lt;<a href=3D"mailto:jeff@nuna.com" target=3D"_blank">jef=
f@nuna.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=
=3D"ltr">Hello,<div><br></div><div>Our crunch pipeline has suffered from ba=
llooning HDFS usage which spikes during the course of the job. Our solution=
 has been to call Pipeline.run() and Pipeline.cleanup() between the major o=
perations, hoping to achieve periodic &quot;garbage collection&quot; of the=
 temporary outputs that are produced during the course of the pipeline.</di=
v><div><br></div><div>The problem is some PCollections from one operation w=
ill need to be used as input to subsequent operations, and cleanup() seems =
to blow away ALL PCollections that have not been explicitly written to a ta=
rget (from reading the source, it seems to just blow away the pipeline temp=
 directory).</div><div><br></div><div>Our workaround has been to explicitly=
 call .write on the PCollections we know we will need across calls to run()=
/cleanup(). This seems to work as far as I can tell, but it feels hacky. Is=
 there a better or more supported way to handle this, and is this approach =
likely to fail in future crunch versions?</div><div><br></div><div>Thanks!<=
/div><div><br></div><div>Jeff</div></div>

<br>
<font size=3D"2" color=3D"#808080"><b style=3D"font-family:Calibri,sans-ser=
if;background-color:rgb(255,255,255)">DISCLAIMER:</b><span style=3D"font-fa=
mily:Calibri,sans-serif;background-color:rgb(255,255,255)">=C2=A0The conten=
ts of this email, including any attachments, may contain information that i=
s confidential, proprietary in nature, protected health information (PHI), =
or otherwise protected by law from disclosure, and is solely for the use of=
 the intended recipient(s). If you are not the intended recipient, you are =
hereby notified that any use, disclosure or copying of this email, includin=
g any attachments, is unauthorized and strictly prohibited. If you have rec=
eived this email in error, please notify the sender of this email. Please d=
elete this and all copies of this email from your system. Any opinions eith=
er expressed or implied in this email and all attachments, are those of its=
 author only, and do not necessarily reflect those of Nuna Health, Inc.</sp=
an></font></blockquote></div><br></div>

<br>
<font size=3D"2" color=3D"#808080"><b style=3D"font-family:Calibri,sans-ser=
if;background-color:rgb(255,255,255)">DISCLAIMER:</b><span style=3D"font-fa=
mily:Calibri,sans-serif;background-color:rgb(255,255,255)">=C2=A0The conten=
ts of this email, including any attachments, may contain information that i=
s confidential, proprietary in nature, protected health information (PHI), =
or otherwise protected by law from disclosure, and is solely for the use of=
 the intended recipient(s). If you are not the intended recipient, you are =
hereby notified that any use, disclosure or copying of this email, includin=
g any attachments, is unauthorized and strictly prohibited. If you have rec=
eived this email in error, please notify the sender of this email. Please d=
elete this and all copies of this email from your system. Any opinions eith=
er expressed or implied in this email and all attachments, are those of its=
 author only, and do not necessarily reflect those of Nuna Health, Inc.</sp=
an></font>
--047d7beb9f6e021307052112923c--