Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: pass (athena.apache.org: domain of sjdurfey@gmail.com designates
 209.85.214.174 as permitted sender)
From: Stephen Durfey <sjdurfey@gmail.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_70CB8129-3343-4940-86B4-54A4B7BAF0B8"
Message-Id: <B460E4AE-3E38-451F-ADD2-AB23440272C9@gmail.com>
Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\))
Subject: Re: Cleaning up after exceptions
Date: Tue, 1 Apr 2014 14:02:19 -0500
References: <6BFA5C59-DF75-4E96-82BF-55D166BA90F7@gmail.com>
 <CANb5z2KR2--P82u-rg9ER6=7qHtazMPZZef7SmkeimGTtPWEgw@mail.gmail.com>
To: user@crunch.apache.org
In-Reply-To: 
 <CANb5z2KR2--P82u-rg9ER6=7qHtazMPZZef7SmkeimGTtPWEgw@mail.gmail.com>


--Apple-Mail=_70CB8129-3343-4940-86B4-54A4B7BAF0B8
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=windows-1252

Thanks. I=92ll look into that. Also, I just noticed that as of 0.8.2, =
crunch has a public cleanup() on the Pipeline interface. I should be =
able to use that, as my code was just updated to that version.=20

On Mar 29, 2014, at 5:58 AM, Josh Wills <josh.wills@gmail.com> wrote:

> IIRC (I'm away from my computer) we added the ability to add arbitrary =
hooks that would always be executed at the end of a pipeline run to the =
PipelineExecution interface-- the one that is returned by runAsync), =
which could be used to ensure that the temp directories were cleaned up =
no matter what happened on the run. Does that work for this problem?
>=20
>=20
> On Fri, Mar 28, 2014 at 10:05 AM, Stephen Durfey <sjdurfey@gmail.com> =
wrote:
> If I have a scenario where I have already called Pipeline#run (and =
some temporary directories were created by Crunch during the run), and =
have continued on to do some additional processing (created some new =
PCollection=92s and specified a write location), and an exception occurs =
in my code, outside of the pipeline, before Pipeline#run is called =
again, I would need a way to ensure the temporary directories created in =
my initial run are always cleaned up. I could call Pipeline#done, which =
calls cleanup() in MRPipeline, but it also calls run(). However, I would =
prefer not to have run() called at all, due to the exception thrown in =
my code.
>=20
> Would it be possible to make cleanup() public in the Pipeline =
interface so that can be used to clean up any temp directories created =
by the pipeline?
>=20


--Apple-Mail=_70CB8129-3343-4940-86B4-54A4B7BAF0B8
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=windows-1252

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dwindows-1252"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space;">Thanks. I=92ll look into that. Also, I just noticed =
that as of 0.8.2, crunch has a public cleanup() on the Pipeline =
interface. I should be able to use that, as my code was just updated to =
that version.&nbsp;<div><br><div><div><div>On Mar 29, 2014, at 5:58 AM, =
Josh Wills &lt;<a =
href=3D"mailto:josh.wills@gmail.com">josh.wills@gmail.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div dir=3D"ltr">IIRC (I'm away from my computer) we added =
the ability to add arbitrary hooks that would always be executed at the =
end of a pipeline run to the PipelineExecution interface-- the one that =
is returned by runAsync), which could be used to ensure that the temp =
directories were cleaned up no matter what happened on the run. Does =
that work for this problem?</div>

<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Fri, =
Mar 28, 2014 at 10:05 AM, Stephen Durfey <span dir=3D"ltr">&lt;<a =
href=3D"mailto:sjdurfey@gmail.com" =
target=3D"_blank">sjdurfey@gmail.com</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex">If I have a scenario =
where I have already called Pipeline#run (and some temporary directories =
were created by Crunch during the run), and have continued on to do some =
additional processing (created some new PCollection=92s and specified a =
write location), and an exception occurs in my code, outside of the =
pipeline, before Pipeline#run is called again, I would need a way to =
ensure the temporary directories created in my initial run are always =
cleaned up. I could call Pipeline#done, which calls cleanup() in =
MRPipeline, but it also calls run(). However, I would prefer not to have =
run() called at all, due to the exception thrown in my code.<br>


<br>
Would it be possible to make cleanup() public in the Pipeline interface =
so that can be used to clean up any temp directories created by the =
pipeline?</blockquote></div><br></div>
</blockquote></div><br></div></div></body></html>=

--Apple-Mail=_70CB8129-3343-4940-86B4-54A4B7BAF0B8--