Return-Path: 
 <user-return-786-apmail-crunch-user-archive=crunch.apache.org@crunch.apache.org>
X-Original-To: apmail-crunch-user-archive@www.apache.org
Delivered-To: apmail-crunch-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4B35E17F71
	for <apmail-crunch-user-archive@www.apache.org>;
 Wed,  1 Apr 2015 19:02:09 +0000 (UTC)
Received: (qmail 15209 invoked by uid 500); 1 Apr 2015 19:02:09 -0000
Delivered-To: apmail-crunch-user-archive@crunch.apache.org
Received: (qmail 15164 invoked by uid 500); 1 Apr 2015 19:02:09 -0000
Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@crunch.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@crunch.apache.org>
List-Post: <mailto:user@crunch.apache.org>
List-Id: <user.crunch.apache.org>
Reply-To: user@crunch.apache.org
Delivered-To: mailing list user@crunch.apache.org
Received: (qmail 15154 invoked by uid 99); 1 Apr 2015 19:02:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2015 19:02:09 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jwills@cloudera.com designates
 209.85.192.44 as permitted sender)
Received: from [209.85.192.44] (HELO mail-qg0-f44.google.com) (209.85.192.44)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2015 19:01:44 +0000
Received: by qgeb100 with SMTP id b100so10398837qge.3
        for <user@crunch.apache.org>; Wed, 01 Apr 2015 11:59:27 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=Fvj8gpw2f6SMsKghVKfs/PeV8MUeRMXzB4v5soJae8c=;
        b=IS6uhG2lV1qrOt0balVyQXFOCjdhTrT6Z4z6nLqraJFutF4D8myENNx4PKxXwOaOm/
         CsB/skL/YYckRZFqo6MBOTy/FQeHPDEIcHZAqyLI42edBvG4AfEZtfrwPTNfnxR+2zvV
         o/uhUUWZdf5rhmCSFsXkD4ZgKLb0SYBfSUOTBofHgq+c3wlBqL+000uSLziTmUmyxtNJ
         hXjhVx8C7kk6pLTBl8hBaa7TtuvZc8t0gTIKl3xGEk5upjKnvYHqeCyU3VB9yBiZqqOx
         3mKeTEdrSC67oJJJlK6iWE3u1i7urg3tG9uiYQy5Jcw3GNAZ3GozjbhESOdih5hj3+FK
         SPAg==
X-Gm-Message-State: 
 ALoCoQnFVqkA6SEfw2zwkWLAbw3nJ9aw4x/A8CA/pRfXTIi30rkpQiF9UXbUdJ4OXlT4BGpEH0UV
X-Received: by 10.140.202.17 with SMTP id x17mr59064531qha.50.1427914767357;
 Wed, 01 Apr 2015 11:59:27 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.140.159.85 with HTTP; Wed, 1 Apr 2015 11:59:07 -0700 (PDT)
In-Reply-To: 
 <CAC79LcbRTDvKT7Xg=tzDANx43mUUOSGAps0KF6hcs-0zpV2mgg@mail.gmail.com>
References: 
 <CAC79LcbRTDvKT7Xg=tzDANx43mUUOSGAps0KF6hcs-0zpV2mgg@mail.gmail.com>
From: Josh Wills <jwills@cloudera.com>
Date: Wed, 1 Apr 2015 08:59:07 -1000
Message-ID: 
 <CAH29n6OPaL5-vij8jfzUc41DOs-q2AYRwacW8XsT1RGxvQGDHQ@mail.gmail.com>
Subject: Re: Deleting Checkpoint dirs after successful run
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=001a1141cd4c7a727e0512ae500f
X-Virus-Checked: Checked by ClamAV on apache.org

--001a1141cd4c7a727e0512ae500f
Content-Type: text/plain; charset=UTF-8

Hey David,

My recommendation is to use the fact that the PipelineExecution object
returned by Pipeline.runAsync() implements Future<PipelineResult>, so you
can add a listener to it that cleans up your checkpoint directories at the
end of the job depending on whether it succeeds or fails.

Josh

On Wed, Apr 1, 2015 at 5:52 AM, David Ortiz <dpo5003@gmail.com> wrote:

> Hey everyone,
>
>      Is there a setting I can tweak in my MRPipeline so when it does its
> cleanup after a run it cleans up the checkpoint dirs I create as well, or
> would I need to add some hdfs code at the end in the case of a successful
> run?
>
> Thanks,
>      Dave Ortiz
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

--001a1141cd4c7a727e0512ae500f
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hey David,<div><br></div><div>My recommendation is to use =
the fact that the PipelineExecution object returned by Pipeline.runAsync() =
implements Future&lt;PipelineResult&gt;, so you can add a listener to it th=
at cleans up your checkpoint directories at the end of the job depending on=
 whether it succeeds or fails.</div><div><br></div><div>Josh</div></div><di=
v class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Apr 1, 2015 =
at 5:52 AM, David Ortiz <span dir=3D"ltr">&lt;<a href=3D"mailto:dpo5003@gma=
il.com" target=3D"_blank">dpo5003@gmail.com</a>&gt;</span> wrote:<br><block=
quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
 solid;padding-left:1ex"><div dir=3D"ltr">Hey everyone,<div><br></div><div>=
=C2=A0 =C2=A0 =C2=A0Is there a setting I can tweak in my MRPipeline so when=
 it does its cleanup after a run it cleans up the checkpoint dirs I create =
as well, or would I need to add some hdfs code at the end in the case of a =
successful run?</div><div><br></div><div>Thanks,</div><div>=C2=A0 =C2=A0 =
=C2=A0Dave Ortiz</div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature"><div>Director of Data Science</div><div><a href=3D"htt=
p://www.cloudera.com" target=3D"_blank">Cloudera</a></div><div>Twitter: <a =
href=3D"http://twitter.com/josh_wills" target=3D"_blank">@josh_wills</a></d=
iv></div>
</div>

--001a1141cd4c7a727e0512ae500f--