Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Date: Wed, 6 Aug 2014 02:20:13 +0000 (UTC)
From: "Micah Whitacre (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.12716906.1401224625322.22342.1407291613282@arcas>
In-Reply-To: <JIRA.12716906.1401224625322@arcas>
References: <JIRA.12716906.1401224625322@arcas>
Subject: [jira] [Commented] (CRUNCH-405) Explore adding support for
 idempotent MRPipeline.plan()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CRUNCH-405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087137#comment-14087137 ] 

Micah Whitacre commented on CRUNCH-405:
---------------------------------------

Err sorry I got my keywords mixed up, "currentExecutor" should be marked volatile not transient. (can't explain what wires got crossed to make that mistake)

Would you make the release method synchronized?  Otherwise marking "currentExecutor" as volatile should make the state change visible sooner but wouldn't eliminate all timing issues if the callable was slow to run.  What if the MRPipeline.plan(...) only threw that exception if the MRExecutor.hasStarted() == false?  Would we want to block at that point or would that might undermine any value with runAsync()?

> Explore adding support for idempotent MRPipeline.plan()
> -------------------------------------------------------
>
>                 Key: CRUNCH-405
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-405
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Micah Whitacre
>            Assignee: Micah Whitacre
>         Attachments: CRUNCH-405.patch, CRUNCH-405_v1.patch
>
>
> Talking through a use case with a consumer, they were interested in having the ability to run the MRPipeline.plan() method one to many times prior to ever calling the Pipeline.run/done methods.  The reason for this was they were looking at pulling information off the MRExecutor to tweak settings inside of their DoFns.
> Currently the MRPipeline implementation however does not have an idempotent plan() method as it alters the state of internal values therefore affecting the full run once done() is called.  
> It would be nice if we added an idempotent plan() method that could be gather this information or perhaps a reset option.  


--
This message was sent by Atlassian JIRA
(v6.2#6252)