crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CRUNCH-405) Explore adding support for idempotent MRPipeline.plan()
Date Tue, 27 May 2014 21:05:02 GMT
Micah Whitacre created CRUNCH-405:
-------------------------------------

             Summary: Explore adding support for idempotent MRPipeline.plan()
                 Key: CRUNCH-405
                 URL: https://issues.apache.org/jira/browse/CRUNCH-405
             Project: Crunch
          Issue Type: Improvement
          Components: Core
            Reporter: Micah Whitacre
            Assignee: Josh Wills


Talking through a use case with a consumer, they were interested in having the ability to
run the MRPipeline.plan() method one to many times prior to ever calling the Pipeline.run/done
methods.  The reason for this was they were looking at pulling information off the MRExecutor
to tweak settings inside of their DoFns.

Currently the MRPipeline implementation however does not have an idempotent plan() method
as it alters the state of internal values therefore affecting the full run once done() is
called.  

It would be nice if we added an idempotent plan() method that could be gather this information
or perhaps a reset option.  




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message