Mailing-List: contact crunch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-dev@incubator.apache.org
Date: Sun, 16 Dec 2012 21:06:12 +0000 (UTC)
From: "Josh Wills (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.12623338.1355210254960.14679.1355691972852@arcas>
In-Reply-To: <JIRA.12623338.1355210254960@arcas>
References: <JIRA.12623338.1355210254960@arcas>
Subject: [jira] [Commented] (CRUNCH-128) Allow one stage of an MR pipeline
 to depend on another target being created
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CRUNCH-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13533517#comment-13533517 ] 

Josh Wills commented on CRUNCH-128:
-----------------------------------

What if we created a ParallelDoOperation class in o.a.c, which could keep us down to five versions of pDo on PCollection, and we could use it for additional advanced options (like the optional SourceTargets) that we invent now and in the future?
                
> Allow one stage of an MR pipeline to depend on another target being created
> ---------------------------------------------------------------------------
>
>                 Key: CRUNCH-128
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-128
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: Josh Wills
>         Attachments: CheckpointingIT.java, CRUNCH-128.patch, CRUNCH-128v2.patch
>
>
> There are a couple of problems (e.g., mapside-joins, total orderings, etc.) where we need to guarantee that one PCollection has been written to the FileSystem before another MapReduce pipeline that depends on that file is allowed to run. This doesn't fit cleanly into the current set of abstractions for Crunch, which is why we force pipelines to execute via the run command to guarantee that the files have been created before the second stage is run.
> We should add the ability for a particular PCollection to require that a SourceTarget instance has been created before it can be executed, and the planner should incorporate this information into the MR pipeline planning process.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira