Mailing-List: contact crunch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-dev@incubator.apache.org
Date: Sun, 16 Dec 2012 20:36:12 +0000 (UTC)
From: "Gabriel Reid (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.12623338.1355210254960.14645.1355690172845@arcas>
In-Reply-To: <JIRA.12623338.1355210254960@arcas>
References: <JIRA.12623338.1355210254960@arcas>
Subject: [jira] [Commented] (CRUNCH-128) Allow one stage of an MR pipeline
 to depend on another target being created
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CRUNCH-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13533506#comment-13533506 ] 

Gabriel Reid commented on CRUNCH-128:
-------------------------------------

Sorry for the super-slow uptake on this again -- I just went to look at it on Reviewboard and saw that there seems to be an issue with how the patch is published there (https://reviews.apache.org/r/8463/diff/2/). I think this came up on a previous patch as well on RB.

In any case, I took a look at it, and I definitely agree that this is a cleaner way of doing it (i.e. adding the dependency in the call to parallelDo), so that looks good to me. I also tried it out with the checkpointing scenario that we were discussing in the past, and it appears to work perfectly in that scenario.

One question that I would like to bring up is whether or not this should be in the PCollection interface, as opposed to just an additional method on PCollectionImpl. It seems that this will typically only be used internally, and adding it to the already large PCollection interface means that there are six variations of parallelDo to choose from, which can be confusing for (new) users, as well as a bit annoying with code completion in an IDE. My preference would be to leave this out of the public PCollection interface. What do you think?
                
> Allow one stage of an MR pipeline to depend on another target being created
> ---------------------------------------------------------------------------
>
>                 Key: CRUNCH-128
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-128
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: Josh Wills
>         Attachments: CheckpointingIT.java, CRUNCH-128.patch, CRUNCH-128v2.patch
>
>
> There are a couple of problems (e.g., mapside-joins, total orderings, etc.) where we need to guarantee that one PCollection has been written to the FileSystem before another MapReduce pipeline that depends on that file is allowed to run. This doesn't fit cleanly into the current set of abstractions for Crunch, which is why we force pipelines to execute via the run command to guarantee that the files have been created before the second stage is run.
> We should add the ability for a particular PCollection to require that a SourceTarget instance has been created before it can be executed, and the planner should incorporate this information into the MR pipeline planning process.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira