Mailing-List: contact crunch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-dev@incubator.apache.org
Date: Sun, 10 Feb 2013 02:15:12 +0000 (UTC)
From: "Josh Wills (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.12623763.1355412253398.270335.1360462512720@arcas>
In-Reply-To: <JIRA.12623763.1355412253398@arcas>
References: <JIRA.12623763.1355412253398@arcas>
Subject: [jira] [Updated] (CRUNCH-132) Add configurable behavior for when a
 pipeline output directory already exists
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/CRUNCH-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Wills updated CRUNCH-132:
------------------------------

    Attachment: CRUNCH-132.patch

Took another crack at this one today and got something that's at least worth considering.

It seems to me that there are two ways to handle the existing output checking: eagerly (as this patch does) or lazily (which seemed to be more difficult to do correctly.)

The eager strategy checks to see whether or not the output path exists immediately upon Pipeline.write(collection, target) being called. The default action in the case that the output exists is to throw a CrunchRuntimeException. This can be overridden to either OVERWRITE or APPEND to the existing output path by calling Pipeline.write(collection, target, strategy) or PCollection.write(target, strategy).

My feeling was that the eager strategy was more in line with user's expectations of how Crunch would work, although it could potentially cause some unexpected failures, e.g.,

pipeline.write(pcollect1, target);
pipeline.run();
pipeline.write(pcollect2, target); // exception thrown

that the user would have to compensate for with, e.g.,

pipeline.write(pcollect1, target);
pipeline.run();
pipeline.write(pcollect2, target, ExistingOutputStrategy.APPEND);

Note that the similar:

pipeline.write(pcollect1, target);
pipeline.write(pcollect2, target);
pipeline.run();

would not throw an exception, as the assumption would be that the user was intentionally writing both collections to the same output target.

If folks have strong feelings about the design here, I am more than happy to hear them.

                
> Add configurable behavior for when a pipeline output directory already exists
> -----------------------------------------------------------------------------
>
>                 Key: CRUNCH-132
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-132
>             Project: Crunch
>          Issue Type: Improvement
>    Affects Versions: 0.4.0
>            Reporter: Dave Beech
>            Assignee: Josh Wills
>         Attachments: CRUNCH-132.patch, CRUNCH-132-proto.patch
>
>
> Usually when you run a mapreduce job and the output directory already exists, the job fails (won't start). A Crunch job does run, but results in the output data being duplicated in the output directory with numbered files that follow on from the previous run. 
> Example
> Run 1, single reducer /output -> /output/part-r-00000
> Run 2, single reducer /output -> /output/part-r-00000, /output/part-r-00001
> I didn't realise I'd run my job twice, so when I looked in the directory it seemed that there had been 2 reducers and somehow the output had been generated twice, which was confusing. 
> I realise this may be by design, but it feels wrong to me. I'd prefer if the behaviour of a standard mapreduce job was preserved.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira