Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <13036080.1163070940277.JavaMail.jira@brutus>
Date: Thu, 9 Nov 2006 03:15:40 -0800 (PST)
From: "Sanjay Dahiya (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Updated: (HADOOP-76) Implement speculative re-execution of
 reduces
In-Reply-To: <1176204687.1142031469402.JavaMail.jira@ajax>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

     [ http://issues.apache.org/jira/browse/HADOOP-76?page=all ]

Sanjay Dahiya updated HADOOP-76:
--------------------------------

    Attachment: Hadoop-76_2.patch

Changes in this patch update 

1. PhasedFileSystem has extra constructor that takes an JobConf
2. PhasedFileSystem now handles file overwrite flag properly. 

3. commit/abort can be done for a single file or for all files created by this FS. 
4. The method for commit is split in 2 function one is pvt with an extra boolean variable to avoid concurrent modification while committing in a iterator loop. 
 4. reduceProgress calculation is changed, earlier it was dividing the progress by reduces.length twice, once while update status and then while checking for spec tasks, which was incorrect. 

Thanks Devaraj for the review. 

> Implement speculative re-execution of reduces
> ---------------------------------------------
>
>                 Key: HADOOP-76
>                 URL: http://issues.apache.org/jira/browse/HADOOP-76
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.1.0
>            Reporter: Doug Cutting
>         Assigned To: Sanjay Dahiya
>            Priority: Minor
>         Attachments: Hadoop-76.patch, Hadoop-76_1.patch, Hadoop-76_2.patch, spec_reducev.patch
>
>
> As a first step, reduce task outputs should go to temporary files which are renamed when the task completes.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira