Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Fri, 4 Jan 2013 15:30:12 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12626054.1357313306507.80902.1357313412904@arcas>
In-Reply-To: <JIRA.12626054.1357313306507@arcas>
References: <JIRA.12626054.1357313306507@arcas>
Subject: [jira] [Created] (MAPREDUCE-4912) Investigate ways to clean up
 double job commit prevention
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Robert Joseph Evans created MAPREDUCE-4912:
----------------------------------------------

             Summary: Investigate ways to clean up double job commit prevention
                 Key: MAPREDUCE-4912
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4912
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mrv2
            Reporter: Robert Joseph Evans


Once MAPREDUCE-4819 goes in it fixes the issue where an OutputCommiter can double commit a job.  So that the output will never be touched after the job informs externally of success or failure.

The code and design could potentially use some cleanup and refactoring.

Issues brought up that should be investigated include:

# reporting KILL for killed jobs if they crash after the kill happens instead of error.
# using the job history log for recording the commit status instead of separate external files in HDFS.
# Placing the recovery/retry logic in the commit handler instead of the MRAppMaster, and having the recovery service replay the logs as it normally does for recovery.

This is not meant to be things that must be done, but alternatives that might clean up the code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira