Return-Path: X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E8FD1D464 for ; Mon, 4 Feb 2013 22:16:14 +0000 (UTC) Received: (qmail 29112 invoked by uid 500); 4 Feb 2013 22:16:13 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 29057 invoked by uid 500); 4 Feb 2013 22:16:13 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 29004 invoked by uid 99); 4 Feb 2013 22:16:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Feb 2013 22:16:13 +0000 Date: Mon, 4 Feb 2013 22:16:13 +0000 (UTC) From: "Karthik Kambatla (JIRA)" To: mapreduce-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (MAPREDUCE-4843) When using DefaultTaskController, JobLocalizer not thread safe MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4843: ---------------------------------------- Attachment: mr-4843.patch Uploading the patch from MAPREDUCE-4964 as that solves this issue in a simpler/cleaner way. The discussion on that JIRA has all the details. Applied the patch to latest branch-1 and it applies cleanly. Also, verified TestJobLocalizer passes. > When using DefaultTaskController, JobLocalizer not thread safe > -------------------------------------------------------------- > > Key: MAPREDUCE-4843 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker > Affects Versions: 1.1.1 > Reporter: zhaoyunjiong > Assignee: Karthik Kambatla > Priority: Critical > Attachments: MAPREDUCE-4843-branch-1.1.patch, mr-4843.patch > > > In our cluster, some times job will failed due to below exception: > 2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories > at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424) > at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058) > at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213) > The root cause is JobLocalizer is not thread safe. > In DefaultTaskController.initializeJob method: > JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid); > but in JobLocalizer, it just simply keep the reference of the conf. > When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance. > So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf. > Then it will cause the previous job's job.xml stored at another user's dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira