Return-Path: X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8ADBBE17D for ; Mon, 4 Feb 2013 22:26:15 +0000 (UTC) Received: (qmail 57881 invoked by uid 500); 4 Feb 2013 22:26:14 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 57838 invoked by uid 500); 4 Feb 2013 22:26:14 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 57785 invoked by uid 99); 4 Feb 2013 22:26:14 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Feb 2013 22:26:14 +0000 Date: Mon, 4 Feb 2013 22:26:14 +0000 (UTC) From: "Alejandro Abdelnur (JIRA)" To: mapreduce-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (MAPREDUCE-4843) When using DefaultTaskController, JobLocalizer not thread safe MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13570694#comment-13570694 ] Alejandro Abdelnur commented on MAPREDUCE-4843: ----------------------------------------------- +1. As per discussion in MAPREDUCE-4964 the latest patch seems a better way of doing it. > When using DefaultTaskController, JobLocalizer not thread safe > -------------------------------------------------------------- > > Key: MAPREDUCE-4843 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker > Affects Versions: 1.1.1 > Reporter: zhaoyunjiong > Assignee: Karthik Kambatla > Priority: Critical > Attachments: MAPREDUCE-4843-branch-1.1.patch, mr-4843.patch > > > In our cluster, some times job will failed due to below exception: > 2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories > at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424) > at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058) > at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213) > The root cause is JobLocalizer is not thread safe. > In DefaultTaskController.initializeJob method: > JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid); > but in JobLocalizer, it just simply keep the reference of the conf. > When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance. > So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf. > Then it will cause the previous job's job.xml stored at another user's dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira