Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 04B239FA3 for ; Fri, 24 Feb 2012 01:53:11 +0000 (UTC) Received: (qmail 18626 invoked by uid 500); 24 Feb 2012 01:53:07 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 18578 invoked by uid 500); 24 Feb 2012 01:53:07 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 18570 invoked by uid 99); 24 Feb 2012 01:53:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Feb 2012 01:53:07 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FSL_RCVD_USER,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of zhenhe.yk@gmail.com designates 209.85.210.176 as permitted sender) Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Feb 2012 01:52:57 +0000 Received: by iagz35 with SMTP id z35so3328007iag.35 for ; Thu, 23 Feb 2012 17:52:36 -0800 (PST) Received-SPF: pass (google.com: domain of zhenhe.yk@gmail.com designates 10.42.135.129 as permitted sender) client-ip=10.42.135.129; Authentication-Results: mr.google.com; spf=pass (google.com: domain of zhenhe.yk@gmail.com designates 10.42.135.129 as permitted sender) smtp.mail=zhenhe.yk@gmail.com; dkim=pass header.i=zhenhe.yk@gmail.com Received: from mr.google.com ([10.42.135.129]) by 10.42.135.129 with SMTP id p1mr240852ict.11.1330048356750 (num_hops = 1); Thu, 23 Feb 2012 17:52:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:to:references:in-reply-to:subject:date:message-id:mime-version :content-type:content-transfer-encoding:x-mailer:thread-index :content-language; bh=Avkd5ih/bBpGtOnoJ4VEJNa+skbclST6Yf6ijPOnNJw=; b=rsQWje38FaRjCsnmOSWibdDNb7DsQBQF3jxildn2lfhkHLf2DV1KwHEQ/5/toyaVDh F/v8yTVJvvbw7pK359FQHg5CeXq7h7gKj0zh0OUa6WADZevKWMXCYlAItNaK5eYYHTw2 bNDZILesMGWc+/t0rze3aZHtxQkecfa5BKlsE= Received: by 10.42.135.129 with SMTP id p1mr200214ict.11.1330048356682; Thu, 23 Feb 2012 17:52:36 -0800 (PST) Received: from TAOBAO130E9655 ([121.0.29.113]) by mx.google.com with ESMTPS id va6sm264875igc.4.2012.02.23.17.52.34 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 23 Feb 2012 17:52:36 -0800 (PST) From: "yuan kui" To: References: <4f4600e5.691d440a.6e94.201bSMTPIN_ADDED@mx.google.com> <001301ccf213$5e070300$1a150900$@tian@ia.ac.cn> <4f46e2db.0718440a.2d8a.328bSMTPIN_ADDED@mx.google.com> In-Reply-To: <4f46e2db.0718440a.2d8a.328bSMTPIN_ADDED@mx.google.com> Subject: The distributed cache object hdfs://xxxxxxxx changed during the job from TIMESTAMP to TIMESTAMP Date: Fri, 24 Feb 2012 09:52:31 +0800 Message-ID: <4f46ed64.46ed320a.217c.0a3c@mx.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: AczyDfU/LHrCU2iUR3SEByJ2RFu6GAAA0T0wAB/JqyAAAVXRgA== Content-Language: zh-cn X-Virus-Checked: Checked by ClamAV on apache.org Hi,all I was running a hbase bulkload job with ImportTsv.jar when I got an = Exception: ----------------------------------------------- 12/02/24 08:05:42 INFO mapred.JobClient: Running job: = job_201202092344_617328 12/02/24 08:05:43 INFO mapred.JobClient: map 0% reduce 0%=20 12/02/24 08:06:52 INFO mapred.JobClient: Task Id : = attempt_201202092344_617328_m_000000_0, Status : FAILED Error initializing attempt_201202092344_617328_m_000000_0: java.io.IOException: The distributed cache object = hdfs://dump002002.cm8:9000/user/admin/partitions_1330041940677#_partition= .lst changed during the job from 2/24/12 8:05 AM to 2/24/12 8:06 AM at = org.apache.hadoop.filecache.TrackerDistributedCacheManager.downloadCacheO= bject(TrackerDistributedCacheManager.java:391) at = org.apache.hadoop.filecache.TrackerDistributedCacheManager.localizePublic= CacheObject(TrackerDistributedCacheManager.java:465) at = org.apache.hadoop.filecache.TrackerDistributedCacheManager.getLocalCache(= TrackerDistributedCacheManager.java:191) at = org.apache.hadoop.filecache.TaskDistributedCacheManager.setupCache(TaskDi= stributedCacheManager.java:183) at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1184) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation= .java:1127) at = org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)= at = org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1090) at = org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2291) at = org.apache.hadoop.mapred.TaskTracker$StartNewTask.run(TaskTracker.java:21= 36) 12/02/24 08:06:52 WARN mapred.JobClient: Error reading task = outputhttp://dump002102.cm8:50060/tasklog?plaintext=3Dtrue&attemptid=3Dat= tempt_201202092344_617328_m_000000_0&filter=3Dstdout 12/02/24 08:06:52 WARN mapred.JobClient: Error reading task = outputhttp://dump002102.cm8:50060/tasklog?plaintext=3Dtrue&attemptid=3Dat= tempt_201202092344_617328_m_000000_0&filter=3Dstderr 12/02/24 08:06:52 INFO mapred.JobClient: Task Id : = attempt_201202092344_617328_m_000001_0, Status : FAILED Error initializing attempt_201202092344_617328_m_000001_0: java.io.IOException: The distributed cache object = hdfs://dump002002.cm8:9000/user/admin/partitions_1330041940677#_partition= .lst changed during the job from 2/24/12 8:05 AM to 2/24/12 8:06 AM at = org.apache.hadoop.filecache.TrackerDistributedCacheManager.downloadCacheO= bject(TrackerDistributedCacheManager.java:391) at = org.apache.hadoop.filecache.TrackerDistributedCacheManager.localizePublic= CacheObject(TrackerDistributedCacheManager.java:465) at = org.apache.hadoop.filecache.TrackerDistributedCacheManager.getLocalCache(= TrackerDistributedCacheManager.java:191) at = org.apache.hadoop.filecache.TaskDistributedCacheManager.setupCache(TaskDi= stributedCacheManager.java:183) at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1184) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation= .java:1127) at = org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)= at = org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1090) at = org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2291) --------------------------------------------------------------------- Although the same job may run successfully later again. But failure = Makes me miserable. Is this a hadoop bug? How to avoid this?