Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 24757 invoked from network); 11 May 2009 17:35:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 11 May 2009 17:35:26 -0000 Received: (qmail 41595 invoked by uid 500); 11 May 2009 17:35:23 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 41500 invoked by uid 500); 11 May 2009 17:35:23 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 41490 invoked by uid 99); 11 May 2009 17:35:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 May 2009 17:35:23 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lance@dotspots.com designates 209.85.132.241 as permitted sender) Received: from [209.85.132.241] (HELO an-out-0708.google.com) (209.85.132.241) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 May 2009 17:35:15 +0000 Received: by an-out-0708.google.com with SMTP id c2so1646166anc.29 for ; Mon, 11 May 2009 10:34:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.100.166.7 with SMTP id o7mr16144450ane.90.1242063292976; Mon, 11 May 2009 10:34:52 -0700 (PDT) Date: Mon, 11 May 2009 10:34:52 -0700 Message-ID: Subject: Constantly getting DiskErrorExceptions - but logged as INFO From: Lance Riedel To: core-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0016e645b71c5dceac0469a6656a X-Virus-Checked: Checked by ClamAV on apache.org --0016e645b71c5dceac0469a6656a Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Trying to figure out what the scenario that would cause the following errors. Since it is logged as INFO I wasn't sure if it was from speculative execution, or if there is something more serious happening. Anyone seen these errors? They occur a lot. Also, note: We have plenty of disk space on this machine, and we aren't using /tmp - so doesn't seem like the folders would be deleted by a cron job/other. Logs: 2009-05-10 23:58:18,737 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_2997/attempt_200905071127_2997_m_000001_1/output/file.out in any of the configured local directories 2009-05-10 23:58:18,737 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_3408/attempt_200905071127_3408_r_000015_1/output/file.out in any of the configured local directories 2009-05-10 23:58:23,808 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_2997/attempt_200905071127_2997_m_000001_1/output/file.out in any of the configured local directories 2009-05-10 23:58:23,809 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_3408/attempt_200905071127_3408_r_000015_1/output/file.out in any of the configured local directories 2009-05-10 23:58:28,810 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_2997/attempt_200905071127_2997_m_000001_1/output/file.out in any of the configured local directories 2009-05-10 23:58:28,810 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_3408/attempt_200905071127_3408_r_000015_1/output/file.out in any of the configured local directories 2009-05-10 23:58:33,811 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_2997/attempt_200905071127_2997_m_000001_1/output/file.out in any of the configured local directories 2009-05-10 23:58:33,812 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_3408/attempt_200905071127_3408_r_000015_1/output/file.out in any of the configured local directories 2009-05-10 23:58:38,813 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_2997/attempt_200905071127_2997_m_000001_1/output/file.out in any of the configured local directories 2009-05-10 23:58:38,813 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_3408/attempt_200905071127_3408_r_000015_1/output/file.out in any of the configured local directories 2009-05-10 23:58:43,814 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_2997/attempt_200905071127_2997_m_000001_1/output/file.out in any of the configured local directories 2009-05-10 23:58:43,814 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_3408/attempt_200905071127_3408_r_000015_1/output/file.out in any of the configured local directories 2009-05-10 23:58:48,815 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_2997/attempt_200905071127_2997_m_000001_1/output/file.out in any of the configured local directories 2009-05-10 23:58:48,815 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_3408/attempt_200905071127_3408_r_000015_1/output/file.out in any of the configured local directories 2009-05-10 23:58:53,816 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200905071127_2997/attempt_200905071127_2997_m_000001_1/output/file.out in any of the configured local directories --0016e645b71c5dceac0469a6656a--