Return-Path: 
 <core-dev-return-68424-apmail-hadoop-core-dev-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-core-dev-archive@www.apache.org
Received: (qmail 99979 invoked from network); 11 May 2009 16:16:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 May 2009 16:16:11 -0000
Received: (qmail 25878 invoked by uid 500); 11 May 2009 16:16:10 -0000
Delivered-To: apmail-hadoop-core-dev-archive@hadoop.apache.org
Received: (qmail 25800 invoked by uid 500); 11 May 2009 16:16:10 -0000
Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:core-dev-help@hadoop.apache.org>
List-Unsubscribe: <mailto:core-dev-unsubscribe@hadoop.apache.org>
List-Post: <mailto:core-dev@hadoop.apache.org>
List-Id: <core-dev.hadoop.apache.org>
Reply-To: core-dev@hadoop.apache.org
Delivered-To: mailing list core-dev@hadoop.apache.org
Received: (qmail 25790 invoked by uid 99); 11 May 2009 16:16:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 May 2009 16:16:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 May 2009 16:16:06 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A633D234C041
	for <core-dev@hadoop.apache.org>; Mon, 11 May 2009 09:15:45 -0700 (PDT)
Message-ID: <1041869991.1242058545679.JavaMail.jira@brutus>
Date: Mon, 11 May 2009 09:15:45 -0700 (PDT)
From: "Arun Jacob (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Created: (HADOOP-5802) issue creating buffer directory when
 using native S3FileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

issue creating buffer directory when using native S3FileSystem
--------------------------------------------------------------

                 Key: HADOOP-5802
                 URL: https://issues.apache.org/jira/browse/HADOOP-5802
             Project: Hadoop Core
          Issue Type: Bug
         Environment: ec2, Cloudera AMI, m-xlarge, 20 nodes. 
            Reporter: Arun Jacob


Note the following settings in hadoop-site.xml, which  means I've configured 40 reduce tasks to run across 20 nodes. I checked Jira, and found HADOOP-4377, which refers to this as a race condition that has been fixed, the bug was closed. I'm re-opening because this may be configuration specific, e.g. I am running multiple reduce tasks on the same node (no mention was made of that in the original bug)

<property>
  <name>mapred.reduce.tasks</name>
  <value>40</value>
</property>
<property>
  <name>mapred.tasktracker.reduce.tasks.maximum</name>
  <value>4</value>
  <final>true</final>
</property>

One of the tasks fails trying to create the S3 buffer directory:


attempt_200905111058_0001_r_000019_0: 1589020 [main] WARN org.apache.hadoop.mapred.TaskTracker  - Error running child
attempt_200905111058_0001_r_000019_0: java.io.IOException: Cannot create S3 buffer directory: /mnt/tmp/hadoop-hadoop/s3
attempt_200905111058_0001_r_000019_0:   at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.newBackupFile(NativeS3FileSystem.java:152)
attempt_200905111058_0001_r_000019_0:   at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.<init>(NativeS3FileSystem.java:136)
attempt_200905111058_0001_r_000019_0:   at org.apache.hadoop.fs.s3native.NativeS3FileSystem.create(NativeS3FileSystem.java:278)
attempt_200905111058_0001_r_000019_0:   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:503)
attempt_200905111058_0001_r_000019_0:   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:403)
attempt_200905111058_0001_r_000019_0:   at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:117)
attempt_200905111058_0001_r_000019_0:   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:293)
attempt_200905111058_0001_r_000019_0:   at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.