hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Changming Sun (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4000) Create jobTokenPassword file fail when run hadoop pipes locally
Date Tue, 13 Mar 2012 07:15:40 GMT
Create jobTokenPassword file fail when run hadoop pipes  locally
----------------------------------------------------------------

                 Key: MAPREDUCE-4000
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4000
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: pipes
    Affects Versions: 0.23.3
         Environment: Fedora release 16
Linux localhost.localdomain 3.2.7-1.fc16.x86_64 #1 SMP Tue Feb 21 01:40:47 UTC 2012 x86_64
x86_64 x86_64 GNU/Linux
Apache Maven 3.0.4 (r1232337; 2012-01-17 16:44:56+0800)
Maven home: /usr/local/apache-maven-3.0.4
Java version: 1.7.0_03, vendor: Oracle Corporation
Java home: /usr/java/jdk1.7.0_03/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.2.7-1.fc16.x86_64", arch: "amd64", family: "unix"
            Reporter: Changming Sun
            Priority: Minor


Create jobTokenPassword file fail when run hadoop pipes  locally.

I have put such settings in my job conf.

<property>
  <name>mapreduce.jobtracker.address</name>
  <value>local</value>
</property>

<property>
  <name>mapreduce.framework.name</name>
  <value>local</value>
</property>

<property>
  <name>fs.defaultFS</name>
  <value>file:///</value>
</property>

This job has one map task ,and one reduce task. It will fail when running PipesReducer, because
"jobTokenPassword" file is created at current working directory,with permission 0400.
org.apache.hadoop.mapred.pipes.Application:
 private void writePasswordToLocalFile(String localPasswordFile,
      byte[] password, JobConf conf) throws IOException {
    FileSystem localFs = FileSystem.getLocal(conf);
    Path localPath = new Path(localPasswordFile);
    FSDataOutputStream out = FileSystem.create(localFs, localPath,
        new FsPermission("400"));
    out.write(password);
    out.close();
  }

So, it will fail at the second time.

In such situation,the application cannot be initialized, but the PipesReducer.close() method
will be called after that ,so there will be NullPointer Exception raise in close() method.





--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message