hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luca Pireddu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4000) Create jobTokenPassword file fail when run hadoop pipes locally
Date Tue, 08 May 2012 14:19:49 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270482#comment-13270482
] 

Luca Pireddu commented on MAPREDUCE-4000:
-----------------------------------------

Stack trace, from Hadoop 1.0.2:

{{
12/05/08 16:11:32 INFO mapred.JobClient: Task Id : attempt_201205081608_0001_m_000000_2, Status
: FAILED
java.io.FileNotFoundException: /u/pireddu/Projects/pydoop.git/examples/wordcount/test/jobTokenPassword
(Permission denied)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:188)
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:184)
        at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:255)
        at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:236)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:435)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:315)
        at org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:170)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:106)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
}}
                
> Create jobTokenPassword file fail when run hadoop pipes  locally
> ----------------------------------------------------------------
>
>                 Key: MAPREDUCE-4000
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4000
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: pipes
>    Affects Versions: 1.0.1, 1.0.2, 0.23.3
>         Environment: Fedora release 16
> Linux localhost.localdomain 3.2.7-1.fc16.x86_64 #1 SMP Tue Feb 21 01:40:47 UTC 2012 x86_64
x86_64 x86_64 GNU/Linux
> Apache Maven 3.0.4 (r1232337; 2012-01-17 16:44:56+0800)
> Maven home: /usr/local/apache-maven-3.0.4
> Java version: 1.7.0_03, vendor: Oracle Corporation
> Java home: /usr/java/jdk1.7.0_03/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.2.7-1.fc16.x86_64", arch: "amd64", family: "unix"
>            Reporter: Changming Sun
>            Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Create jobTokenPassword file fail when run hadoop pipes  locally.
> I have put such settings in my job conf.
> {code:xml} 
> <property>
>   <name>mapreduce.jobtracker.address</name>
>   <value>local</value>
> </property>
> <property>
>   <name>mapreduce.framework.name</name>
>   <value>local</value>
> </property>
> <property>
>   <name>fs.defaultFS</name>
>   <value>file:///</value>
> </property>
> {code} 
> This job has one map task ,and one reduce task. It will fail when running PipesReducer,
because "jobTokenPassword" file is created at current working directory,with permission 0400.
> {code:title=org.apache.hadoop.mapred.pipes.Application.java} 
>  private void writePasswordToLocalFile(String localPasswordFile,
>       byte[] password, JobConf conf) throws IOException {
>     FileSystem localFs = FileSystem.getLocal(conf);
>     Path localPath = new Path(localPasswordFile);
>     FSDataOutputStream out = FileSystem.create(localFs, localPath,
>         new FsPermission("400"));
>     out.write(password);
>     out.close();
>   }
> {code}
> So, it will fail at the second time.
> In such situation,the application cannot be initialized, but the PipesReducer.close()
method will be called after that ,so there will be NullPointer Exception raise in close()
method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message