hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Lynch (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2007) Jobs use incorrect path to job.xml for different users
Date Sun, 07 Oct 2007 22:22:50 GMT
Jobs use incorrect path to job.xml for different users

                 Key: HADOOP-2007
                 URL: https://issues.apache.org/jira/browse/HADOOP-2007
             Project: Hadoop
          Issue Type: Bug
          Components: mapred
    Affects Versions: 0.14.1
         Environment: OS: Red Hat Enterprise Linux AS release 4 (Nahant Update 5) - AMD64
JRE: java full version "1.6.0_02-b05"
            Reporter: Ryan Lynch

We run hadoop/hdfs under a generic username of 'hadoop'.  When submitting a job as user 'fred',
the job.xml file gets created correctly as /tmp/*hadoop-fred*/mapred/system/<job_id>/job.xml,
but the JobInProgress constructor appears to use the incorrect path of /tmp/*hadoop-hadoop*/mapred/system/<job_id>/job.xml
for the copyToLocalFile operation.  This results in an exception because the job.xml file
does not exist under /tmp/hadoop-hadoop.

I think the incorrect path gets created on line 133 in JobInProgress.java right before the
call to copyToLocalFile.  If I hardcode the correct path (just to test my theory) of "/tmp/hadoop-fred/mapred/system"
in place of the call to default_conf.getSystemDir(), the job kicks off and runs to completion
as the user 'fred' with hadoop/hdfs running as the user 'hadoop'.

Here's a portion of the output when the job is submitted by user 'fred':

[fred@hdm01 ~]$ hadoop --config /usr/local/hadoop/conf jar /usr/local/hadoop/current/hadoop-0.14.1-examples.jar
wordcount mytest/input/data output2
07/10/07 17:52:14 INFO mapred.FileInputFormat: Total input paths to process : 62
org.apache.hadoop.ipc.RemoteException: java.io.IOException: /tmp/hadoop-hadoop/mapred/system/job_200710071751_0002/job.xml:
No such file or directory
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:138)
        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:803)
        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:784)
        at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:134)
        at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1479)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message