hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gera Shegalov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5960) JobSubmitter's check whether job.jar is local is incorrect with no authority in job jar path.
Date Mon, 13 Oct 2014 23:44:35 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gera Shegalov updated MAPREDUCE-5960:
-------------------------------------
    Attachment: MAPREDUCE-5960.v02.patch

[~jlowe], thanks for looking into this JIRA. 

I agree that it might not be safe to assume that the remote source be accessible from the
server side. However, the user will find this out soon enough because the job will fail. It's
equally wrong to assume the job jar won't be accessible, and deprive users from the option
to store job bits at some other place central, such as s3. We can argue that it would be more
efficient to first automatically copy the source jar to staging dir, and have NM localize
from there as opposed to the original source. 

Setting this discussion aside, it does not look good that the submission simply fails. I added
a unit test for the scenario I believe just has to work functionally. 

> JobSubmitter's check whether job.jar is local is incorrect with no authority in job jar
path.
> ---------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5960
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5960
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 2.4.0, 2.4.1
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5960.v01.patch, MAPREDUCE-5960.v02.patch
>
>
> {code}
> $ mapred job -submit myjob.xml 
> 2014-07-06 15:06:34.731 java[12120:1903] Unable to load realm info from SCDynamicStore
> 14/07/06 15:06:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
> 14/07/06 15:06:34 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
> 14/07/06 15:06:37 WARN conf.Configuration: bad conf file: element not <property>
> 14/07/06 15:06:37 WARN conf.Configuration: bad conf file: element not <property>
> 14/07/06 15:06:37 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
> 14/07/06 15:11:11 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/gshegalov/.staging/job_1404679996131_0004
> Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: viewfs:/user/gshegalov/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar,
expected: file:///
> 	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:525)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:739)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
> 	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> 	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1902)
> 	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1870)
> 	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1835)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.copyJar(JobSubmitter.java:286)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:254)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626)
> 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> 	at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:260)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> 	at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1239)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message