hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shanyu zhao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-9924) FileUtil.createJarWithClassPath() does not generate relative classpath correctly
Date Fri, 06 Sep 2013 21:45:52 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

shanyu zhao updated HADOOP-9924:
--------------------------------

    Attachment: HADOOP-9924.addendum.patch

There is a corner case the previous patch does not handler correctly. When there's empty string
included in the classpath, "new Path(classPathEntry)" will through an exception. 

The manifest of this problem is that when you try to run a pig job, the job will fail because
the application master is failed to start.

The addendum patch will handle the empty string scenario. Unit test cases are updated to test
this scenario as well.
                
> FileUtil.createJarWithClassPath() does not generate relative classpath correctly
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-9924
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9924
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.1.0-beta, 0.23.9
>            Reporter: shanyu zhao
>            Assignee: shanyu zhao
>             Fix For: 2.1.1-beta
>
>         Attachments: HADOOP-9924-2.patch, HADOOP-9924-3.patch, HADOOP-9924-4.patch, HADOOP-9924.addendum.patch,
HADOOP-9924.patch
>
>
> On Windows, FileUtil.createJarWithClassPath() is called to generate a manifest jar file
to pack classpath - to avoid the problem of classpath being too long.
> However, the relative classpath is not handled correctly. It relies on Java's File(relativePath)
to resolve the relative path. But it really should be using the given pwd parameter to resolve
the relative path.
> To reproduce this bug, you can try some pig job on Windows, it will fail and the pig
log on the application master will look like this:
> 2013-08-29 23:25:55,498 INFO [main] org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat
not found
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat not found
> This is because the PigOutputFormat class is in the job.jar file but the classpath manifest
has:
> file:/c:/apps/dist/hadoop-2.1.0-beta/bin/job.jar/job.jar
> When it really should be:
> file:/<job container folder>/job.jar/job.jar

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message