hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmed El Baz (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-8899) Classpath exceeds maximum OS limit
Date Tue, 09 Oct 2012 20:30:03 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ahmed El Baz updated HADOOP-8899:

    Attachment: HADOOP-8899.branch-1-win.classpath.patch

Please note the following:

- The generated Jar does not encapsulate any other jars, it simply references all entries
in the classpath in its Manifest, so this is just another way to propagate the long list of
Jars to the tasks to launch.

- Classpath elements are referenced in the generated Jar's Manifest in the same order.

- This change is only gated on Windows where the Java classpath argument now has the generated
Jar (which references the other ones), instead of the long list which is giving the problem.
For non-Windows OS, the behavior is just the same as before.

- We do not explicitly unjar the file, we feed it to Jave to lookup the classpath elements
from the manifest.

- The patch also includes testing performance changes (in TestJvmManager): in Windows, launchTask
is only affected by ~3%, in Linux no impact since the logic is the same.

I have attached a patch with the changes (including test changes)
> Classpath exceeds maximum OS limit
> ----------------------------------
>                 Key: HADOOP-8899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 1-win
>            Reporter: Ahmed El Baz
>         Attachments: HADOOP-8899.branch-1-win.classpath.patch
> We hit errors in TT due to long classpath value. One example is Oozie trying to start
a hive job, and it fails with the following error:
>  java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This
is causing the classpath to be too long giving the error above. A viable long term fix is
to generate a temporary JAR file in the task cache directory which includes all elements in
the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned.
This is gated for Windows only.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message