hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joep Rottinghuis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Date Mon, 08 Oct 2012 21:04:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471855#comment-13471855
] 

Joep Rottinghuis commented on HADOOP-8899:
------------------------------------------

IIRC this is not an OS limit, but this is a limit of cmd.exe

Wouldn't jarring and unjarring end up being slow? (suppose better slow than not working).
You'd also have to make sure that the classpath order is preserved. Classloaders will generally
load the first hit, so if you create your own jar you need to either drop duplicates, or make
sure they are in order.
With signed jars duplicate entries can be a challenge.

I haven't looked at the code, but can this be avoided through the user of an environment variable
instead of expanding the classpath on the command-line?
Along similar lines I recall switching from System.exec to the user of Procbuilder for similar
problems. Not sure if relevant here.
                
> Classpath exceeds maximum OS limit
> ----------------------------------
>
>                 Key: HADOOP-8899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 1-win
>            Reporter: Ahmed El Baz
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start
a hive job, and it fails with the following error:
>  java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set
CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This
is causing the classpath to be too long giving the error above. A viable long term fix is
to generate a temporary JAR file in the task cache directory which includes all elements in
the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned.
This is gated for Windows only.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message