hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-316) YARN container launch may exceed maximum Windows command line length due to long classpath
Date Tue, 22 Jan 2013 01:05:11 GMT

     [ https://issues.apache.org/jira/browse/YARN-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth updated YARN-316:

    Attachment: YARN-316-branch-trunk-win.1.patch

The attached patch is a port of the strategy used for branch-1-win MapReduce in HADOOP-8899.
 On Windows, we build a temporary jar that contains only a manifest.  The manifest's Class-Path
attribute references all of the classpath entries.  The Class-Path attribute is not subject
to the Windows command line length limitation, so it can grow as large as needed.

Within the Class-Path attribute, environment variables are not expanded automatically and
wildcards are not supported.  The patch includes code to evaluate environment variables and
resolve wildcard patterns before writing the Class-Path attribute, so Windows can still use
classpath entries like %HADOOP_HOME%/share/hadoop/common/*.

While working on this, I discovered a few related problems for Windows compatibility.  {{RawLocalFileSystem#setPermission}}
used an extra leading '0' in a chmod command.  This was harmless on Linux, bu winutils rejects
it, so I removed it.  Several lines of code in {{DefaultContainerExecutor}} and {{ContainerLaunch}}
called {{Path.toUri().getPath()}} instead of {{Path.toString()}}.  {{Path.toString()}} is
needed to get the expected behavior for handling a Windows path with a drive spec: it removes
the leading '/' before the drive spec.

I also discovered a problem across the YARN test suites on Windows.  The tests tend to create
very long container working directories deep in the Maven build hierarchy.  cmd.exe cannot
execute a script at an absolute path longer than 260 characters.  This was causing numerous
tests to fail while trying to run default_container_executor.cmd.  To resolve this, I shortened
the directories by switching calls to {{Class#getName}} to {{Class#getSimpleName}}.  Then,
I created a symlink at a shorter path in temporary storage, targeting the full path to the
working directory.  Tests then access the working directory through the short symlink to avoid
hitting the path length limitation.  (See changes in {{MiniMRClientClusterFactory}} and {{MiniYARNCluster}}.)

{{org.apache.hadoop.mapred.TestClusterMapReduceTestCase}} is an example of a test suite that
was failing on Windows, but it passes now with this patch.

> YARN container launch may exceed maximum Windows command line length due to long classpath
> ------------------------------------------------------------------------------------------
>                 Key: YARN-316
>                 URL: https://issues.apache.org/jira/browse/YARN-316
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 3.0.0, trunk-win
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: YARN-316-branch-trunk-win.1.patch
> On Windows, a command line longer than 8192 characters will fail.  This can cause YARN
container launch to fail on Windows if the classpath argument exceeds this limit.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message