hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4322) Fix command-line length abort issues on Windows
Date Wed, 27 Jun 2012 02:23:52 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401890#comment-13401890

Bikas Saha commented on MAPREDUCE-4322:

1) TaskLog.java
Wouldnt renaming these from bash* to shell* be better. Since these are private members the
renaming would not cause much grief upon merging the code.
  private static final String bashCommand = (Shell.WINDOWS)? "cmd": "bash";
  private static final String bashCommandSufix = (Shell.WINDOWS) ? "/c" : "-c";
  private static final String bashCommandNullOutput = 
      (Shell.WINDOWS) ? "< nul" : "< /dev/null";
2) TestTaskLog.java
Could you please replace 8192 with TaskLog.MAX_CMD_LINE_LENGTH
for (int i = 0; i < 8192; ++i) {
3) TestTaskLog.java
For the 2 places you really mean "setup.toString()" and "cmd.toString()" instead of sb.toString()

The current refactoring makes it cleaner. Agree on separate jira for a better implementation.
> Fix command-line length abort issues on Windows
> -----------------------------------------------
>                 Key: MAPREDUCE-4322
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4322
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>         Environment: Windows, downstream applications with long aggregate classpaths
>            Reporter: John Gordon
>            Assignee: Ivan Mitic
>         Attachments: MAPREDUCE-4322-branch-1-win(2).patch, MAPREDUCE-4322-branch-1-win(3).patch,
>   Original Estimate: 12h
>  Remaining Estimate: 12h
> When a task is started on the tasktracker, it creates a small batch file to invoke java
and runs that batch.  Within the batch file, the invocation of Java currently has -classpath
${CLASSPATH} inline to the command.  That line often exceeds 8000 characters.  This is ok
for most linux distributions because the line limit env variable is often set much higher
than this.  However, for Windows this cause cmd to abort execution.  This surfaces in Hadoop
as an unknown failure mode for the task.
> I think the easiest and most natural way to fix this is to push the -classpath option
into a config file to take the longest variable part of the line and put it somewhere that
scales better.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message