hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Mitic (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4369) Fix streaming job failures with WindowsResourceCalculatorPlugin
Date Thu, 28 Jun 2012 18:53:44 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403360#comment-13403360
] 

Ivan Mitic commented on MAPREDUCE-4369:
---------------------------------------

bq. Actually, the cause of this bug was a null JVMContext and not an invalid ProcessPid. I
should have kept the other process pid changes separate from this bug fix.
Right, there are two fixes in your change. I still believe that it would be useful to refactor
{{ResourceCalculatorPlugin#getResourceCalculatorPlugin()}} to accept processPid and explicitly
pass it on the call site.

On a related note, I'm not a fan of the fallback to {{System.getenv().get("JVM_PID")}} on
Windows, that's why I asked for the list of tests :) Might be fine at this point, but it is
something we should take a look at later.
                
> Fix streaming job failures with WindowsResourceCalculatorPlugin
> ---------------------------------------------------------------
>
>                 Key: MAPREDUCE-4369
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4369
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>         Attachments: MAPREDUCE-4369.branch-1-win.1.patch
>
>
> Some streaming jobs use local mode job runs that do not start tasks trackers. In these
cases, the jvm context is not setup and hence local mode execution causes the code to crash.
> Fix is to not not use ResourceCalculatorPlugin in such cases or make the local job run
creating dummy jvm contexts. Choosing the first option because thats the current implicit
behavior in Linux. The ProcfsBasedProcessTree (used inside the LinuxResourceCalculatorPlugin)
does no real work when the process pid is not setup correctly. This is what happens when local
job mode runs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message