hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4369) Fix streaming job failures with WindowsResourceCalculatorPlugin
Date Fri, 29 Jun 2012 05:16:45 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403701#comment-13403701

Bikas Saha commented on MAPREDUCE-4369:

1) Thats because ResourceCalculatorPlugin.getResourceCalculatorPlugin() is called multiple
times and it makes to initialize it once. Also, the process id is not available at all call
sites. Ideally I would like to replace use of ResourceCalculatorPlugin with ResourceCalculatorProcessTree
in all these places but I dont want to do cross file refactoring in a branch.

2) Thats because in the normal case the JVM_PID is defined and used. In the local runner case,
there is no JVM_PID defined and hence I chose to leave resourceCalculator null in that case.
The code knows how to deal with null resourceCalculator but I am wary of leaving a poorly
initialized resource calculator. Until now we were lucky that the poorly initialized resourceCalculator
did not behave badly and safely ignored the null pid. However, it would return incorrect resource
values. I think its better to null it and be explicit that it is not present vs use an invalid
plugin. Also, I have changed direct use of JVM_PID env with using the jvmContext. That leaves
the dependence of initializing task pid at 1 place(initialization of jvmContext).
> Fix streaming job failures with WindowsResourceCalculatorPlugin
> ---------------------------------------------------------------
>                 Key: MAPREDUCE-4369
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4369
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>         Attachments: MAPREDUCE-4369.branch-1-win.1.patch, MAPREDUCE-4369.branch-1-win.2.patch
> Some streaming jobs use local mode job runs that do not start tasks trackers. In these
cases, the jvm context is not setup and hence local mode execution causes the code to crash.
> Fix is to not not use ResourceCalculatorPlugin in such cases or make the local job run
creating dummy jvm contexts. Choosing the first option because thats the current implicit
behavior in Linux. The ProcfsBasedProcessTree (used inside the LinuxResourceCalculatorPlugin)
does no real work when the process pid is not setup correctly. This is what happens when local
job mode runs.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message