hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1218) Collecting cpu and memory usage for TaskTrackers
Date Wed, 06 Jan 2010 06:13:54 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797001#action_12797001

Vinod K V commented on MAPREDUCE-1218:

Looked at that latest patch. Coming out good. Some comments, most of them are finishing touches.

 - LinuxResourceCalculatorPlugin.java
    -- The method {{#getCpuUsage()}} in itself doesn't update the cpu-usage. So, if one makes
two calls to this method separated by a time interval, the result won't reflect the updated
cpu-usage unless calls to {{#getCumulativeCpuTime()}} are explicitly made in between. This
should be fixed by calling {{#readProcStatFile()}} in this method also. The main method should
be modified to call this method twice and the tests should also verify this.
    -- You've changed {{CPU_TIME_FORMAT}} from _"^cpu[0-9]*[ \t]*([0-9]*)[ \t]*([0-9]*)[ \t]*([0-9]*)[
\t].*"_ to _"^cpu[ \t]*([0-9]*)[ \t]*([0-9]*)[ \t]*([0-9]*)[ \t].*"_. I guess the earlier
is correct with cpu names not having any space/tab?
    -- Why even have MINIMUM_UPDATE_INTERVAL in updateCpuUsage()? If this is mainly used for
making sure sampleTime is not equal to lastSampleTime, then we can do so directly and remove

 - TaskTracker.java
    -- To handle the deprecation of {{mapreduce.tasktracker.memorycalculatorplugin}} in TaskTracker,
for memory calculations we should first try to use the class denoted by this configuration
if present, otherwise only we should fall back to the new resource-calculator. To facilitate
this we will also retain a deprecated {{TTConfig.TT_MEMORY_CALCULATOR_PLUGIN}} constant.
    -- Nit: +661 "LOG.info(" Using MemoryCalculatorPlugin : " + resourceCalculatorPlugin);"
should instead be "LOG.info(" Using ResourceCalculatorPlugin : " + resourceCalculatorPlugin);"

 - Please annotate Dummy{Resource|Memory}CalculatorPlugin classes as @InterfaceAudience.Private
because they both are only test specific.
 - We should document {{mapreduce.tasktracker.resourcecalculatorplugin}} in mapred-default.xml
and remove the documentation for {{mapreduce.tasktracker.memorycalculatorplugin}} from the
 - Please convert TestLinuxResourceCalculatorPlugin and TestTTResourceReporting into Junit
4 testcases.

> Collecting cpu and memory usage for TaskTrackers
> ------------------------------------------------
>                 Key: MAPREDUCE-1218
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>         Environment: linux
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>         Attachments: MAPREDUCE-1218-rename.sh, MAPREDUCE-1218-v2.patch, MAPREDUCE-1218-v3.patch,
MAPREDUCE-1218-v4.patch, MAPREDUCE-1218.patch
> The information can be used for resource aware scheduling.
> Note that this is related to MAPREDUCE-220. There the per task resource information is
> This one collects the per machine information.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message