hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-968) Reduce shuffle and merge should be done a child JVM
Date Tue, 17 Apr 2007 06:30:15 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Devaraj Das updated HADOOP-968:
-------------------------------

    Attachment: 968-with-metrics-fix.patch

This patch addresses an issue to do with metrics reporting wherein stopMonitoring was not
called for the ReduceTask. This potentially could lead to hung ReduceTasks after they are
finished since the task JVM might not be able to exit until the (non-daemon) monitoring thread
goes away.

> Reduce shuffle and merge should be done a child JVM
> ---------------------------------------------------
>
>                 Key: HADOOP-968
>                 URL: https://issues.apache.org/jira/browse/HADOOP-968
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.10.1
>            Reporter: Owen O'Malley
>         Assigned To: Devaraj Das
>             Fix For: 0.13.0
>
>         Attachments: 968-reindent.patch, 968-with-metrics-fix.patch, 968.apr06.patch,
968.apr10.patch, 968.apr14.patch, 968.apr14.patch, 968.patch
>
>
> The Reduce's shuffle and initial merge is done in the TaskTracker's JVM. It would be
better to have it run in the Task's child JVM. The advantages are:
>   1. The class path and environment would be set up correctly.
>   2. User code doesn't need to be loaded into the TaskTracker.
>   3. Lower memory usage and contention in the TaskTracker.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message