hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM
Date Sat, 20 Aug 2011 00:27:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088090#comment-13088090

Todd Lipcon commented on MAPREDUCE-2649:

Coming late to see this patch. One question from an MR2 n00b: in MR1, these configs were very
error prone since different jobs took up wildly variant amounts of RAM. So a value of 100
retained jobs would work fine for small jobs but fail miserably with OOME if bigger jobs were
submitted. Then we added all of the limits to try to curtail this, but it's a bit of a stopgap
rather than a real solution.

Is this better in MR2 because the state on the RM is much smaller? Or is there still per-container
state being retained?

One thing we should consider is to adopt an interface like HBase's "HeapSize" -- objects that
are going to be retained/cached need to implement it, and then the limit scan be set based
on memory usage rather than strict counts.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch,
MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
> Today RM keeps the references of finished application for ever. Though this is not sustainable
long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message