hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Derek Dagit (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-256) Increase retention of applications in AppManager without sending more applications to the UI
Date Mon, 10 Dec 2012 06:19:22 GMT

     [ https://issues.apache.org/jira/browse/YARN-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Derek Dagit updated YARN-256:
-----------------------------

    Attachment: YARN-256.patch

New patch with JavaDoc and test.

Using the LinkedHashMap did not work, because although it is a doubly-linked list, it does
not keep track of the tail of the list.  This means we cannot use it to efficiently add the
most recently completed apps, as we would have to iterate through to the end.

Instead, we use a Map to store ID->App mappings as before, but we move the list of completed
apps to the RMContext and add a Set of non-completed Apps for guaranteeing we do not forget
to send the UI every app that is still running.

If the max number of apps configured for the UI is not met by just the non-completed apps,
then we back-fill the most recently completed apps from the completed apps list.
                
> Increase retention of applications in AppManager without sending more applications to
the UI
> --------------------------------------------------------------------------------------------
>
>                 Key: YARN-256
>                 URL: https://issues.apache.org/jira/browse/YARN-256
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 0.23.5
>            Reporter: Derek Dagit
>            Assignee: Derek Dagit
>         Attachments: YARN-256-branch-0.23.patch, YARN-256-branch-0.23-wip.patch, YARN-256-branch-0.23-wip.patch,
YARN-256.patch
>
>
> In very busy clusters we would like to retain applications longer so that users' links
will not expire too soon.  Very often links to application history expire before they can
be followed.
> Simply increasing max-completed applications has an adverse performance impact on the
applications list in the web UI because it presents the entire list of applications with a
request.
> Therefore, we would like some way to be able to increase the retention of applications
without increasing the number of applications sent to the Web UI.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message