hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Vasudev (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-3294) Allow dumping of Capacity Scheduler debug logs via web UI for a fixed time period
Date Fri, 20 Mar 2015 14:23:38 GMT

     [ https://issues.apache.org/jira/browse/YARN-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Varun Vasudev updated YARN-3294:
    Attachment: apache-yarn-3294.2.patch

Thanks for the review Jian!

1. will it dump the logs for package “org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity”
or “org.apache.hadoop.yarn.server.resourcemanager.scheduler”. I think it should be dumping
latter as some scheduler logs sit in common

Log log = LogFactory.getLog(CapacityScheduler.class.getPackage().getName());

2. the newly added code can be in a separate class, maybe call it SchedulerLogDumper

3. Overall, do you think it's useful to make this as a generic log dumper ? i.e. parameterize
the class name of the log to be dumped.


Addressed all three. I made it a standalone class called AdHocLogDumper that takes the log
name and dumps the log. It now dumps the log for "org.apache.hadoop.yarn.server.resourcemanager.scheduler".

    private Logger packageLogger; does not need to be a global value


    dumping logs is useful for large amount of logs. Do you think it's also useful to surface
the logs on the web page.

I think we should skip it for now. On a single node cluster with one pi job running, it dumped
about 200K of logs in 1 min. I'm worried with larger clusters, the log size will be too large.
The log will still be accessible via the /logs link.

> Allow dumping of Capacity Scheduler debug logs via web UI for a fixed time period
> ---------------------------------------------------------------------------------
>                 Key: YARN-3294
>                 URL: https://issues.apache.org/jira/browse/YARN-3294
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: Screen Shot 2015-03-12 at 8.51.25 PM.png, apache-yarn-3294.0.patch,
apache-yarn-3294.1.patch, apache-yarn-3294.2.patch
> It would be nice to have a button on the web UI that would allow dumping of debug logs
for just the capacity scheduler for a fixed period of time(1 min, 5 min or so) in a separate
log file. It would be useful when debugging scheduler behavior without affecting the rest
of the resourcemanager.

This message was sent by Atlassian JIRA

View raw message