hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4517) Too many INFO messages written out during AM to RM heartbeat
Date Tue, 13 Nov 2012 20:14:13 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe updated MAPREDUCE-4517:
----------------------------------

    Attachment: MAPREDUCE-4517.patch

Thanks for the review, Tom.  I modified the logging to only log a before and after for the
entire heartbeat (and only if something changed).  Also abbreviated field names in the log
message so it's not such a long line.

I moved out the headroom logging and log only in debug since that's likely to change often
while the job is running and it often isn't interesting when it does.  It's usually interesting
when the AM isn't preempting reducers to make room for maps, and we can always turn on debug
via the logLevel webpage to dynamically enable the logging of the headroom to figure things
out.
                
> Too many INFO messages written out during AM to RM heartbeat
> ------------------------------------------------------------
>
>                 Key: MAPREDUCE-4517
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4517
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster
>    Affects Versions: 0.23.1, 2.0.1-alpha
>            Reporter: James Kinley
>            Assignee: Jason Lowe
>            Priority: Minor
>              Labels: patch
>         Attachments: MAPREDUCE-4517.patch, MAPREDUCE-4517.patch, MAPREDUCE-4517.patch,
MAPREDUCE-4517.patch, MAPREDUCE-4517.patch
>
>
> Too many INFO log messages written out during AM to RM heartbeat. Based on default frequency
of 1000ms (scheduler.heartbeat.interval-ms) either 2 or 4 INFO messages are written out per
second:
> LOG.info("Before Scheduling: " + getStat());
> List<Container> allocatedContainers = getResources();
> LOG.info("After Scheduling: " + getStat());
> if (allocatedContainers.size() > 0) {
>   LOG.info("Before Assign: " + getStat());
>   scheduledRequests.assign(allocatedContainers);
>   LOG.info("After Assign: " + getStat());
> }
> These should probably be changed to DEBUG message to save the log growing too quickly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message