hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Bowen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1406) Metrics based on Map-Reduce Counters are not cleaned up
Date Tue, 22 May 2007 01:55:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497667

David Bowen commented on HADOOP-1406:

As part of fixing this, I would like to propose two small API changes in org.apache.hadoop.metrics:

[1] Currently, to remove metrics records from the metrics library's internal table you have
to iterate over them one by one, since removal requires the record name and all of its tags
(name/value pairs).  I propose to generalize this so that you can specify the name and a subset
of the tags - saving the client code the burden of having to remember all the tag/values that
it has used.  This doesn't change any API signatures, but it changes the meaning of MetricsRecord.remove().

[2] Also, as a convenience, I propose to add a method MetricsRecord.removeTag(String tagName).
 This method would be added to the MetricsRecordImpl class in the spi package also.  This
enables the use pattern:

   On module initialization:
       create a metrics record named foo
       set long-lived tag (e.g. job id)

   Then repeatedly:
       set short-lived tags (e.g. counter name)
       set metric values
       call MetricsRecord.update

   And finally, on cleanup:
       remove the short-lived tags
       call MetricsRecord.remove to clean up all the metric data created

Both of these should be binary compatible changes for people using the public package o.a.h.metrics.

> Metrics based on Map-Reduce Counters are not cleaned up
> -------------------------------------------------------
>                 Key: HADOOP-1406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1406
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: David Bowen
>         Assigned To: David Bowen
> When map-reduce jobs are finished, the metrics corresponding to their counters are not
cleaned up.  This is a memory leak, but worse it means that an ever-increasing amount of metric
data is sent to the metrics system (if one is enabled).
> The fix is for JobInProgress to clean up the metrics it created when the job is complete.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message