hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Bowen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1406) Metrics based on Map-Reduce Counters are not cleaned up
Date Tue, 22 May 2007 01:55:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497667
] 

David Bowen commented on HADOOP-1406:
-------------------------------------


As part of fixing this, I would like to propose two small API changes in org.apache.hadoop.metrics:

[1] Currently, to remove metrics records from the metrics library's internal table you have
to iterate over them one by one, since removal requires the record name and all of its tags
(name/value pairs).  I propose to generalize this so that you can specify the name and a subset
of the tags - saving the client code the burden of having to remember all the tag/values that
it has used.  This doesn't change any API signatures, but it changes the meaning of MetricsRecord.remove().

[2] Also, as a convenience, I propose to add a method MetricsRecord.removeTag(String tagName).
 This method would be added to the MetricsRecordImpl class in the spi package also.  This
enables the use pattern:

   On module initialization:
       create a metrics record named foo
       set long-lived tag (e.g. job id)

   Then repeatedly:
       set short-lived tags (e.g. counter name)
       set metric values
       call MetricsRecord.update

   And finally, on cleanup:
       remove the short-lived tags
       call MetricsRecord.remove to clean up all the metric data created

Both of these should be binary compatible changes for people using the public package o.a.h.metrics.



> Metrics based on Map-Reduce Counters are not cleaned up
> -------------------------------------------------------
>
>                 Key: HADOOP-1406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1406
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: David Bowen
>         Assigned To: David Bowen
>
> When map-reduce jobs are finished, the metrics corresponding to their counters are not
cleaned up.  This is a memory leak, but worse it means that an ever-increasing amount of metric
data is sent to the metrics system (if one is enabled).
> The fix is for JobInProgress to clean up the metrics it created when the job is complete.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message