hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks
Date Thu, 19 Apr 2018 22:29:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444891#comment-16444891
] 

Sean Mackrory edited comment on HADOOP-15392 at 4/19/18 10:28 PM:
------------------------------------------------------------------

{quote}the behavior in the event no sinks are configured would not be to just leak memory
forever, so I wonder if there's something else we're doing that's wrong that we can just fix.{quote}
In the hopes of finding an answer to this today, I compared this to the YARN Resource Manager
cluster class and I'm not seeing any significant differences in terms of how the MetricsRegistry
is instantiated & referenced and how the raw data flows to it. If memory got leaked further
down in metrics2, I would think it would have manifested in somebody's ResourceManager by
now.

I also looked through detailed logs of what happens when closing filesystems in all of the
tests (as some tests instantiate dozens of filesystems and then either close them or just
end) and the ref counting appears to be functioning perfectly in all of those cases.

And probably not directly related, but even without any changes I'm getting this one test
failure, also in the metrics:

{code}ITestS3AMetrics.testMetricsRegister:42->Assert.assertNotNull:621->Assert.assertTrue:41->Assert.fail:88
No metrics under test fs for S3AMetrics1-mackrory{code}


was (Author: mackrorysd):
{quote}the behavior in the event no sinks are configured would not be to just leak memory
forever, so I wonder if there's something else we're doing that's wrong that we can just fix.{quote}
In the hopes of finding an answer to this today, I compared this to the YARN Resource Manager
cluster class and I'm not seeing any significant differences in terms of how the MetricsRegistry
is instantiated & referenced and how the raw data flows to it. If memory got leaked further
down in metrics2, I would think it would have manifested in somebody's ResourceManager by
now.

I also looked through detailed logs of what happens when closing filesystems in all of the
tests (as some tests instantiate dozens of filesystems and then either close them or just
end) and the ref counting appears to be functioning perfectly in all of those cases.

> S3A Metrics in S3AInstrumentation Cause Memory Leaks
> ----------------------------------------------------
>
>                 Key: HADOOP-15392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15392
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Voyta
>            Priority: Major
>
> While using HBase S3A Export Snapshot utility we started to experience memory leaks of
the process after version upgrade.
> By running code analysis we traced the cause to revision 6555af81a26b0b72ec3bee7034e01f5bd84b1564
that added the following static reference (singleton):
> private static MetricsSystem metricsSystem = null;
> When application uses S3AFileSystem instance that is not closed immediately metrics
are accumulated in this instance and memory grows without any limit.
>  
> Expectation:
>  * It would be nice to have an option to disable metrics completely as this is not needed
for Export Snapshot utility.
>  * Usage of S3AFileSystem should not contain any static object that can grow indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message