hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks
Date Tue, 24 Apr 2018 21:26:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451243#comment-16451243
] 

Sean Mackrory edited comment on HADOOP-15392 at 4/24/18 9:25 PM:
-----------------------------------------------------------------

{quote}if you or someone else would care to provide a switch to turn this off{quote}

If there's a problem we can't fix I'd be happy to, but there shouldn't be any significant
difference between how the relatively short-lived connector is accumulating metrics and how
long-running daemons do it. If there's a memory leak then either we're doing it wrong and
we should fix it (unless the fix is worse than disabling metrics by default or something),
or this is a very big problem indeed that may be affecting other Hadoop daemons. I ran a profiler
and saw 2 metrics-related threads pop up: one is an executor for MetricsSinkAdapter that tends
to grow by about 6kb every period of the metrics system (which by default is 10 seconds) and
there's another one for MutableQuantiles that grows by about 64 bytes per second. Now I don't
see why either of those needs to grow indefinitely like that, but that rate also sounds insignificant
compared to the kind of growth being described here, and the behavior is the same between
the other Hadoop daemons that were unaffected by the change in question and S3a clients. It
also doesn't appear to happen unless you have metrics sinks configured, etc. and the default
configuration doesn't have any of that. I also ran a bunch of HBase snapshot exports to S3
both with and without this change and I'm not seeing any particular pattern in the memory
usage that matches what you're describing. So a couple of follow-up questions:

* To be clear, exactly where are you seeing the growth in memory usage? As I understand it
this is in the MapReduce job that exports snapshots to S3, right? If that's the case, can
you identify a particular thread that seems to be accumulating all the memory? I initially
thought you were referring to one of the HBase daemons, but if this is the MR job then whether
or not it closes the FileSystem is probably rather academic because it would close the FileSystem
as the JVM was about to shut down anyway, so it likely doesn't affect whether or not you see
a problem with memory usage.
* Do you have anything in hadoop-metrics2.properties at all?


was (Author: mackrorysd):
{quote}if you or someone else would care to provide a switch to turn this off{quote}

If there's a problem we can't fix I'd be happy to, but there shouldn't be any significant
difference between how the relatively short-lived connector is accumulating metrics and how
long-running daemons do it. If there's a memory leak then either we're doing it wrong and
we should fix it (unless the fix is worse than disabling metrics by default or something),
or this is a very big problem indeed that may be affecting other Hadoop daemons. I ran a profiler
and saw 2 metrics-related threads pop up: one is an executor for MetricsSinkAdapter that tends
to grow by about 6kb every period of the metrics system (which by default is 10 seconds) and
there's another one for MutableQuantiles that grows by about 64 bytes per second. Now I don't
see why either of those needs to grow indefinitely like that, but that rate also sounds insignificant
compared to the kind of growth being described here, and the behavior is the same between
the other Hadoop daemons that were unaffected by the change in question and S3a clients. It
also doesn't appear to happen unless you have metrics sinks configured, etc. and the default
configuration doesn't have any of that. I also ran a bunch of HBase snapshot exports to S3
both with and without this change and I'm not seeing any particular pattern in the memory
usage that matches what you're describing. So a couple of follow-up questions:

*To be clear, exactly where are you seeing the growth in memory usage? As I understand it
this is in the MapReduce job that exports snapshots to S3, right? If that's the case, can
you identify a particular thread that seems to be accumulating all the memory? I initially
thought you were referring to one of the HBase daemons, but if this is the MR job then whether
or not it closes the FileSystem is probably rather academic because it would close the FileSystem
as the JVM was about to shut down anyway, so it likely doesn't affect whether or not you see
a problem with memory usage.
* Do you have anything in hadoop-metrics2.properties at all?

> S3A Metrics in S3AInstrumentation Cause Memory Leaks
> ----------------------------------------------------
>
>                 Key: HADOOP-15392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15392
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Voyta
>            Priority: Major
>
> While using HBase S3A Export Snapshot utility we started to experience memory leaks of
the process after version upgrade.
> By running code analysis we traced the cause to revision 6555af81a26b0b72ec3bee7034e01f5bd84b1564
that added the following static reference (singleton):
> private static MetricsSystem metricsSystem = null;
> When application uses S3AFileSystem instance that is not closed immediately metrics
are accumulated in this instance and memory grows without any limit.
>  
> Expectation:
>  * It would be nice to have an option to disable metrics completely as this is not needed
for Export Snapshot utility.
>  * Usage of S3AFileSystem should not contain any static object that can grow indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message