hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12984) BlockPoolSlice can leak in a mini dfs cluster
Date Mon, 08 Jan 2018 17:27:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316626#comment-16316626
] 

Robert Joseph Evans commented on HDFS-12984:
--------------------------------------------

[~ajayydv],

I also ran into issues trying to reproduce this in some environments.  Specifically I could
never make it happen on my MBP and I don't know why.  But if you look at the code inside the
BlockPoolSlice 

https://github.com/apache/hadoop/blob/01f3f2167ec20b52a18bc2cf250fb4229cfd2c14/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java#L165-L173

If an instance of this is ever created it can never be collected.  I am not sure why BlockPoolSlice
instances are created some times by a MiniDFSCluster and not others.  I am not that familiar
with the internals of the DataNode to say off the top of my head.  Glad to see you going in
the right direction, and I agree that removing everything from the ShutdownHooksManager is
far from ideal, but I didn't see this happening, at least not with 2.7.5 and 2.6.2.

> BlockPoolSlice can leak in a mini dfs cluster
> ---------------------------------------------
>
>                 Key: HDFS-12984
>                 URL: https://issues.apache.org/jira/browse/HDFS-12984
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.7.5
>            Reporter: Robert Joseph Evans
>            Assignee: Ajay Kumar
>         Attachments: Screen Shot 2018-01-05 at 4.38.06 PM.png, Screen Shot 2018-01-05
at 5.26.54 PM.png, Screen Shot 2018-01-05 at 5.31.52 PM.png
>
>
> When running some unit tests for storm we found that we would occasionally get out of
memory errors on the HDFS integration tests.
> When I got a heap dump I found that the ShutdownHookManager was full of BlockPoolSlice$1
instances.  Which hold a reference to the BlockPoolSlice which then in turn holds a reference
to the DataNode etc....
> It looks like when shutdown is called on the BlockPoolSlice there is no way to remove
the shut down hook in because no reference to it is saved.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message