hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
Date Tue, 01 Aug 2017 05:28:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108392#comment-16108392
] 

Weiwei Yang edited comment on HDFS-12163 at 8/1/17 5:27 AM:
------------------------------------------------------------

Hi [~anu]

Here is the result after set 20 handler for KSM,

| (distributed,1) | init | 6 |
| (distributed,1) | MiniOzoneCluster | 222 |
| (distributed,1) | shutdown | 80 |
| (distributed,1) | sleep | 13 |

I will overwrite this value to 20 in {{MiniOzoneCluster}} since 200 is not necessary for testing.


bq. Xiaoyu Yao has fixed a bunch of leak issues, they were due to us not closing the miniOzoneCluster.

Yes, it looks like some of the leaks are already fixed, we can see the number of open threads
decreased to 13 now. I will take a look if there is any other potential leak and fix them
if I found any.


was (Author: cheersyang):
Hi [~anu]

Here is the result after set 20 handler for KSM,

| (distributed,1) | init | 6 |
| (distributed,1) | MiniOzoneCluster | 222 |
| (distributed,1) | shutdown | 80 |
| (distributed,1) | sleep | 13 |

I will overwrite this value to 20 in {{MiniOzoneCluster}} since 200 is not necessary for testing.


> Ozone: MiniOzoneCluster uses 400+ threads
> -----------------------------------------
>
>                 Key: HDFS-12163
>                 URL: https://issues.apache.org/jira/browse/HDFS-12163
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone, test
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Weiwei Yang
>         Attachments: most_used_threads.png, TestOzoneThreadCount20170719.patch, thread_dump.png
>
>
> Checked the number of active threads used in MiniOzoneCluster with various settings:
> - Local handlers
> - Distributed handlers
> - Ratis-Netty
> - Ratis-gRPC
> The results are similar for all the settings.  It uses 400+ threads for an 1-datanode
MiniOzoneCluster.
> Moreover, there is a thread leak -- a number of the threads do not shutdown after the
test is finished.  Therefore, when tests run consecutively, the later tests use more threads.
> Will post the details in comments.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message