hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12711) deadly hdfs test
Date Fri, 27 Oct 2017 21:16:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222841#comment-16222841
] 

Allen Wittenauer commented on HDFS-12711:
-----------------------------------------

Adding some better metrics gathering for YETUS-561 so we have some raw numbers to look at.
 Also working on some automatic child killing drones in YETUS-570.

In the process of this, learned that while running trunk HDFS unit tests, thread count spikes
up to ~4000 threads on their own. I think some tests or maybe even some core code need some
major tuning.  It's pretty easy to see that if those JVMs were to stall, the thread count
is going to climb to ridiculous heights.

In other news, all Yetus-based Hadoop jobs on Jenkins are now running this code with the process
limit set to 5k.  This is likely too few for mapreduce, given it's proclivity to bringing
up multiple clusters in their entirety on a regular basis.


> deadly hdfs test
> ----------------
>
>                 Key: HDFS-12711
>                 URL: https://issues.apache.org/jira/browse/HDFS-12711
>             Project: Hadoop HDFS
>          Issue Type: Test
>    Affects Versions: 2.9.0, 2.8.2
>            Reporter: Allen Wittenauer
>            Priority: Critical
>         Attachments: HDFS-12711.branch-2.00.patch, MAPREDUCE-4980.016.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message