hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12711) deadly hdfs test
Date Fri, 27 Oct 2017 21:16:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222841#comment-16222841

Allen Wittenauer commented on HDFS-12711:

Adding some better metrics gathering for YETUS-561 so we have some raw numbers to look at.
 Also working on some automatic child killing drones in YETUS-570.

In the process of this, learned that while running trunk HDFS unit tests, thread count spikes
up to ~4000 threads on their own. I think some tests or maybe even some core code need some
major tuning.  It's pretty easy to see that if those JVMs were to stall, the thread count
is going to climb to ridiculous heights.

In other news, all Yetus-based Hadoop jobs on Jenkins are now running this code with the process
limit set to 5k.  This is likely too few for mapreduce, given it's proclivity to bringing
up multiple clusters in their entirety on a regular basis.

> deadly hdfs test
> ----------------
>                 Key: HDFS-12711
>                 URL: https://issues.apache.org/jira/browse/HDFS-12711
>             Project: Hadoop HDFS
>          Issue Type: Test
>    Affects Versions: 2.9.0, 2.8.2
>            Reporter: Allen Wittenauer
>            Priority: Critical
>         Attachments: HDFS-12711.branch-2.00.patch, MAPREDUCE-4980.016.patch

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message