hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5270) Use thread pools in the datenode daemons
Date Fri, 27 Sep 2013 22:17:02 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Haohui Mai updated HDFS-5270:

    Attachment: TestConcurrentAccess.java

Here is a test to generate concurrent workloads.

I ran the test to write 50000 files into a 3-node cluster using 320 threads.

When running the test multiple times, the test fails (DFSClient retries) starting from the
second runs, while the patch allows the test to run correctly.
> Use thread pools in the datenode daemons
> ----------------------------------------
>                 Key: HDFS-5270
>                 URL: https://issues.apache.org/jira/browse/HDFS-5270
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>         Attachments: HDFS-5270.000.patch, TestConcurrentAccess.java
> The current implementation of the datanode creates a thread when a new request comes
in. This incurs high overheads for the creation / destruction of threads, making the datanode
unstable under high concurrent loads.
> This JIRA proposes to use a thread pool to reduce the overheads.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message