hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yingqi Lu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-9861) Add SO_REUSEPORT into HDFS
Date Fri, 26 Feb 2016 05:21:18 GMT
Yingqi Lu created HDFS-9861:

             Summary: Add SO_REUSEPORT into HDFS
                 Key: HDFS-9861
                 URL: https://issues.apache.org/jira/browse/HDFS-9861
             Project: Hadoop HDFS
          Issue Type: New Feature
            Reporter: Yingqi Lu

Hello All,

SO_REUSEPORT is a kernel feature which enables multiple listen sockets to listen to the same
IP:PORT pair. This can potentially improve the scalability and parallelism of network traffic
handling, especially for big core count systems or large sized clusters. The feature is enabled
for both TCP and UDP sockets. Linux, BSD and majority of other UNIX based OSes already have
the feature supported. Windows has not enabled it yet. For more details on the feature itself,
please refer to https://lwn.net/Articles/542629/. 

Recently, SO_REUSEPORT is added into OpenJDK9 as one of the socket options. For more details,
please refer to http://hg.openjdk.java.net/jdk9/dev/jdk/rev/f92af283ab18#l13.1.

Here, we are proposing to add SO_REUSEPORT support into HDFS for its NameNode and DataNode
implementations. We have a quick prototype which shows good throughput and latency improvements
on given micro workload (purely HDFS workload, not MapReduce based). We will wrap up all the
details together with a patch and send them here soon for further discussion and review. At
the meantime, please let us know your comment and feedback.


This message was sent by Atlassian JIRA

View raw message