hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jinglong.liujl (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-7105) [IPC] Improvement of lock mechanism in Listener and Reader thread
Date Fri, 14 Jan 2011 17:05:46 GMT
[IPC] Improvement of lock mechanism in Listener and Reader thread
-----------------------------------------------------------------

                 Key: HADOOP-7105
                 URL: https://issues.apache.org/jira/browse/HADOOP-7105
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc
    Affects Versions: 0.21.0
            Reporter: jinglong.liujl
         Attachments: improveListenerLock.patch

In many client cocurrent access, single thread Listener will become bottleneck. Many client
can't be served, and get connection time out.
To improve Listener capacity, we make 2 modification.
1.  Tuning ipc.server.listen.queue.size to a larger value to avoid client retry.
2. In currently implement, Listener will call registerChannel(), and finishAdd() in Reader,
which will request Reader synchronized lock. Listener will cost too many time in waiting for
this lock.

We have made test, 

./bin/hadoop org.apache.hadoop.hdfs.NNThroughputBenchmark  -op create -threads 10000 -files
10000

case 1 : Currently 
can not pass. and report 
hadoop-rd101.jx.baidu.com/10.65.25.166:59310. Already tried 0 time(s).

case 2 : tuning back log to 10240
average cost : 1285.72 ms

case 3 : tuning back log to 10240 , and improve lock mechanism in patch
average cost :  941.32 ms


performance in average cost will improve 26%



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message