hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jinglong.liujl (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-7105) [IPC] Improvement of lock mechanism in Listener and Reader thread
Date Wed, 09 Feb 2011 14:53:57 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992507#comment-12992507
] 

jinglong.liujl commented on HADOOP-7105:
----------------------------------------

1. Firstly, My test is take on baidu's branch and made comperation between with and without
this patch.
We use  NNThroughputBenchmark, but it has a little different with community trunk. it add
nn.throughput.bench.rpcmode in NNThroughputBenchmark to make rpc test. I'll create an extra
issue for it.

2. Our dfs.name.dir is single SATA diskļ¼Œ 7 reader threads, backlog is 10240.
./bin/hadoop org.apache.hadoop.hdfs.NNThroughputBenchmark -op create -threads 10000 -files
10000
with nn.throughput.bench.rpcmode is true (means use rpc to call namenode function)




> [IPC] Improvement of lock mechanism in Listener and Reader thread
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: jinglong.liujl
>         Attachments: improveListenerLock.patch, improveListenerLock2.patch
>
>
> In many client cocurrent access, single thread Listener will become bottleneck. Many
client can't be served, and get connection time out.
> To improve Listener capacity, we make 2 modification.
> 1.  Tuning ipc.server.listen.queue.size to a larger value to avoid client retry.
> 2. In currently implement, Listener will call registerChannel(), and finishAdd() in Reader,
which will request Reader synchronized lock. Listener will cost too many time in waiting for
this lock.
> We have made test, 
> ./bin/hadoop org.apache.hadoop.hdfs.NNThroughputBenchmark  -op create -threads 10000
-files 10000
> case 1 : Currently 
> can not pass. and report 
> hadoop-rd101.jx.baidu.com/10.65.25.166:59310. Already tried 0 time(s).
> case 2 : tuning back log to 10240
> average cost : 1285.72 ms
> case 3 : tuning back log to 10240 , and improve lock mechanism in patch
> average cost :  941.32 ms
> performance in average cost will improve 26%

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message