hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Min Zhou <coderp...@gmail.com>
Subject Questions on HADOOP-6713
Date Fri, 17 Dec 2010 04:45:14 GMT
Hi all,

After apply the patch from HADOOP-6713 into our hadoop source tree,  I
did a benchmark on multi-readers RPC, and didn't found any  throughput
improvement .
The attachment of this mail is a patch of my rpc benmark. Test
scenario listed below.

Network: Gigabit LAN
CPU: 8 core Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
Memory: 8GB

================== scenario 1 =======================
server side:
# export HADOOP_OPS="-Xmx4096m -Xms256m"
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
-handlers 1 -readers 10 -durationPerRPC 100

client side:
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
-host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
10/12/17 12:25:07 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
10/12/17 12:26:02 INFO ipc.RPCBenchmark: # operations: 100
10/12/17 12:26:02 INFO ipc.RPCBenchmark: Elapsed Time: 55073 ms
10/12/17 12:26:02 INFO ipc.RPCBenchmark:  Ops per sec: 1.815771793800955
10/12/17 12:26:02 INFO ipc.RPCBenchmark: Average Time: 5249 ms

================== scenario 2 =======================
server side:
# export HADOOP_OPS="-Xmx4096m -Xms256m"
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
-handlers 10 -readers 10 -durationPerRPC 100

client side:
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
-host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
10/12/17 12:28:23 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
10/12/17 12:29:11 INFO ipc.RPCBenchmark: # operations: 100
10/12/17 12:29:11 INFO ipc.RPCBenchmark: Elapsed Time: 47784 ms
10/12/17 12:29:11 INFO ipc.RPCBenchmark:  Ops per sec: 2.092750711535242
10/12/17 12:29:11 INFO ipc.RPCBenchmark: Average Time: 4581 ms

================== scenario 3 =======================
server side:
# export HADOOP_OPS="-Xmx4096m -Xms256m"
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
-handlers 10 -readers 1 -durationPerRPC 100

client side:
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
-host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
10/12/17 12:31:19 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
10/12/17 12:32:08 INFO ipc.RPCBenchmark: # operations: 100
10/12/17 12:32:08 INFO ipc.RPCBenchmark: Elapsed Time: 49155 ms
10/12/17 12:32:08 INFO ipc.RPCBenchmark:  Ops per sec: 2.034381039568711
10/12/17 12:32:08 INFO ipc.RPCBenchmark: Average Time: 4722 ms

================== scenario 4 =======================
server side:
# export HADOOP_OPS="-Xmx4096m -Xms256m
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
-handlers 1 -readers 1 -durationPerRPC 100

client side:
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
-host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
10/12/17 12:36:04 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
10/12/17 12:36:59 INFO ipc.RPCBenchmark: # operations: 100
10/12/17 12:36:59 INFO ipc.RPCBenchmark: Elapsed Time: 55054 ms
10/12/17 12:36:59 INFO ipc.RPCBenchmark:  Ops per sec: 1.816398445162931
10/12/17 12:36:59 INFO ipc.RPCBenchmark: Average Time: 5278 ms

Statistics above indicates that a single reader can consume 200,000
8-bytes Strings very fast.  Ten concurrent readers should consume the
same data more quickly, but reader isn't the bottleneck of RPC at
least in these scenarios. Can you give an example which scenario
HADOOP-6713 will take affect?

Thanks,
Min
-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message