hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "ZooKeeper/ServiceLatencyOverview" by PatrickHunt
Date Tue, 27 Oct 2009 05:31:30 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "ZooKeeper/ServiceLatencyOverview" page has been changed by PatrickHunt.
http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview?action=diff&rev1=4&rev2=5

--------------------------------------------------

  == ZooKeeper service latencies under various loads & configurations ==
  
- The intent of this document is to test the performance of various loads on a ZooKeeper service.
+ The intent of this document is to examine the performance a ZooKeeper service under various
loads. This is not an exhaustive test but meant to shed some light on the how much load can
be supported for various hardware configurations. In particular I was interested in seeing
how non-dedicated servers performed while varying the number of cpus and clients.
  
  === Hardware ===
  
+ All systems were :
+ 
- All systems were dual quad core Intel(R) Xeon(R) CPUs running at 2.50GHz. Eight cores were
available however as noted below Linux's processor affinity feature was used to limit CPU
availability to the JVM.
+ dual quad core Intel(R) Xeon(R) CPUs running at 2.50GHz. Eight cores were available however
as noted below Linux's processor affinity feature was used to limit CPU availability to the
JVM.
  
  All systems had 16gig memory available, however unless specifically noted the JVM's -Xmx
option was used to limit the size of the JVM heap to 512m.
+ 
+ All systems had 7200RPM SATA drives.
  
  === Operating System ===
  
  Linux version 2.6.18-53.1.13.el5 compiled using gcc version 4.1.2 20070626 (Red Hat 4.1.2-14))
  
+ === Software ===
+ 
+  * ZooKeeper 3.2.1
+  * java version "1.6.0_05" (build 1.6.0_05-b13)
+  * Java HotSpot(TM) Server VM (build 10.0-b19, mixed mode)
+ 
+ 
  === ZooKeeper Configuration ===
  
  The following basic configuration was used on all serving nodes:
  
+ {{{
  tickTime=2000
  initLimit=10
  syncLimit=5
  dataDir=<data directory>
  clientPort=2181
+ }}}
+ 
  A dedicated log device was not used, the data directory is store on the same device as the
operating system, the ZK snapshots and log4j logs.
  
  The standard zkServer.sh script that ships with the release was used to start the service.
No modification were made other than to control processor affinity and heap size as noted
in this document.
  
  === Test Client ===
  
- zk-latencies.py from http://github.com/phunt/zk-smoketest was used as the test client.
+  * zk-latencies.py from http://github.com/phunt/zk-smoketest was used as the test client.
+  * python 2.4.3
- 
- The client was run as follows unless otherwise noted:
- 
- {{{
- zk-latencies.py --servers \"host60:2181\" --root_znode=/`hostname` --znode_size=100 --znode_count=10000
--timeout=5000 --watch_multiple=5
- }}}
- 
- This means each client:
-  * has a 5 second timeout
-  * creates 10k znodes of size 100 bytes
-  * sets 5 watches on each of the znodes it creates, for 50k znodes by each client
  
  == Results ==
  
- the results are as follows for standalone and 3 node ensemble
- 
- == Results for Standalone Server ==
- 
- === 1 Core - 1 Client ===
- 
- 110k operations against the server with elapsed time of 18 seconds
- 
- {{{
- host40: Testing latencies on server host60:2181 using asynchronous calls
- host40: created   10000 permanent znodes  in   1219 ms (0.121911 ms/op 8202.684317/sec)
- host40: set       10000           znodes  in   1600 ms (0.160094 ms/op 6246.310789/sec)
- host40: get       10000           znodes  in   1744 ms (0.174442 ms/op 5732.564236/sec)
- host40: deleted   10000 permanent znodes  in   1386 ms (0.138648 ms/op 7212.503931/sec)
- host40: created   10000 ephemeral znodes  in   1608 ms (0.160836 ms/op 6217.525272/sec)
- host40: watched   50000           znodes  in   7509 ms (0.150180 ms/op 6658.670068/sec)
- host40: deleted   10000 ephemeral znodes  in   2139 ms (0.213949 ms/op 4674.011338/sec)
- host40: notif     50000           watches in      0 ms (included in prior)
- host40: Latency test complete
- host40: 15.20user 7.06system 0:17.74elapsed 125%CPU (0avgtext+0avgdata 0maxresident)k
- host40: 0inputs+0outputs (0major+71244minor)pagefaults 0swaps
- }}}
- 
- === 1 Core - 10 Client ===
- 
- 1.1 million operations against the server with elapsed time of 58 seconds
- 
- {{{
- host46: Testing latencies on server host60:2181 using asynchronous calls
- host46: created   10000 permanent znodes  in   6493 ms (0.649358 ms/op 1539.982995/sec)
- host46: set       10000           znodes  in   4538 ms (0.453877 ms/op 2203.239641/sec)
- host46: get       10000           znodes  in  13229 ms (1.322971 ms/op 755.874244/sec)
- host46: deleted   10000 permanent znodes  in   4967 ms (0.496763 ms/op 2013.030379/sec)
- host46: created   10000 ephemeral znodes  in   5196 ms (0.519624 ms/op 1924.468834/sec)
- host46: watched   50000           znodes  in  17421 ms (0.348424 ms/op 2870.067528/sec)
- host46: deleted   10000 ephemeral znodes  in   4901 ms (0.490112 ms/op 2040.351571/sec)
- host46: notif     50000           watches in      0 ms (included in prior)
- host46: Latency test complete
- host46: 15.94user 6.40system 0:57.40elapsed 38%CPU (0avgtext+0avgdata 0maxresident)k
- host46: 0inputs+0outputs (2major+69981minor)pagefaults 0swaps
- }}}
- 
- === 1 Core - 20 Client ===
- 
- 2.2 million operations against the server with elapsed time of 150 seconds
- 
- {{{
- host67: Testing latencies on server host60:2181 using asynchronous calls
- host67: created   10000 permanent znodes  in  12881 ms (1.288103 ms/op 776.335485/sec)
- host67: set       10000           znodes  in  10448 ms (1.044807 ms/op 957.114388/sec)
- host67: get       10000           znodes  in  14859 ms (1.485911 ms/op 672.988006/sec)
- host67: deleted   10000 permanent znodes  in  17420 ms (1.742049 ms/op 574.036736/sec)
- host67: created   10000 ephemeral znodes  in  13450 ms (1.345097 ms/op 743.440587/sec)
- host67: watched   50000           znodes  in  53507 ms (1.070143 ms/op 934.454518/sec)
- host67: deleted   10000 ephemeral znodes  in  26196 ms (2.619615 ms/op 381.735542/sec)
- host67: notif     50000           watches in      0 ms (included in prior)
- host67: Latency test complete
- host67: 16.16user 6.43system 2:29.38elapsed 15%CPU (0avgtext+0avgdata 0maxresident)k
- host67: 0inputs+0outputs (2major+73149minor)pagefaults 0swaps
- }}}
- 
- === 2 Core - 1 Client ===
- 

Mime
View raw message