hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "ZooKeeper/ServiceLatencyOverview" by PatrickHunt
Date Wed, 28 Oct 2009 04:55:45 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "ZooKeeper/ServiceLatencyOverview" page has been changed by PatrickHunt.
http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview?action=diff&rev1=16&rev2=17

--------------------------------------------------

   * zk-latencies.py from http://github.com/phunt/zk-smoketest was used as the test client.
   * python 2.4.3
  
+ == Result Summary ==
+ 
+ In the following two graphs you can see the results of this experiment. The X axis shows
the number of operations per second that were processed by the server for 1, 2 & 4 cores,
where 1, 10 & 20 clients were testing the cluster.
+ 
+ ||'''Standalone Server'''||'''Ensemble with 3 Serving Nodes'''||
+ ||{{attachment:standalone_perf.png|Standalone Server}}||{{attachment:ensemble_3_perf.png|Ensemble
with 3 serving nodes}}||
+ 
+ Each client is performing 110,000 asynchronous operations against the service, where the
workload is:
+  * create 10k permanent znodes of size 100 bytes
+  * set 10k znodes of size 100 bytes
+  * get 10k znodes (100 bytes returned)
+  * delete all 10k znodes
+  * create 10k ephemeral znodes of size 100 bytes
+  * watch each of the 10k znodes 5 times (50k watches)
+  * delete 10k znodes and measure time for notifications to be received by the client (from
the 50k watches)
+ 
+ Of the 110k operations 50k are write operations, the remaining 60k are read operations.
Read operations being local to the connected server while writes must be propagated through
the leader & atomic broadcast protocol (ie copied to all servers in an ensemble)
+ 
+ === Standalone Result ===
+ 
+ From the graph and the data below we can see that moving from a workload of 1 to 10 clients
shows an increase in overall operations performed by the server. However moving from 10 to
20 clients we see no further increase (rather some decrease, potentially due to the overhead
of selector processing and connection management in the server networking code). Increasing
the number of cores provides modest increase in the overall output of the server. This makes
sense as there are a limited number of threads running in a standalone server, it is unable
to take advantage of the additional cores.
+ 
+ === Ensemble with 3 Serving Nodes Result ===
+ 
+ Here we see distinct increase in the output of the cluster as we move the workload from
1 to 10 to 20 clients. There is some additional headroom that we could see if we were to test
with higher workloads. Additionally as the core count increases we see output gains there
as well. Ensemble servers are running additional threads to process the quorum communication,
these threads are taking advantage of the additional parallelism.
+ 
+ === Conclusion ===
+ 
+ I hope this sheds more light on the tradeoffs that can be made when setting up a ZK cluster.
In particular we can see the performance of a small cluster (reliable even in the face of
a single server going down) is quite solid even under a heavy (30k+ operations per second
on dual core with a high write to read ratio, much higher than you should typically see in
a production deployment) client workload. Additionally the session stability was rock solid
- no timeouts or session expirations were seen during any of the tests.
+ 
+ === Notes, Futures ===
+ 
+ I used the out of the box zkServer.sh for this test. I did look at using the incremental
garbage collector however as far as I could tell there was no benefit seen for these tests.
It might be interesting to look at this more, is there some benefit to be had? 
+ 
+ I also did not use the --server flag, again, I did some limited testing with this and afaics
there is no benefit to using this option with a ZK cluster.
+ 
+ It would be interesting to test other workloads. Here I've weighted things more to a balanced
write/read workload. In production deployments we typically see a heavy read dominant workload.
In this case the service performance should be even better than what we are seeing here.
+ 
- == Results ==
+ == Result Details ==
  
  the results are as follows for standalone and 3 node ensemble
  

Mime
View raw message