Thanks a lot Michi. Yes that works like a charm. Appreciated your help. Now coming back to my original question as I am interested in doing load testing on my 3 node ZK cluster. So I ran "zk-latencies.py" script to do the load test and this is what I got on my screen - check@phx5tg03c-cfc7:~/zooksmoketesting$ PYTHONPATH=lib.linux-i686-2.6 LD_LIBRARY_PATH=lib.linux-x86_64-2.6 ./zk-latencies.py --servers "host1:2181,host2:2181,host3:2181" --znode_count=100 --znode_size=100 --synchronous Connecting to host1:2181 Connected in 547 ms, handle is 0 Connecting to host2:2181 Connected in 3 ms, handle is 1 Connecting to host3:2181 Connected in 8 ms, handle is 2 Testing latencies on server host1:2181 using syncronous calls created 100 permanent znodes in 117 ms (1.178288 ms/op 848.688614/sec) set 100 znodes in 112 ms (1.129220 ms/op 885.567396/sec) get 100 znodes in 39 ms (0.390978 ms/op 2557.689587/sec) deleted 100 permanent znodes in 107 ms (1.071839 ms/op 932.975652/sec) created 100 ephemeral znodes in 111 ms (1.118169 ms/op 894.319330/sec) watched 100 znodes in 35 ms (0.354149 ms/op 2823.667540/sec) deleted 100 ephemeral znodes in 110 ms (1.102810 ms/op 906.774590/sec) notif 100 watches in 1000 ms (10.007010 ms/op 99.929954/sec) Testing latencies on server host2:2181 using syncronous calls created 100 permanent znodes in 108 ms (1.088130 ms/op 919.007439/sec) set 100 znodes in 104 ms (1.049511 ms/op 952.824304/sec) get 100 znodes in 27 ms (0.277209 ms/op 3607.382816/sec) deleted 100 permanent znodes in 133 ms (1.330690 ms/op 751.489612/sec) created 100 ephemeral znodes in 102 ms (1.021519 ms/op 978.934593/sec) watched 100 znodes in 32 ms (0.327361 ms/op 3054.735079/sec) deleted 100 ephemeral znodes in 110 ms (1.109312 ms/op 901.459984/sec) notif 100 watches in 1001 ms (10.010960 ms/op 99.890519/sec) Testing latencies on server host3:2181 using syncronous calls created 100 permanent znodes in 87 ms (0.874031 ms/op 1144.124715/sec) set 100 znodes in 85 ms (0.854771 ms/op 1169.903798/sec) get 100 znodes in 33 ms (0.338831 ms/op 2951.324270/sec) deleted 100 permanent znodes in 81 ms (0.818131 ms/op 1222.298185/sec) created 100 ephemeral znodes in 81 ms (0.814011 ms/op 1228.484481/sec) watched 100 znodes in 27 ms (0.279410 ms/op 3578.971440/sec) deleted 100 ephemeral znodes in 99 ms (0.997062 ms/op 1002.946929/sec) notif 100 watches in 1001 ms (10.011010 ms/op 99.890019/sec) Latency test complete So my question is what does this line mean at the end? Does it say, it can do a get request call at a rate of 2557 requests per second? get 100 znodes in 39 ms (0.390978 ms/op 2557.689587/sec) Also I was interested to see how much load a 3 node ZK cluster can handle. In my case, most of the call will be "get" (get the data from only one znode) on the ZK cluster So with the above test I should be able to get some idea right? On Mon, Apr 7, 2014 at 11:40 PM, Michi Mutsuzaki wrote: > Does this work? > > https://paste.apache.org/9Op5?action=download > > On Mon, Apr 7, 2014 at 9:10 PM, Check Peck > wrote: > > I manually copied the jar file into the directory it was trying to copy. > Now > > if I do "ant", then I see like this - > > > > check@phx5tg03c-cfc7:~/zooksmoketest/zookeeper-3.4.6$ ant > > Buildfile: /home/check/zooksmoketest/zookeeper-3.4.6/build.xml > > > > init: > > > > ivy-download: > > > > ivy-taskdef: > > > > ivy-init: > > > > ivy-retrieve: > > [ivy:retrieve] :: Ivy 2.2.0 - 20100923230623 :: > http://ant.apache.org/ivy/ > > :: > > [ivy:retrieve] :: loading settings :: file = > > /home/check/zooksmoketest/zookeeper-3.4.6/ivysettings.xml > > [ivy:retrieve] :: resolving dependencies :: > > org.apache.zookeeper#zookeeper;3.4.6 > > [ivy:retrieve] confs: [default] > > > > > > And then after this, it doesn't move further, I waited for a long time > and > > nothing came up.. And afterwards I did ctrl + c to break. > > > > Any idea what might be wrong? > > > > > > > > On Mon, Apr 7, 2014 at 8:42 PM, Michi Mutsuzaki > > wrote: > >> > >> You probably need to set the proxy. > >> > >> http://ant.apache.org/manual/proxy.html > >> > >> On Mon, Apr 7, 2014 at 8:40 PM, Check Peck > >> wrote: > >> > Thanks a lot Michi. Appreciated your help.. I tried your suggestion > on > >> > one > >> > of my production box but when I tried "fifth" step which is "ant". I > got > >> > an > >> > error as shown below. Any thoughts what is wrong? I guess firewall is > >> > blocking. Is there any other way around for this? I manually copied > >> > "zookeeper-3.4.6.tar.gz" into that machine using "scp" as well. > >> > check@phx5tg03c-cfc7:~/zooksmoketest/zookeeper-3.4.6$ ant > >> > Buildfile: /home/check/zooksmoketest/zookeeper-3.4.6/build.xml > >> > > >> > init: > >> > > >> > ivy-download: > >> > [get] Getting: > >> > http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar > >> > [get] To: > >> > /home/check/zooksmoketest/zookeeper-3.4.6/src/java/lib/ivy-2.2.0.jar > >> > [get] Error getting > >> > http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jarto > >> > /home/check/zooksmoketest/zookeeper-3.4.6/src/java/lib/ivy-2.2.0.jar > >> > > >> > BUILD FAILED > >> > /home/check/zooksmoketest/zookeeper-3.4.6/build.xml:348: > >> > java.net.ConnectException: Connection timed out > >> > at java.net.PlainSocketImpl.socketConnect(Native Method) > >> > at > >> > > >> > > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) > >> > at > >> > > >> > > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) > >> > at > >> > > >> > > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) > >> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384) > >> > at java.net.Socket.connect(Socket.java:546) > >> > at java.net.Socket.connect(Socket.java:495) > >> > at sun.net.NetworkClient.doConnect(NetworkClient.java:178) > >> > at sun.net.www.http.HttpClient.openServer(HttpClient.java:409) > >> > at sun.net.www.http.HttpClient.openServer(HttpClient.java:530) > >> > at sun.net.www.http.HttpClient.(HttpClient.java:240) > >> > at sun.net.www.http.HttpClient.New(HttpClient.java:321) > >> > at sun.net.www.http.HttpClient.New(HttpClient.java:338) > >> > at > >> > > >> > > sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935) > >> > at > >> > > >> > > sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876) > >> > at > >> > > >> > > sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801) > >> > at > >> > > org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:660) > >> > at > org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:579) > >> > at > org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:569) > >> > > >> > Total time: 1 minute 3 seconds > >> > > >> > > >> > > >> > On Mon, Apr 7, 2014 at 3:08 PM, Michi Mutsuzaki < > michi@cs.stanford.edu> > >> > wrote: > >> >> > >> >> Hi Check, > >> >> > >> >> Something like this should work on ubuntu. > >> >> > >> >> https://paste.apache.org/6kao?action=download > >> >> > >> >> > >> >> On Sun, Apr 6, 2014 at 11:11 PM, Check Peck > > >> >> wrote: > >> >> > I am not sure from where to start to install zkpython.. I tried > >> >> > following > >> >> > that github link but was not able to understand. If you can guide > me > >> >> > what I > >> >> > am supposed to do step by step, then it will be of great help to > me. > >> >> > Thanks. > >> >> > > >> >> > > >> >> > On Sun, Apr 6, 2014 at 10:57 PM, Pavan Sudheendra > >> >> > wrote: > >> >> > > >> >> >> Hi Check, > >> >> >> > >> >> >> Where are you getting stuck? > >> >> >> > >> >> >> > >> >> >> On Mon, Apr 7, 2014 at 11:13 AM, Check Peck > >> >> >> > >> >> >> wrote: > >> >> >> > >> >> >> > Thanks. I tried going through that github link but wasn't able > to > >> >> >> > understand what I am supposed to do clearly. May be I am not > that > >> >> >> > much > >> >> >> > familiar with Python so that's why I am having hard time > figuring > >> >> >> > that > >> >> >> out? > >> >> >> > > >> >> >> > > >> >> >> > On Sun, Apr 6, 2014 at 10:06 PM, Michi Mutsuzaki > >> >> >> > >> >> >> > >wrote: > >> >> >> > > >> >> >> > > You need to install zkpython. The steps are described in the > >> >> >> > > README > >> >> >> > > file. Let us know if you have trouble installing zkpython. > >> >> >> > > > >> >> >> > > > >> >> >> > > >> >> >> > >> >> >> > >> >> >> > https://github.com/apache/zookeeper/blob/trunk/src/contrib/zkpython/README > >> >> >> > > > >> >> >> > > On Sun, Apr 6, 2014 at 9:21 PM, Check Peck > >> >> >> > > > >> >> >> > > wrote: > >> >> >> > > > I tried following the steps mentioned on this github link - > >> >> >> > > > https://github.com/phunt/zk-smoketest for zk-latencies.py. > >> >> >> > > > > >> >> >> > > > I have 3 node ZK cluster- machineA, machineB, machineC > >> >> >> > > > > >> >> >> > > > Now I copied the Python program zk-latencies.py and > >> >> >> > > > zkclient.py > >> >> >> > > > in > >> >> >> > > another > >> >> >> > > > machineD in which no zookeeper is running. It's a plain > ubuntu > >> >> >> > > > box. > >> >> >> > > > > >> >> >> > > > Now when I ran the program like this - > >> >> >> > > > > >> >> >> > > > david@machineD:~/zooksmoketest$ ./zk-latencies.py --servers > >> >> >> > > > "machineA:2181,machineB::2181,machineC:2181" > --znode_count=100 > >> >> >> > > > --znode_size=100 --synchronous > >> >> >> > > > Traceback (most recent call last): > >> >> >> > > > File "./zk-latencies.py", line 22, in > >> >> >> > > > import zkclient > >> >> >> > > > File "/home/david/zooksmoketest/zkclient.py", line 17, in > >> >> >> > > > > >> >> >> > > > import zookeeper, time, threading > >> >> >> > > > > >> >> >> > > > *ImportError: No module named zookeeper* > >> >> >> > > > > >> >> >> > > > I got an error as shown above. Any thoughts what wrong I am > >> >> >> > > > doing > >> >> >> > here? I > >> >> >> > > > don't have that much experience with Python. Any suggestions > >> >> >> > > > will > >> >> >> > > > be > >> >> >> of > >> >> >> > > > great help. > >> >> >> > > > > >> >> >> > > > @Raul: Any thoughts? > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > On Sun, Apr 6, 2014 at 6:46 PM, Check Peck > >> >> >> > > > > >> >> >> > > wrote: > >> >> >> > > > > >> >> >> > > >> Thanks guys.. I am going to do load test in couple of days > >> >> >> > > >> using > >> >> >> > > >> the > >> >> >> > > >> exiting Python framework. And I will share the results > once I > >> >> >> > > >> am > >> >> >> done. > >> >> >> > > >> > >> >> >> > > >> > >> >> >> > > >> On Tue, Mar 25, 2014 at 11:19 PM, Rakesh R > >> >> >> > > >> > >> >> >> > wrote: > >> >> >> > > >> > >> >> >> > > >>> > >> >> >> > > >>> If I understood the case correctly, > >> >> >> > > >>> Test :- 3 node ZK cluster, 600 sessions each will do one > >> >> >> > > >>> read > >> >> >> > > >>> operation(getChildren call) every one minute. > >> >> >> > > >>> > >> >> >> > > >>> I'm assuming the calls are equally distributed in time > frame > >> >> >> > > >>> and it > >> >> >> > > would > >> >> >> > > >>> be like : 600ops/60secs, average 10 ops/sec will go to 3 > >> >> >> > > >>> node > >> >> >> cluster > >> >> >> > > >>> I also feel it should be, also it would be great if you > >> >> >> > > >>> publish > >> >> >> > > >>> the > >> >> >> > > >>> results after your tests. > >> >> >> > > >>> > >> >> >> > > >>> > >> >> >> > > >>> -Rakesh > >> >> >> > > >>> > >> >> >> > > >>> -----Original Message----- > >> >> >> > > >>> From: Check Peck [mailto:comptechgeeky@gmail.com] > >> >> >> > > >>> Sent: 26 March 2014 11:19 > >> >> >> > > >>> To: user > >> >> >> > > >>> Subject: Re: Load on Zookeeper cluster? > >> >> >> > > >>> > >> >> >> > > >>> Thanks that is also quite useful. > >> >> >> > > >>> > >> >> >> > > >>> For my above use case, I just need to get the children of > a > >> >> >> > particular > >> >> >> > > >>> znode.. And that's what will be happening from all the 600 > >> >> >> > application > >> >> >> > > >>> machines.. > >> >> >> > > >>> > >> >> >> > > >>> From the wiki you provided, it looks it should be fine > >> >> >> > > >>> right? > >> >> >> > > >>> > >> >> >> > > >>> > >> >> >> > > >>> On Tue, Mar 25, 2014 at 10:13 PM, Rakesh R > >> >> >> > > >>> > >> >> >> > wrote: > >> >> >> > > >>> > >> >> >> > > >>> > > >> >> >> > > >>> > Please go through the below one, hope it will be helpful > >> >> >> > > >>> > to > >> >> >> > > >>> > you. > >> >> >> > > Also, > >> >> >> > > >>> > I would recommend to have a look at the basic zk > >> >> >> > > >>> > configurations > >> >> >> > too. > >> >> >> > > >>> > All the > >> >> >> > > >>> > best:) > >> >> >> > > >>> > > >> >> >> > > >>> > > >> >> >> > > >>> > > >> >> >> > > > >> >> >> > > > >> >> >> > > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ServiceLatencyOv > >> >> >> > > >>> > erview > >> >> >> > > >>> > > >> >> >> > > >>> > > >> >> >> > > >>> > What client operations(for ex: simply creating many > >> >> >> > > >>> > znodes, > >> >> >> > deleting, > >> >> >> > > >>> > setwatches et.) you are planning to perform. > >> >> >> > > >>> > Actually I'd like to know what each zkclient will be > >> >> >> > > >>> > doing. > >> >> >> > > >>> > > >> >> >> > > >>> > -Rakesh > >> >> >> > > >>> > > >> >> >> > > >>> > -----Original Message----- > >> >> >> > > >>> > From: Check Peck [mailto:comptechgeeky@gmail.com] > >> >> >> > > >>> > Sent: 26 March 2014 09:40 > >> >> >> > > >>> > To: user > >> >> >> > > >>> > Subject: Re: Load on Zookeeper cluster? > >> >> >> > > >>> > > >> >> >> > > >>> > On Tue, Mar 25, 2014 at 9:00 PM, Raúl Gutiérrez Segalés > < > >> >> >> > > >>> > rgs@itevenworks.net > >> >> >> > > >>> > > wrote: > >> >> >> > > >>> > > >> >> >> > > >>> > > 1 ZK se > >> >> >> > > >>> > > >> >> >> > > >>> > > >> >> >> > > >>> > > >> >> >> > > >>> > Yes one ZK session per application. Zookeeper version I > am > >> >> >> running > >> >> >> > > >>> > currently is 3.4.5 in production. > >> >> >> > > >>> > > >> >> >> > > >>> > >> >> >> > > >> > >> >> >> > > >> > >> >> >> > > > >> >> >> > > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> Regards- > >> >> >> Pavan > >> >> >> > >> > > >> > > > > > >