zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Check Peck <comptechge...@gmail.com>
Subject Re: Load on Zookeeper cluster?
Date Tue, 08 Apr 2014 16:17:43 GMT
Thanks a lot Michi. Yes that works like a charm. Appreciated your help.

Now coming back to my original question as I am interested in doing load
testing on my 3 node ZK cluster. So I ran "zk-latencies.py" script to do
the load test and this is what I got on my screen -

    check@phx5tg03c-cfc7:~/zooksmoketesting$ PYTHONPATH=lib.linux-i686-2.6
LD_LIBRARY_PATH=lib.linux-x86_64-2.6 ./zk-latencies.py --servers
"host1:2181,host2:2181,host3:2181" --znode_count=100 --znode_size=100
--synchronous
    Connecting to host1:2181
    Connected in 547 ms, handle is 0
    Connecting to host2:2181
    Connected in 3 ms, handle is 1
    Connecting to host3:2181
    Connected in 8 ms, handle is 2
    Testing latencies on server host1:2181 using syncronous calls
    created     100 permanent znodes  in    117 ms (1.178288 ms/op
848.688614/sec)
    set         100           znodes  in    112 ms (1.129220 ms/op
885.567396/sec)
    get         100           znodes  in     39 ms (0.390978 ms/op
2557.689587/sec)
    deleted     100 permanent znodes  in    107 ms (1.071839 ms/op
932.975652/sec)
    created     100 ephemeral znodes  in    111 ms (1.118169 ms/op
894.319330/sec)
    watched     100           znodes  in     35 ms (0.354149 ms/op
2823.667540/sec)
    deleted     100 ephemeral znodes  in    110 ms (1.102810 ms/op
906.774590/sec)
    notif       100           watches in   1000 ms (10.007010 ms/op
99.929954/sec)
    Testing latencies on server host2:2181 using syncronous calls
    created     100 permanent znodes  in    108 ms (1.088130 ms/op
919.007439/sec)
    set         100           znodes  in    104 ms (1.049511 ms/op
952.824304/sec)
    get         100           znodes  in     27 ms (0.277209 ms/op
3607.382816/sec)
    deleted     100 permanent znodes  in    133 ms (1.330690 ms/op
751.489612/sec)
    created     100 ephemeral znodes  in    102 ms (1.021519 ms/op
978.934593/sec)
    watched     100           znodes  in     32 ms (0.327361 ms/op
3054.735079/sec)
    deleted     100 ephemeral znodes  in    110 ms (1.109312 ms/op
901.459984/sec)
    notif       100           watches in   1001 ms (10.010960 ms/op
99.890519/sec)
    Testing latencies on server host3:2181 using syncronous calls
    created     100 permanent znodes  in     87 ms (0.874031 ms/op
1144.124715/sec)
    set         100           znodes  in     85 ms (0.854771 ms/op
1169.903798/sec)
    get         100           znodes  in     33 ms (0.338831 ms/op
2951.324270/sec)
    deleted     100 permanent znodes  in     81 ms (0.818131 ms/op
1222.298185/sec)
    created     100 ephemeral znodes  in     81 ms (0.814011 ms/op
1228.484481/sec)
    watched     100           znodes  in     27 ms (0.279410 ms/op
3578.971440/sec)
    deleted     100 ephemeral znodes  in     99 ms (0.997062 ms/op
1002.946929/sec)
    notif       100           watches in   1001 ms (10.011010 ms/op
99.890019/sec)
    Latency test complete

So my question is what does this line mean at the end? Does it say, it can
do a get request call at a rate of 2557 requests per second?

     get         100           znodes  in     39 ms (0.390978 ms/op
2557.689587/sec)

Also I was interested to see how much load a 3 node ZK cluster can handle.
In my case, most of the call will be "get" (get the data from only one
znode) on the ZK cluster So with the above test I should be able to get
some idea right?



On Mon, Apr 7, 2014 at 11:40 PM, Michi Mutsuzaki <michi@cs.stanford.edu>wrote:

> Does this work?
>
> https://paste.apache.org/9Op5?action=download
>
> On Mon, Apr 7, 2014 at 9:10 PM, Check Peck <comptechgeeky@gmail.com>
> wrote:
> > I manually copied the jar file into the directory it was trying to copy.
> Now
> > if I do "ant", then I see like this -
> >
> > check@phx5tg03c-cfc7:~/zooksmoketest/zookeeper-3.4.6$ ant
> > Buildfile: /home/check/zooksmoketest/zookeeper-3.4.6/build.xml
> >
> > init:
> >
> > ivy-download:
> >
> > ivy-taskdef:
> >
> > ivy-init:
> >
> > ivy-retrieve:
> > [ivy:retrieve] :: Ivy 2.2.0 - 20100923230623 ::
> http://ant.apache.org/ivy/
> > ::
> > [ivy:retrieve] :: loading settings :: file =
> > /home/check/zooksmoketest/zookeeper-3.4.6/ivysettings.xml
> > [ivy:retrieve] :: resolving dependencies ::
> > org.apache.zookeeper#zookeeper;3.4.6
> > [ivy:retrieve]  confs: [default]
> >
> >
> > And then after this, it doesn't move further, I waited for  a long time
> and
> > nothing came up.. And afterwards I did ctrl + c to break.
> >
> > Any idea what might be wrong?
> >
> >
> >
> > On Mon, Apr 7, 2014 at 8:42 PM, Michi Mutsuzaki <michi@cs.stanford.edu>
> > wrote:
> >>
> >> You probably need to set the proxy.
> >>
> >> http://ant.apache.org/manual/proxy.html
> >>
> >> On Mon, Apr 7, 2014 at 8:40 PM, Check Peck <comptechgeeky@gmail.com>
> >> wrote:
> >> > Thanks a lot Michi. Appreciated  your help.. I tried your suggestion
> on
> >> > one
> >> > of my production box but when I tried "fifth" step which is "ant". I
> got
> >> > an
> >> > error as shown below. Any thoughts what is wrong? I guess firewall is
> >> > blocking. Is there any other way around for this? I manually copied
> >> > "zookeeper-3.4.6.tar.gz" into that machine using "scp" as well.
> >> > check@phx5tg03c-cfc7:~/zooksmoketest/zookeeper-3.4.6$ ant
> >> > Buildfile: /home/check/zooksmoketest/zookeeper-3.4.6/build.xml
> >> >
> >> > init:
> >> >
> >> > ivy-download:
> >> >       [get] Getting:
> >> > http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar
> >> >       [get] To:
> >> > /home/check/zooksmoketest/zookeeper-3.4.6/src/java/lib/ivy-2.2.0.jar
> >> >       [get] Error getting
> >> > http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jarto
> >> > /home/check/zooksmoketest/zookeeper-3.4.6/src/java/lib/ivy-2.2.0.jar
> >> >
> >> > BUILD FAILED
> >> > /home/check/zooksmoketest/zookeeper-3.4.6/build.xml:348:
> >> > java.net.ConnectException: Connection timed out
> >> >         at java.net.PlainSocketImpl.socketConnect(Native Method)
> >> >         at
> >> >
> >> >
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
> >> >         at
> >> >
> >> >
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
> >> >         at
> >> >
> >> >
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> >> >         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
> >> >         at java.net.Socket.connect(Socket.java:546)
> >> >         at java.net.Socket.connect(Socket.java:495)
> >> >         at sun.net.NetworkClient.doConnect(NetworkClient.java:178)
> >> >         at sun.net.www.http.HttpClient.openServer(HttpClient.java:409)
> >> >         at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
> >> >         at sun.net.www.http.HttpClient.<init>(HttpClient.java:240)
> >> >         at sun.net.www.http.HttpClient.New(HttpClient.java:321)
> >> >         at sun.net.www.http.HttpClient.New(HttpClient.java:338)
> >> >         at
> >> >
> >> >
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935)
> >> >         at
> >> >
> >> >
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876)
> >> >         at
> >> >
> >> >
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801)
> >> >         at
> >> >
> org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:660)
> >> >         at
> org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:579)
> >> >         at
> org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:569)
> >> >
> >> > Total time: 1 minute 3 seconds
> >> >
> >> >
> >> >
> >> > On Mon, Apr 7, 2014 at 3:08 PM, Michi Mutsuzaki <
> michi@cs.stanford.edu>
> >> > wrote:
> >> >>
> >> >> Hi Check,
> >> >>
> >> >> Something like this should work on ubuntu.
> >> >>
> >> >> https://paste.apache.org/6kao?action=download
> >> >>
> >> >>
> >> >> On Sun, Apr 6, 2014 at 11:11 PM, Check Peck <comptechgeeky@gmail.com
> >
> >> >> wrote:
> >> >> > I am not sure from where to start to install zkpython.. I tried
> >> >> > following
> >> >> > that github link but was not able to understand. If you can guide
> me
> >> >> > what I
> >> >> > am supposed to do step by step, then it will be of great help
to
> me.
> >> >> > Thanks.
> >> >> >
> >> >> >
> >> >> > On Sun, Apr 6, 2014 at 10:57 PM, Pavan Sudheendra
> >> >> > <pavan0591@gmail.com>wrote:
> >> >> >
> >> >> >> Hi Check,
> >> >> >>
> >> >> >> Where are you getting stuck?
> >> >> >>
> >> >> >>
> >> >> >> On Mon, Apr 7, 2014 at 11:13 AM, Check Peck
> >> >> >> <comptechgeeky@gmail.com>
> >> >> >> wrote:
> >> >> >>
> >> >> >> > Thanks. I tried going through that github link but wasn't
able
> to
> >> >> >> > understand what I am supposed to do clearly. May be I
am not
> that
> >> >> >> > much
> >> >> >> > familiar with Python so that's why I am having hard time
> figuring
> >> >> >> > that
> >> >> >> out?
> >> >> >> >
> >> >> >> >
> >> >> >> > On Sun, Apr 6, 2014 at 10:06 PM, Michi Mutsuzaki
> >> >> >> > <michi@cs.stanford.edu
> >> >> >> > >wrote:
> >> >> >> >
> >> >> >> > > You need to install zkpython. The steps are described
in the
> >> >> >> > > README
> >> >> >> > > file. Let us know if you have trouble installing
zkpython.
> >> >> >> > >
> >> >> >> > >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> https://github.com/apache/zookeeper/blob/trunk/src/contrib/zkpython/README
> >> >> >> > >
> >> >> >> > > On Sun, Apr 6, 2014 at 9:21 PM, Check Peck
> >> >> >> > > <comptechgeeky@gmail.com>
> >> >> >> > > wrote:
> >> >> >> > > > I tried following the steps mentioned on this
github link -
> >> >> >> > > > https://github.com/phunt/zk-smoketest for zk-latencies.py.
> >> >> >> > > >
> >> >> >> > > > I have 3 node ZK cluster-  machineA, machineB,
machineC
> >> >> >> > > >
> >> >> >> > > > Now I copied the Python program zk-latencies.py
and
> >> >> >> > > > zkclient.py
> >> >> >> > > > in
> >> >> >> > > another
> >> >> >> > > > machineD in which no zookeeper is running.
It's a plain
> ubuntu
> >> >> >> > > > box.
> >> >> >> > > >
> >> >> >> > > > Now when I ran the program like this -
> >> >> >> > > >
> >> >> >> > > > david@machineD:~/zooksmoketest$ ./zk-latencies.py
--servers
> >> >> >> > > > "machineA:2181,machineB::2181,machineC:2181"
> --znode_count=100
> >> >> >> > > > --znode_size=100 --synchronous
> >> >> >> > > > Traceback (most recent call last):
> >> >> >> > > >   File "./zk-latencies.py", line 22, in <module>
> >> >> >> > > >     import zkclient
> >> >> >> > > >   File "/home/david/zooksmoketest/zkclient.py",
line 17, in
> >> >> >> > > > <module>
> >> >> >> > > >     import zookeeper, time, threading
> >> >> >> > > >
> >> >> >> > > > *ImportError: No module named zookeeper*
> >> >> >> > > >
> >> >> >> > > > I got an error as shown above. Any thoughts
what wrong I am
> >> >> >> > > > doing
> >> >> >> > here? I
> >> >> >> > > > don't have that much experience with Python.
Any suggestions
> >> >> >> > > > will
> >> >> >> > > > be
> >> >> >> of
> >> >> >> > > > great help.
> >> >> >> > > >
> >> >> >> > > > @Raul: Any thoughts?
> >> >> >> > > >
> >> >> >> > > >
> >> >> >> > > >
> >> >> >> > > > On Sun, Apr 6, 2014 at 6:46 PM, Check Peck
> >> >> >> > > > <comptechgeeky@gmail.com>
> >> >> >> > > wrote:
> >> >> >> > > >
> >> >> >> > > >> Thanks guys.. I am going to do load test
in couple of days
> >> >> >> > > >> using
> >> >> >> > > >> the
> >> >> >> > > >> exiting Python framework. And I will share
the results
> once I
> >> >> >> > > >> am
> >> >> >> done.
> >> >> >> > > >>
> >> >> >> > > >>
> >> >> >> > > >> On Tue, Mar 25, 2014 at 11:19 PM, Rakesh
R
> >> >> >> > > >> <rakeshr@huawei.com>
> >> >> >> > wrote:
> >> >> >> > > >>
> >> >> >> > > >>>
> >> >> >> > > >>> If I understood the case correctly,
> >> >> >> > > >>> Test :- 3 node ZK cluster, 600 sessions
each will do one
> >> >> >> > > >>> read
> >> >> >> > > >>> operation(getChildren call) every one
minute.
> >> >> >> > > >>>
> >> >> >> > > >>> I'm assuming the calls are equally
distributed in time
> frame
> >> >> >> > > >>> and it
> >> >> >> > > would
> >> >> >> > > >>> be like : 600ops/60secs, average 10
ops/sec will go to 3
> >> >> >> > > >>> node
> >> >> >> cluster
> >> >> >> > > >>> I also feel it should be, also it would
be great if you
> >> >> >> > > >>> publish
> >> >> >> > > >>> the
> >> >> >> > > >>> results after your tests.
> >> >> >> > > >>>
> >> >> >> > > >>>
> >> >> >> > > >>> -Rakesh
> >> >> >> > > >>>
> >> >> >> > > >>> -----Original Message-----
> >> >> >> > > >>> From: Check Peck [mailto:comptechgeeky@gmail.com]
> >> >> >> > > >>> Sent: 26 March 2014 11:19
> >> >> >> > > >>> To: user
> >> >> >> > > >>> Subject: Re: Load on Zookeeper cluster?
> >> >> >> > > >>>
> >> >> >> > > >>> Thanks that is also quite useful.
> >> >> >> > > >>>
> >> >> >> > > >>> For my above use case, I just need
to get the children of
> a
> >> >> >> > particular
> >> >> >> > > >>> znode.. And that's what will be happening
from all the 600
> >> >> >> > application
> >> >> >> > > >>> machines..
> >> >> >> > > >>>
> >> >> >> > > >>> From the wiki you provided, it looks
it should be fine
> >> >> >> > > >>> right?
> >> >> >> > > >>>
> >> >> >> > > >>>
> >> >> >> > > >>> On Tue, Mar 25, 2014 at 10:13 PM, Rakesh
R
> >> >> >> > > >>> <rakeshr@huawei.com>
> >> >> >> > wrote:
> >> >> >> > > >>>
> >> >> >> > > >>> >
> >> >> >> > > >>> > Please go through the below one,
hope it will be helpful
> >> >> >> > > >>> > to
> >> >> >> > > >>> > you.
> >> >> >> > > Also,
> >> >> >> > > >>> > I would recommend to have a look
at the basic zk
> >> >> >> > > >>> > configurations
> >> >> >> > too.
> >> >> >> > > >>> > All the
> >> >> >> > > >>> > best:)
> >> >> >> > > >>> >
> >> >> >> > > >>> >
> >> >> >> > > >>> >
> >> >> >> > >
> >> >> >> > >
> >> >> >> > >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ServiceLatencyOv
> >> >> >> > > >>> > erview
> >> >> >> > > >>> >
> >> >> >> > > >>> >
> >> >> >> > > >>> > What client operations(for ex:
simply creating many
> >> >> >> > > >>> > znodes,
> >> >> >> > deleting,
> >> >> >> > > >>> > setwatches et.) you are planning
to perform.
> >> >> >> > > >>> > Actually I'd like to know what
each zkclient will be
> >> >> >> > > >>> > doing.
> >> >> >> > > >>> >
> >> >> >> > > >>> > -Rakesh
> >> >> >> > > >>> >
> >> >> >> > > >>> > -----Original Message-----
> >> >> >> > > >>> > From: Check Peck [mailto:comptechgeeky@gmail.com]
> >> >> >> > > >>> > Sent: 26 March 2014 09:40
> >> >> >> > > >>> > To: user
> >> >> >> > > >>> > Subject: Re: Load on Zookeeper
cluster?
> >> >> >> > > >>> >
> >> >> >> > > >>> > On Tue, Mar 25, 2014 at 9:00 PM,
Raúl Gutiérrez Segalés
> <
> >> >> >> > > >>> > rgs@itevenworks.net
> >> >> >> > > >>> > > wrote:
> >> >> >> > > >>> >
> >> >> >> > > >>> > > 1 ZK se
> >> >> >> > > >>> >
> >> >> >> > > >>> >
> >> >> >> > > >>> >
> >> >> >> > > >>> > Yes one ZK session per application.
Zookeeper version I
> am
> >> >> >> running
> >> >> >> > > >>> > currently is 3.4.5 in production.
> >> >> >> > > >>> >
> >> >> >> > > >>>
> >> >> >> > > >>
> >> >> >> > > >>
> >> >> >> > >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Regards-
> >> >> >> Pavan
> >> >> >>
> >> >
> >> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message