zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From N Keywal <nkey...@gmail.com>
Subject Re: sync vs. async vs. multi performances
Date Tue, 14 Feb 2012 16:16:22 GMT
Yes. But it's dedicated to ZK (the os and tmp are on another one).

On Tue, Feb 14, 2012 at 5:11 PM, Flavio Junqueira <fpj@yahoo-inc.com> wrote:

> From your description, it sounds like you're using one disk device for the
> ZooKeeper server. Is it correct?
>
> -Flavio
>
> On Feb 14, 2012, at 5:00 PM, N Keywal wrote:
>
> > Hi,
> >
> > I've done a test with Zookeeper 3.4.2 to compare the performances of
> > synchronous vs. asynchronous vs. multi when creating znode (variations
> > around:
> > calling 10000 times zk.create("/dummyTest", "dummy".getBytes(),
> > ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);) The code is at the
> > end of the mail.
> >
> > I've tested different environments:
> > - 1 linux server with the client and 1 zookeeper node on the same machine
> > - 1 linux server for the client, 1 for 1 zookeeper node.
> > - 6 linux servers, 1 for the client, 5 for 5 zookeeper nodes.
> >
> > Server are middle range, with 4*2 cores, jdk 1.6. ZK was on its own HD.
> >
> > But the results are comparable:
> >
> > Using the sync API, it takes 200 seconds for 10K creations, so around
> 0.02
> > second per call.
> > Using the async API, it takes 2 seconds for 10K (including waiting for
> the
> > last callback message)
> > Using the "multi" available since 3.4, it takes less than 1 second, again
> > for 10K.
> >
> > I'm surprised by the time taken by the sync operation, I was not
> expecting
> > it to be that slow. The gap between async & sync is quite huge.
> >
> > Is this something expected? Zookeeper is used in critical functions in
> > Hadoop/Hbase, I was looking at the possible benefits of using "multi",
> but
> > it seems low compared to async (well ~3 times faster :-). There are many
> > small data creations/deletions with the sync API in the existing hbase
> > algorithms, it would not be simple to replace them all by asynchronous
> > calls...
> >
> > Cheers,
> >
> > N.
> >
> > --
> >
> > public class ZookeeperTest {
> >  static ZooKeeper zk;
> >  static int nbTests = 10000;
> >
> >  private ZookeeperTest() {
> >  }
> >
> >  public static void test11() throws Exception {
> >    for (int i = 0; i < nbTests; ++i) {
> >      zk.create("/dummyTest_" + i, "dummy".getBytes(),
> > ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
> >    }
> >  }
> >
> >
> >  public static void test51() throws Exception {
> >    final AtomicInteger counter = new AtomicInteger(0);
> >
> >    for (int i = 0; i < nbTests; ++i) {
> >      zk.create("/dummyTest_" + i, "dummy".getBytes(),
> > ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT,
> >        new AsyncCallback.StringCallback() {
> >          public void processResult(int i, String s, Object o, String s1)
> {
> >            counter.incrementAndGet();
> >          }
> >        }
> >        , null);
> >    }
> >
> >    while (counter.get() != nbTests) {
> >      Thread.sleep(1);
> >    }
> >  }
> >
> >  public static void test41() throws Exception {
> >    ArrayList<Op> ops = new ArrayList<Op>(nbTests);
> >    for (int i = 0; i < nbTests; ++i) {
> >      ops.add(
> >        Op.create("/dummyTest_" + i, "dummy".getBytes(),
> > ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
> >      );
> >    }
> >
> >    zk.multi(ops);
> >  }
> >
> >  public static void delete() throws Exception {
> >    ArrayList<Op> ops = new ArrayList<Op>(nbTests);
> >
> >    for (int i = 0; i < nbTests; ++i) {
> >      ops.add(
> >        Op.delete("/dummyTest_" + i,-1)
> >      );
> >    }
> >
> >    zk.multi(ops);
> >  }
> >
> >
> >  public static void test(String connection, String testName) throws
> > Throwable{
> >    Method m = ZookeeperTest.class.getMethod(testName);
> >
> >    zk = new ZooKeeper(connection, 20000, new Watcher() {
> >      public void process(WatchedEvent watchedEvent) {
> >      }
> >    });
> >
> >    final long start = System.currentTimeMillis();
> >
> >    try {
> >      m.invoke(null);
> >    } catch (IllegalAccessException e) {
> >      throw e;
> >    } catch (InvocationTargetException e) {
> >      throw e.getTargetException();
> >    }
> >
> >    final long end = System.currentTimeMillis();
> >
> >    zk.close();
> >
> >    final long endClose = System.currentTimeMillis();
> >
> >    System.out.println(testName+":  ExeTime= " + (end - start) );
> >  }
> >
> >  public static void main(String... args) throws Throwable {
> >      test(args[0], args[1]);
> >  }
> > }
>
> flavio
> junqueira
>
> research scientist
>
> fpj@yahoo-inc.com
> direct +34 93-183-8828
>
> avinguda diagonal 177, 8th floor, barcelona, 08018, es
> phone (408) 349 3300    fax (408) 349 3301
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message