accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-2757) millions of zookeeper watches
Date Wed, 02 Jul 2014 15:10:25 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14050012#comment-14050012
] 

Eric Newton commented on ACCUMULO-2757:
---------------------------------------

Over in ZOOKEEPER-1177, it is observed:

bq. It seems a single watch takes about 100 bytes

On a very large production cluster running 1.4, a 9 zookeeper quorum has 10-20M watches. 
This corresponds to 1-2.2M watches per node.

1.4 does not have table namespaces, but it has logger entries, which are watched.

This cluster is large enough to run zookeepers on their own separate nodes.  Presently they
each have 48G JVMs.  This could support at least ~40G for watches, which is 400M watches per
node, or two orders of magnitude more than the observed number of watches.

If accumulo wanted to support 1K tables per instance, that would increase the number of watches
routinely set by clients and servers, and that would begin to approach the zookeeper limits.


> millions of zookeeper watches
> -----------------------------
>
>                 Key: ACCUMULO-2757
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2757
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: client, master, tserver
>         Environment: very large clusters
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>
> I recently learned how to get the number of watches in a zookeeper node.
> {noformat}
> $ echo wchp | nc localhost 2181
> {noformat}
> There are a surprising number of watches supporting table configuration.  A single tserver
accumulo instance with one data table is using 872 watches.  Large clusters, even with many
zookeepers, can have millions of watches.
> We may need to watch a single notification node, or serialize the table configs into
a single node per table.  That would accomplish ACCUMULO-1568, too.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message