Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Wed, 2 Jul 2014 15:10:25 +0000 (UTC)
From: "Eric Newton (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12711270.1398796289268.88193.1404313825981@arcas>
In-Reply-To: <JIRA.12711270.1398796289268@arcas>
References: <JIRA.12711270.1398796289268@arcas>
Subject: [jira] [Commented] (ACCUMULO-2757) millions of zookeeper watches
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/ACCUMULO-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14050012#comment-14050012 ] 

Eric Newton commented on ACCUMULO-2757:
---------------------------------------

Over in ZOOKEEPER-1177, it is observed:

bq. It seems a single watch takes about 100 bytes

On a very large production cluster running 1.4, a 9 zookeeper quorum has 10-20M watches.  This corresponds to 1-2.2M watches per node.

1.4 does not have table namespaces, but it has logger entries, which are watched.

This cluster is large enough to run zookeepers on their own separate nodes.  Presently they each have 48G JVMs.  This could support at least ~40G for watches, which is 400M watches per node, or two orders of magnitude more than the observed number of watches.

If accumulo wanted to support 1K tables per instance, that would increase the number of watches routinely set by clients and servers, and that would begin to approach the zookeeper limits.


> millions of zookeeper watches
> -----------------------------
>
>                 Key: ACCUMULO-2757
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2757
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: client, master, tserver
>         Environment: very large clusters
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>
> I recently learned how to get the number of watches in a zookeeper node.
> {noformat}
> $ echo wchp | nc localhost 2181
> {noformat}
> There are a surprising number of watches supporting table configuration.  A single tserver accumulo instance with one data table is using 872 watches.  Large clusters, even with many zookeepers, can have millions of watches.
> We may need to watch a single notification node, or serialize the table configs into a single node per table.  That would accomplish ACCUMULO-1568, too.


--
This message was sent by Atlassian JIRA
(v6.2#6252)