Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 8 Aug 2017 20:52:00 +0000 (UTC)
From: "Mike Drob (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13087712.1500320754000.121362.1502225520434@Atlassian.JIRA>
In-Reply-To: <JIRA.13087712.1500320754000@Atlassian.JIRA>
References: <JIRA.13087712.1500320754000@Atlassian.JIRA> <JIRA.13087712.1500320754737@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-18396) Encode ZNode names to reduce
 ZooKeeper jute buffer length requirements and thus reduce memory usage
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 08 Aug 2017 20:52:05 -0000


    [ https://issues.apache.org/jira/browse/HBASE-18396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119010#comment-16119010 ] 

Mike Drob commented on HBASE-18396:
-----------------------------------

I don't know the exact sizes of paths, data, how many requests come in a batch, etc, so this is all going to be hypothetical.

Say we have a typical ZK path of 100 bytes (and no data) and we're able to reduce that to 40 bytes. I don't expect to be able to go much further since I think there still need to be IDs or other non-colliding bits that exist, but say we map "replication" to "r" and so on. Theoretically, what used to take 1M will now only take 400K! That's a huge win!

But! You go on to say that we can use this win to fit more requests in a single buffer. Well, then you'll have the same jute.maxbuffer len error, since it doesn't matter how big you make your buffer or how small you make the items if you keep adding until it overflows. I think saving network is a fine goal, but at the end of the day it's much more valuable to address the cause of the buffer exceeding instead of trying to optimize away the problems.

Also, most systems I interact with have increased the maxbuffer to 4M and are running without issue.

> Encode ZNode names to reduce ZooKeeper jute buffer length requirements and thus reduce memory usage
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-18396
>                 URL: https://issues.apache.org/jira/browse/HBASE-18396
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Karan Mehta
>
> In our production environment, we hit the error {{ZooKeeper connectionLoss due to jute.maxbuffer len of 1M getting exceeded}}. Usually 1 MB is a lot, but in case of multi requests, it can exceed the maximum buffer length that is allocated.
> This JIRA is a discussion for encoding various znode names. IMO, this will reduce the path lengths, thus reducing the size of buffer required as well as network packet size and also pack more requests in a single multi. As with encoding, this will introduce overhead, but we need to determine how feasible this idea is.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)