hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Drob (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18396) Encode ZNode names to reduce ZooKeeper jute buffer length requirements and thus reduce memory usage
Date Tue, 08 Aug 2017 20:52:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119010#comment-16119010

Mike Drob commented on HBASE-18396:

I don't know the exact sizes of paths, data, how many requests come in a batch, etc, so this
is all going to be hypothetical.

Say we have a typical ZK path of 100 bytes (and no data) and we're able to reduce that to
40 bytes. I don't expect to be able to go much further since I think there still need to be
IDs or other non-colliding bits that exist, but say we map "replication" to "r" and so on.
Theoretically, what used to take 1M will now only take 400K! That's a huge win!

But! You go on to say that we can use this win to fit more requests in a single buffer. Well,
then you'll have the same jute.maxbuffer len error, since it doesn't matter how big you make
your buffer or how small you make the items if you keep adding until it overflows. I think
saving network is a fine goal, but at the end of the day it's much more valuable to address
the cause of the buffer exceeding instead of trying to optimize away the problems.

Also, most systems I interact with have increased the maxbuffer to 4M and are running without

> Encode ZNode names to reduce ZooKeeper jute buffer length requirements and thus reduce
memory usage
> ---------------------------------------------------------------------------------------------------
>                 Key: HBASE-18396
>                 URL: https://issues.apache.org/jira/browse/HBASE-18396
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Karan Mehta
> In our production environment, we hit the error {{ZooKeeper connectionLoss due to jute.maxbuffer
len of 1M getting exceeded}}. Usually 1 MB is a lot, but in case of multi requests, it can
exceed the maximum buffer length that is allocated.
> This JIRA is a discussion for encoding various znode names. IMO, this will reduce the
path lengths, thus reducing the size of buffer required as well as network packet size and
also pack more requests in a single multi. As with encoding, this will introduce overhead,
but we need to determine how feasible this idea is.

This message was sent by Atlassian JIRA

View raw message