hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18396) Encode ZNode names to reduce ZooKeeper jute buffer length requirements and thus reduce memory usage
Date Fri, 27 Oct 2017 21:21:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222848#comment-16222848
] 

Andrew Purtell commented on HBASE-18396:
----------------------------------------

FWIW we have a replication backup and related issues currently producing actually observed
jute messages of ~26MB.  Encoding can help relieve pressure until we fix the underlying cause.
HBase shouldn't manage replication state in ZooKeeper, full stop. We have other JIRAs in place
proposing a migration to HBase hosted system tables. While doing that we can work on keeping
queue state tracking scalable as number of queues and WALs increase. I think that's the long
term direction. In the interim maybe we can consider half measures like this as pain relief.


> Encode ZNode names to reduce ZooKeeper jute buffer length requirements and thus reduce
memory usage
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-18396
>                 URL: https://issues.apache.org/jira/browse/HBASE-18396
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Karan Mehta
>
> In our production environment, we hit the error {{ZooKeeper connectionLoss due to jute.maxbuffer
len of 1M getting exceeded}}. Usually 1 MB is a lot, but in case of multi requests, it can
exceed the maximum buffer length that is allocated.
> This JIRA is a discussion for encoding various znode names. IMO, this will reduce the
path lengths, thus reducing the size of buffer required as well as network packet size and
also pack more requests in a single multi. As with encoding, this will introduce overhead,
but we need to determine how feasible this idea is.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message