zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rammohan ganapavarapu <rammohanga...@gmail.com>
Subject Re: How to find the actual size of zk data
Date Tue, 20 Mar 2018 15:41:37 GMT
Mark,

Thanks, yes it would be nice to have as part of JMX. Please let me know if
its already there or any tool that does.

Ram

On Tue, Mar 20, 2018 at 2:27 AM, Mark Fenes <mfenes@cloudera.com> wrote:

> Ok, understood. Maybe the best way of calculating the ZK data size would
> be to count/estimate the number of znodes and their average data size per
> node. This would eliminate file size differences by file format.
> Maybe we could write a simple utility to read a snapshot file and
> calculate these statistics. Also transactions after the latest snapshot
> file can cause file size differences. Or perhaps the ZK server itself could
> publish online statistics about node count and average node size via JMX -
> I'll check if this is already implemented or not.
>
> Mark
>
> On Mon, Mar 19, 2018 at 4:05 PM, rammohan ganapavarapu <
> rammohanganap@gmail.com> wrote:
>
>> zkTreeutil has option to dump every thing into a directory and create
>> same tree in filesystem under that directory, i am talking about that size.
>>
>> Ram
>>
>> On Mon, Mar 19, 2018 at 5:42 AM, Mark Fenes <mfenes@cloudera.com> wrote:
>>
>>>
>>> Hi Ram,
>>>
>>> If the format of the snapshot file and the zktreeutil dump file is
>>> different, they will have different sizes. Does zktreeutil export the
>>> zk data in XML format?
>>>
>>> Mark
>>>
>>>
>>> On Fri, Mar 16, 2018 at 2:58 AM, rammohan ganapavarapu <
>>> rammohanganap@gmail.com> wrote:
>>>
>>>> Jordan,
>>>>
>>>> Thank you, even though you have multiple snapshot files in dataDir
>>>> zookeeper uses only one latest file right? If my latest snapshot file is
>>>> 1GB means my total zk data side is 1GB ?  But when I use zktreeutil and
>>>> dump to filesystem i am getting lot more GB space then snapshot size.
>>>>
>>>> Ram
>>>>
>>>> On Thu, Mar 15, 2018, 4:02 PM Jordan Zimmerman <
>>>> jordan@jordanzimmerman.com>
>>>> wrote:
>>>>
>>>> > > How to find  actual size of zookeeper data since it store and
>>>> operate
>>>> > from
>>>> > > in-memory? and snapshot may not give me actual size?
>>>> >
>>>> > ZooKeeper transactions/snapshots are stored on disk in a series of
>>>> files.
>>>> > You can use your file system to get the combined sizes of these
>>>> files. The
>>>> > directory is specified by the "dataDir" value in zoo.cfg.
>>>> >
>>>> > > One more question, since its a in-memory datastore, what if my
zk
>>>> data
>>>> > > grows beyond available RAM? does it swap?
>>>> >
>>>> > No. ZooKeeper does not swap. You will get OutOfMemory exceptions if
>>>> store
>>>> > too much data. The ZooKeeper database is always limited by memory.
>>>> >
>>>> > -JZ
>>>> >
>>>> > > On Mar 15, 2018, at 5:57 PM, rammohan ganapavarapu <
>>>> > rammohanganap@gmail.com> wrote:
>>>> > >
>>>> > > Hi,
>>>> > >
>>>> > > How to find  actual size of zookeeper data since it store and
>>>> operate
>>>> > from
>>>> > > in-memory? and snapshot may not give me actual size? I used
>>>> zkteeUtils to
>>>> > > dump the data is it the actual size ?
>>>> > >
>>>> > > One more question, since its a in-memory datastore, what if my
zk
>>>> data
>>>> > > grows beyond available RAM? does it swap?
>>>> > >
>>>> > > Thanks,
>>>> > > Ram
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message