zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andor Molnar <an...@cloudera.com>
Subject Re: Zookeeper general log or “monitor” command
Date Wed, 07 Mar 2018 12:30:14 GMT
Cool, monitoring the transaction makes much sense I think.

Interesting to see this tool, I didn't know about it yet. For some reason
it replicates most of the logic in FileSnap and FileTxnLog classes, but at
the same time it uses ZooKeeper as a dependency.

Regards,
Andor



On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna <zodvik@gmail.com> wrote:

> As you mentioned, enabling trace/debug might impact performance.
>
>
> Here's the approach I'm using now.
>
>    -
>
>    Parse the Zookeeper transaction logs. The best repo I've found to do
>    this is https://github.com/alenca/zklogtool. It's feature packed. The
>    only downside is that it won't record the read queries.
>
>
> -
> zodvik
>
> On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar <andor@cloudera.com> wrote:
>
> > Hi Zod,
> >
> > Have you checked TRACE logging of client requests? (
> > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> html#sc_authOptions
> > )
> > Not sure how much different is from what you refer as 'debug logs', but
> it
> > might worth a try. It generates log messages like:
> >
> > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > :Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
> > zxid:0xfffffffffffffffe txntype:unknown reqpath:/
> > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > :Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
> > zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
> >
> > Which looks like quite easy to parse and contains all required
> information.
> >
> > Regards,
> > Andor
> >
> >
> > On Thu, Mar 1, 2018 at 5:33 AM, Zod <zodvik@gmail.com> wrote:
> >
> > > Unfortunately, parsing debug log is no-go since it doesn't print the
> > > complete command.
> > >
> > > I've 2 approached in mind as of now
> > >
> > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > "tailing".
> > > 2. Use ngrep on my local to monitor the network traffic to remote ZK
> > host &
> > > port 2181
> > >
> > > ​Will get back with what works better.​
> > >
> > > -
> > > zodvik
> > >
> > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <andor@cloudera.com>
> wrote:
> > >
> > > > Hi Zod,
> > > >
> > > > I think there's no such thing in ZooKeeper currently other than debug
> > > logs.
> > > > You might want to attach some processing engine to the debug log feed
> > and
> > > > parse and output data which is important for you into MySQL or Redis
> > > > database.
> > > >
> > > > The downside of this approach is debug logging has significant
> > > performance
> > > > impact on the entire ensemble, but it might be acceptable for you
> > > analysis.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > >
> > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zodvik@gmail.com> wrote:
> > > >
> > > > > I want see all commands issued to my Zookeeper cluster, something
> > like
> > > > > general log in MySQL or "monitor" command in Redis. How can this
be
> > > done?
> > > > >
> > > > > The intent is to see how Storm uses Zookeeper (for state
> management &
> > > > > ack-ing) and a good way to would be to run a sample topology and
> see
> > > all
> > > > > the commands issued to Zookeeper by Storm.
> > > > >
> > > > > I tried enabling debug log for Zookeeper, but that is insufficient
> > has
> > > a
> > > > > lot of noise. For example, issue a create /node prints
> > > > >
> > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > -
> > > > > zodvik
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message