zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikhil Bafna <zod...@gmail.com>
Subject Re: Zookeeper general log or “monitor” command
Date Fri, 09 Mar 2018 07:34:35 GMT
> If I understand correctly, a slow operation log with the threshold for
> logging set to 0 should accomplish what you are trying to accomplish.

Yes, that would work.

You may also want to consider taking a look and reviewing the change
> proposed here: https://github.com/apache/zookeeper/pull/307

Checked the PR. IMO, the actual proposed change there doesn't solve the
intent of the mentioned problem (though it solves mine) and have left a
comment there with my viewpoint.


-
zodvik

On Fri, Mar 9, 2018 at 12:36 AM, Abraham Fine <afine@apache.org> wrote:

> You may also want to consider taking a look and reviewing the change
> proposed here: https://github.com/apache/zookeeper/pull/307
>
> If I understand correctly, a slow operation log with the threshold for
> logging set to 0 should accomplish what you are trying to accomplish.
>
> Thanks,
> Abe
>
> On Wed, Mar 7, 2018, at 04:30, Andor Molnar wrote:
> > Cool, monitoring the transaction makes much sense I think.
> >
> > Interesting to see this tool, I didn't know about it yet. For some reason
> > it replicates most of the logic in FileSnap and FileTxnLog classes, but
> at
> > the same time it uses ZooKeeper as a dependency.
> >
> > Regards,
> > Andor
> >
> >
> >
> > On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna <zodvik@gmail.com> wrote:
> >
> > > As you mentioned, enabling trace/debug might impact performance.
> > >
> > >
> > > Here's the approach I'm using now.
> > >
> > >    -
> > >
> > >    Parse the Zookeeper transaction logs. The best repo I've found to do
> > >    this is https://github.com/alenca/zklogtool. It's feature packed.
> The
> > >    only downside is that it won't record the read queries.
> > >
> > >
> > > -
> > > zodvik
> > >
> > > On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar <andor@cloudera.com>
> wrote:
> > >
> > > > Hi Zod,
> > > >
> > > > Have you checked TRACE logging of client requests? (
> > > > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> > > html#sc_authOptions
> > > > )
> > > > Not sure how much different is from what you refer as 'debug logs',
> but
> > > it
> > > > might worth a try. It generates log messages like:
> > > >
> > > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > > :Esessionid:0x8061f60cfd200000 type:getChildren2 cxid:0x6
> > > > zxid:0xfffffffffffffffe txntype:unknown reqpath:/
> > > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > > :Esessionid:0x8061f60cfd200000 type:getData cxid:0x7
> > > > zxid:0xfffffffffffffffe txntype:unknown reqpath:/andor2
> > > >
> > > > Which looks like quite easy to parse and contains all required
> > > information.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > > On Thu, Mar 1, 2018 at 5:33 AM, Zod <zodvik@gmail.com> wrote:
> > > >
> > > > > Unfortunately, parsing debug log is no-go since it doesn't print
> the
> > > > > complete command.
> > > > >
> > > > > I've 2 approached in mind as of now
> > > > >
> > > > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > > > "tailing".
> > > > > 2. Use ngrep on my local to monitor the network traffic to remote
> ZK
> > > > host &
> > > > > port 2181
> > > > >
> > > > > ​Will get back with what works better.​
> > > > >
> > > > > -
> > > > > zodvik
> > > > >
> > > > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar <andor@cloudera.com>
> > > wrote:
> > > > >
> > > > > > Hi Zod,
> > > > > >
> > > > > > I think there's no such thing in ZooKeeper currently other than
> debug
> > > > > logs.
> > > > > > You might want to attach some processing engine to the debug
log
> feed
> > > > and
> > > > > > parse and output data which is important for you into MySQL
or
> Redis
> > > > > > database.
> > > > > >
> > > > > > The downside of this approach is debug logging has significant
> > > > > performance
> > > > > > impact on the entire ensemble, but it might be acceptable for
you
> > > > > analysis.
> > > > > >
> > > > > > Regards,
> > > > > > Andor
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod <zodvik@gmail.com>
wrote:
> > > > > >
> > > > > > > I want see all commands issued to my Zookeeper cluster,
> something
> > > > like
> > > > > > > general log in MySQL or "monitor" command in Redis. How
can
> this be
> > > > > done?
> > > > > > >
> > > > > > > The intent is to see how Storm uses Zookeeper (for state
> > > management &
> > > > > > > ack-ing) and a good way to would be to run a sample topology
> and
> > > see
> > > > > all
> > > > > > > the commands issued to Zookeeper by Storm.
> > > > > > >
> > > > > > > I tried enabling debug log for Zookeeper, but that is
> insufficient
> > > > has
> > > > > a
> > > > > > > lot of noise. For example, issue a create /node prints
> > > > > > >
> > > > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > > > -
> > > > > > > zodvik
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message