kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From S Ahmed <sahmed1...@gmail.com>
Subject Re: anecdotal uptime and service monitoring
Date Wed, 30 Jan 2013 01:38:38 GMT
Jun,

Great list.   I'm haven't really setup monitoring before, so for starters,
what should I be researching in order to monitor those metrics, are they
exposed via those yammer metrics library that can be exported to a csv
file, or are these jmx related items?




On Fri, Dec 28, 2012 at 5:47 PM, Jun Rao <junrao@gmail.com> wrote:

> At LinkedIn, the most common failure of a Kafka broker is when we have to
> deploy new Kafka code/config. Otherwise, the broker can be up for a long
> time (e..g, months). It woud be good to monitor the following metrics at
> the broker: log flush time/rate, produce/fetch requests/messages rate, GC
> rate/time, network bandwidth utilization,  and disk space and I/O
> utilization. For the clients, it would be good to monitor message
> size/rate, request time/rate, dropped event rate (for async producers) and
> consumption lag (for consumers). For ZK, ideally, one should monitor ZK
> request latency and GCs.
>
> Thanks,
>
> Jun
>
> On Fri, Dec 28, 2012 at 7:27 AM, S Ahmed <sahmed1020@gmail.com> wrote:
>
> > Curious what kind of uptime have you guys experienced using kafka?
> >
> > What sort of monitoring do you suggest should be in place for kafka?
> >
> > If the service crashes, does it usually make sense to have something like
> > upstart restart the service?
> >
> > There are allot of moving parts (hard drive space, zooker, producers,
> > consumers, etc.)
> >
> > Also if the consumers can't keep up with new messages...
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message