samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jordan Shaw <jor...@pubnub.com>
Subject Re: Monitoring consumer lag
Date Mon, 16 Nov 2015 18:04:47 GMT
Michael,
It depends on how you define lag.

1) If you define lag as the total number of messages behind then burrow is
a good tool as long as all your infrastructure is on 0.8.2, it basically
works by inspecting the __consumer_offsets topic which was introduced in
0.8.2 (they said they were going to support <0.8.2 but i don't think that's
a thing yet).

2. If you define lag in time units I would recommend sending a timestamp
with the msg and doing some "manual" inspecting on the consumer end.
-Jordan

On Mon, Nov 16, 2015 at 9:16 AM, Michael Ravits <michaelr524@gmail.com>
wrote:

> Hi,
>
> I'd like to monitor consumer's lag.
> Found this tool https://github.com/linkedin/Burrow.
> But now realized that Samza is using it's own checkpointing mechanism.
>
> So question is what's the best way to monitor whether and how much the
> consumer is lagging?
>
> On a related subject, I'd also like to monitor throughput per topic in
> terms of messages per second and bytes per second. Should I query brokers
> periodically, or maybe there is a better way?
>
> Thanks,
> Michael
>



-- 
Jordan Shaw
Full Stack Software Engineer
PubNub Inc
1045 17th St
San Francisco, CA 94107

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message