Mailing-List: contact dev-help@samza.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@samza.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CA+T4S-iV1Sbtvp4VSOj0xZdxvdnihke28r7RdHKZTU3XkJegVQ@mail.gmail.com>
References: 
 <CA+T4S-iV1Sbtvp4VSOj0xZdxvdnihke28r7RdHKZTU3XkJegVQ@mail.gmail.com>
Date: Mon, 16 Nov 2015 10:02:26 -0800
Message-ID: 
 <CAMd3yjh59Zo=Kz11+n1pRKp2O2WZoZ5tpDmhBfRMjmOqqsK6fw@mail.gmail.com>
Subject: Re: Monitoring consumer lag
From: Jagadish Venkatraman <jagadish1989@gmail.com>
To: dev@samza.apache.org
Content-Type: multipart/alternative; boundary=94eb2c06d10040a7ea0524ac36c7

--94eb2c06d10040a7ea0524ac36c7
Content-Type: text/plain; charset=UTF-8

The following metrics are reported via JMX.

1. Samza exposes a per-partition metric called "$topicname-$partition_name-
messages-behind-high-watermark". This measures the number of messages
behind the high watermark of your consumer. Ideally, at steady state, you
would expect this metric to be zero (with occasional spikes depending on
your traffic).

2. For your second usecase,  "$topicname-$partition_name-bytes-read",
""$topicname-$partition_name-messages-read" are interesting.

You can query these metrics and get alerted based on them.


On Mon, Nov 16, 2015 at 9:16 AM, Michael Ravits <michaelr524@gmail.com>
wrote:

> Hi,
>
> I'd like to monitor consumer's lag.
> Found this tool https://github.com/linkedin/Burrow.
> But now realized that Samza is using it's own checkpointing mechanism.
>
> So question is what's the best way to monitor whether and how much the
> consumer is lagging?
>
> On a related subject, I'd also like to monitor throughput per topic in
> terms of messages per second and bytes per second. Should I query brokers
> periodically, or maybe there is a better way?
>
> Thanks,
> Michael
>


-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

--94eb2c06d10040a7ea0524ac36c7--