kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin McCabe" <cmcc...@apache.org>
Subject Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted
Date Mon, 22 Jul 2019 15:51:14 GMT
Hi Jose,

One issue that I see here is that the number of log messages could be huge.  I've seen people
create tens of thousands of consumer groups.  People can also have settings that create pretty
small log files.  A message per log file per group could be quite a lot of messages.

A log message on the broker is also not that useful for detecting bad client behavior.  People
generally only look at the server logs after they become aware that something is wrong through
some other means.

Perhaps the clients should just monitor their lag?  There is a JMX metric for this, which
means it can be hooked into traditional metrics / reporting systems.

best,
Colin


On Mon, Jul 22, 2019, at 03:12, Jose M wrote:
> Hello,
> 
> I didn't get any feedback on this small KIP-490
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-490%3A+log+when+consumer+groups+lose+a+message+because+offset+has+been+deleted>.
> In summary, I propose a way to be noticed when messages are being 
> removed
> due to retention policy, without being consumed by a given consumer 
> group.
> It will be useful to realize that some important messages have been 
> lost.
> 
> As Im new to the codebase, I have technical questions about how to achieve
> this, but before going deeper, I would like your feedback on the feature.
> 
> Thanks a lot,
> 
> 
> Jose Morales
> 
> On Sun, Jul 14, 2019 at 12:51 AM Jose M <yozies@gmail.com> wrote:
> 
> > Hello,
> >
> > I would like to know what do you think on KIP-490:
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-490%3A+log+when+consumer+groups+lose+a+message+because+offset+has+been+deleted
> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-490%3A+log+when+consumer+groups+lose+a+message+because+offset+has+expired>
> >
> >
> > Thanks a lot !
> > --
> > Jose M
> >
> 
> 
> -- 
> J
>

Mime
View raw message