kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From D C <dragos.g.cri...@gmail.com>
Subject Re: Highwater mark interpretation
Date Sat, 20 Jun 2020 21:55:15 GMT
The short answer is : yes, a consumer can only consume messages up to the
High Watermark.

The long answer is not exactly, for the following reasons:

At the partition level you have 3 major offsets that are important to the
health of the partition and accessibility from the consumer pov:
LeO (log end offset) - which represents the highest offset in the highest
segment
High Watermark - which represents the latest offset that has been
replicated to all the followers
LSO (Last stable offset) - which is important when you use producers that
create transactions - which represents the the highest offset that has been
committed by a transaction and that is allowed to be read with isolation
level = read_commited.

The LeO can only be higher or equal to the High Watermark (for obvious
reasons)
The High Watermark can only be higher or equal to the LSO (the messages up
to this point may have been committed to all the followers but the
transaction isn't yet finished)
And coming to your question, in case the transaction hasn't finished, the
LSO may be lower than the High Watermark so if your consumer is accessing
the data in Read_Committed, it won't be able to surpass the LSO.

Cheers,
D

On Sat, Jun 20, 2020 at 9:05 PM Nag Y <andriod.nag.user@gmail.com> wrote:

> As I understand it, the consumer can only read "committed" messages - which
> I believe, if we look at internals of it, committed messages are nothing
> but messages which are upto the high watermark.
> *The high watermark is the offset of the last message that was successfully
> copied to all of the log’s replicas. *
>
> *Having said that, if one of the replica is down, will high water mark be*
> *advanced?*
>
> *If replica can't come forever, can we consider this message cant be
> consumed by the consumer since it is never committed *
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message