kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Transactional Consistency beyond KIP-98 (transactional consumer)
Date Wed, 05 Jul 2017 23:20:01 GMT
Hello Werner,

1) Regarding the long transaction, we did discuss about this in the design
doc (
https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#bookmark=id.z9b0a6k2a3zm),
and as stated the main reason we chose to go with the current design is to
preserve offset ordering; and if it is becoming a critical feature request,
we may consider allow users to read either in offset ordering or
transaction ordering, in latter case a single long transaction will not
stall others.

2) Regarding the transactional consumption behavior, my understanding is
you want to have `consumer.poll()` return all records committed in a single
transaction right? My take is that this is again depending on the
transaction ordering mentioned above, since a single topic partition may
have multiple producers sending to it, resulting in multiple transactions
interleaving with each other on that log. In order to return a single
completed transaction's all messages you are likely to get
"non-consecutive" messages. But again if we decide transaction ordering
based consumption is common and necessary we may likely just to this while
implementing the feature.


Guozhang



On Tue, Jul 4, 2017 at 4:29 AM, Daehn, Werner <werner.daehn@sap.com> wrote:

> While the new transactional consistency feature is definitely a move into
> the right direction, I find the current design limitations quite severe.
> Understandable but severe.
>
> First is the offset being the add-time, not the commit time. The
> consequence of that is, if one transactional producer adds rows every once
> a while and does commit minutes or hours later, all consumers with
> read=committed will not see any other producer's data until this one did
> the commit. Will decrease the stability of the overall solution, won't it?
> Would have been better to use the commit time as the offset value. Harder
> to implement but more stable.
>
> The other issue is at the consumer side. Current implementation does not
> provide transaction guarantees on the consumers. Again, for logical reasons
> and to simplify the implementation, but as a user you want to have both -
> when data is committed together you want to read the entire transaction as
> one unit and complete.
>
> Are there any plans to go beyond the current implementation?
>
> Thanks in advance
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message