Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
From: "Daehn, Werner" <werner.daehn@sap.com>
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Subject: Transactional Consistency beyond KIP-98 (transactional consumer)
Thread-Topic: Transactional Consistency beyond KIP-98 (transactional consumer)
Thread-Index: AdL0uNRi3qtwixMXRzaoLrwUO2TajQ==
Date: Tue, 4 Jul 2017 11:29:43 +0000
Message-ID: <1589edb4cf434b159402895bc0229b21@sap.com>
Accept-Language: de-DE, en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_1589edb4cf434b159402895bc0229b21sapcom_"
MIME-Version: 1.0
archived-at: Wed, 05 Jul 2017 14:24:42 -0000

--_000_1589edb4cf434b159402895bc0229b21sapcom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

While the new transactional consistency feature is definitely a move into t=
he right direction, I find the current design limitations quite severe. Und=
erstandable but severe.

First is the offset being the add-time, not the commit time. The consequenc=
e of that is, if one transactional producer adds rows every once a while an=
d does commit minutes or hours later, all consumers with read=3Dcommitted w=
ill not see any other producer's data until this one did the commit. Will d=
ecrease the stability of the overall solution, won't it?
Would have been better to use the commit time as the offset value. Harder t=
o implement but more stable.

The other issue is at the consumer side. Current implementation does not pr=
ovide transaction guarantees on the consumers. Again, for logical reasons a=
nd to simplify the implementation, but as a user you want to have both - wh=
en data is committed together you want to read the entire transaction as on=
e unit and complete.

Are there any plans to go beyond the current implementation?

Thanks in advance

--_000_1589edb4cf434b159402895bc0229b21sapcom_--