kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Pearce <Michael.Pea...@ig.com>
Subject Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging
Date Fri, 17 Feb 2017 11:28:26 GMT

I think need some unified agreement on the VarInts.

Would this also change in all other area’s of the protocol, e.g. value and key length in
message protocol, to keep this uniform across all protocols going forwards?

On 17/02/2017, 00:23, "Apurva Mehta" <apurva@confluent.io> wrote:

    Hi Jun,

    Thanks for the reply. Comments inline.

    On Thu, Feb 16, 2017 at 2:29 PM, Jun Rao <jun@confluent.io> wrote:

    > Hi, Apurva,
    > Thanks for the reply. A couple of comment below.
    > On Wed, Feb 15, 2017 at 9:45 PM, Apurva Mehta <apurva@confluent.io> wrote:
    > > Hi Jun,
    > >
    > > Answers inline:
    > >
    > > 210. Pid snapshots: Is the number of pid snapshot configurable or
    > hardcoded
    > > > with 2? When do we decide to roll a new snapshot? Based on time, byte,
    > or
    > > > offset? Is that configurable too?
    > > >

    > When a replica becomes a follower, we do a bit log truncation. Having an
    > older snapshot allows us to recover the PID->sequence mapping much quicker
    > than rescanning the whole log.

    This is a good point. I have updated the doc with a more detailed proposal.
    Essentially, snapshots will be created on a periodic basis. A reasonable
    period would be every 30 or 60 seconds. We will keep at most 2 copies of
    the snapshot file. With this setup, we would have to replay at most 60 or
    120 seconds of the log in the event of log truncation during leader

    If we need to make any of this configurable, we can expose a config in the
    future. It would be easier to add a config we need than remove one with
    marginal utility.

    > > >
    > > > 211. I am wondering if we should store ExpirationTime in the producer
    > > > transactionalId mapping message as we do in the producer transaction
    > > status
    > > > message. If a producer only calls initTransactions(), but never
    > publishes
    > > > any data, we still want to be able to expire and remove the producer
    > > > transactionalId mapping message.
    > > >
    > > >
    > > Actually, the document was inaccurate. The transactionalId will be
    > expired
    > > only if there is no active transaction, and the age of the last
    > transaction
    > > with that transactionalId is older than the transactioanlId expiration
    > > time. With these semantics, storing the expiration time in the
    > > transactionalId mapping message won't be useful, since the expiration
    > time
    > > is a moving target based on transaction activity.
    > >
    > > I have updated the doc with a clarification.
    > >
    > >
    > >
    > Currently, the producer transactionalId mapping message doesn't carry
    > ExpirationTime, but the producer transaction status message does.  It would
    > be useful if they are consistent.
    You are right. The document has been updated to remove the ExpirationTime
    from the transaction status messages as well. Any utility for this field
    can be achieved by using the timestamp of the message itself along with
    another expiration time (like transactionalId expiration time, transaction
    expiration time, etc.).


The information contained in this email is strictly confidential and for the use of the addressee
only, unless otherwise indicated. If you are not the intended recipient, please do not read,
copy, use or disclose to others this message or any attachment. Please also notify the sender
by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and
any copies of it. Opinions, conclusion (etc) that do not relate to the official business of
this company shall be understood as neither given nor endorsed by it. IG is a trading name
of IG Markets Limited (a company registered in England and Wales, company number 04008957)
and IG Index Limited (a company registered in England and Wales, company number 01190902).
Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets
Limited (register number 195355) and IG Index Limited (register number 114059) are authorised
and regulated by the Financial Conduct Authority.
View raw message