kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <...@confluent.io>
Subject Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
Date Tue, 06 Dec 2016 18:25:28 GMT
Hi, Michael,

4. Hmm, does that mean the new client library can never send a null message
even to a regular topic? This seems like a change of the existing behavior.

Thanks,

Jun

On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <Michael.Pearce@ig.com>
wrote:

> Hi Jun,
>
> Re 4) That's because we expect the tombstone value to be set correctly if
> message bit is 2, as such if an older client sends in on old message the
> message is upcast and the bit is set correctly. And such no longer need to
> check the value. Mayuresh can you confirm my thinking and understanding of
> what we've discussed?
>
> The second point I understand what you're getting at now my apologies. Yes
> this makes sense to save on touching the message, if we're the only kip
> going in, in this release.
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jun Rao <jun@confluent.io>
> Sent: Tuesday, December 6, 2016 5:22:13 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hi, Michael,
>
> 4. Is this updated in the wiki? The text "If the magic byte on message is
> 2, the broker should use the tombstone bit for log compaction." doesn't
> seem to have changed.
>
> 2. My point is that if we change the message format just for this KIP, we
> should consider whether it's worth optimizing the down conversion path
> (i.e., decide whether a conversion is needed by just looking at the
> tombstone bit in the wrapper message) since tombstone will be used rarely.
> However, if the message format change here is combined with other KIPs,
> then this optimization likely won't be needed. The latter probably makes
> the code simpler. Jiangjie, Mayuresh, what do you think?
>
> Other than those, +1 from me,
>
> Thanks,
>
> Jun
>
> On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <Michael.Pearce@ig.com>
> wrote:
>
> > Hi Jun
> >
> > do we have your vote on this now?
> >
> > Any other concerns?
> >
> > Cheers
> > Mike
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Michael Pearce <Michael.Pearce@ig.com>
> > Sent: Saturday, December 3, 2016 1:37:45 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hi Jun,
> >
> > Have updated. Thanks again for the feedback.
> >
> > Agree yes we should align up when it gets to that, I assume you’ve
> flagged
> > the same in the other KIP?
> >
> > I think for now let’s get this KIP approved, then we can start the work
> to
> > get an initial PR, then we can discuss how to align the two if needed.
> >
> > Cheers,
> > Mike
> >
> > On 02/12/2016, 18:26, "Jun Rao" <jun@confluent.io> wrote:
> >
> >     Hi, Michael,
> >
> >     For 2), this is fine. Could you update the KIP wiki to make this and
> > other
> >     points clearer? Other than that, the KIP looks good to me.
> >
> >     An orthogonal thing is that there are other KIPs such as KIP-98 that
> > also
> >     intend to change the message format. If they all get approved, we
> > should
> >     think about whether it's better to just bump up the magic byte once
> to
> >     incorporate multiple format changes like we did in KIP-31/KIP-32.
> >
> >     Thanks,
> >
> >     Jun
> >
> >
> >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> >     wrote:
> >
> >     > Hi Jao
> >     >
> >     > Thanks for the response. Sorry for slow reply, both with personal
> > sickness
> >     > and also battling some critical issues encountered since upgrading
> to
> >     > 0.10.1.0
> >     >
> >     > 1) Thans for spotting, Document error where we branched this KIP
> from
> >     > KIP-82, will get that removed.
> >     > 2) Intent is to do this just at the record message level.
> >     > 3) Thanks for spotting, Will ensure this is corrected.
> >     > 4) As per discussion thread we will support tombstone + null value,
> >     > tombstone + non null value, no tombstone + null value.
> >     > 5) I believe this was in the discussion thread, @Mayuresh is this
> >     > something we’ve overlooked? I thought we would down convert and
> > remove the
> >     > value so the old consumer had existing behavior, or is there
> > something we
> >     > haven’t thought about?
> >     >
> >     > Cheers
> >     > Mike
> >     >
> >     > On 30/11/2016, 18:12, "Jun Rao" <jun@confluent.io> wrote:
> >     >
> >     >     Hi, Michael,
> >     >
> >     >     Thanks for the KIP. A few comments below.
> >     >
> >     >     1. The message format change contains "HeadersLength Headers".
> > Is that
> >     >     intended?
> >     >
> >     >     2. For compressed messageset, is the tombstone bit only set at
> > the
> >     > shallow
> >     >     level? Do we always leave that bit in the wrapper message
> unset?
> > An
> >     >     alternative is to set the tombstone bit in the wrapper if at
> > least one
> >     >     inner message has the tombstone bit set. This makes things a
> bit
> > more
> >     >     complicated, but we could potentially exploit that for
> > optimizing down
> >     >     conversion. For example, we only need to convert messages with
> > magic 2
> >     > to
> >     >     magic 1 if the wrapper's tombstone bit is set (conversion is
> > always
> >     > needed
> >     >     from magic 2 to magic 0). Not sure if the optimization is worth
> > the
> >     >     complexity though.
> >     >
> >     >     3. The referencing of the new version of
> > ProducerRequest/FetchRequest
> >     > is
> >     >     inconsistent (v4 vs v3). Since our convention starts at version
> > at 0, I
> >     >     think the new version would be 3.
> >     >
> >     >     4. "If the magic byte on message is 2, the broker should use
> the
> >     > tombstone
> >     >     bit for log compaction." What about null value? My
> understanding
> > is
> >     > that
> >     >     null value will be treated the same as setting the tombstone
> bit.
> >     >
> >     >     5. For the migration path, it would be useful to describe the
> > down
> >     >     conversion path to consumers (i.e., brokers on message format
> > 0.10.2
> >     > and
> >     >     consumers on older version).
> >     >
> >     >     Thanks,
> >     >
> >     >     Jun
> >     >
> >     >
> >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> >     > >
> >     >     wrote:
> >     >
> >     >     > Hi All,
> >     >     >
> >     >     > We have been discussing in the below thread and final changes
> > have
> >     > been
> >     >     > made to the KIP wiki based on these discussions.
> >     >     >
> >     >     > We would now like to put to the vote the following KIP:
> >     >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
> >     >     > Add+Compaction+Tombstone+Flag
> >     >     >
> >     >     > This kip is for having a distinct compaction attribute
> > “tombstone”
> >     > flag
> >     >     > instead of relying on null value, allowing non-null value
> > delete
> >     > messages.
> >     >     >
> >     >     > Many thanks,
> >     >     > Michael
> >     >     >
> >     >     >
> >     >     >
> >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> Michael.Pearce@ig.com>
> >     > wrote:
> >     >     >
> >     >     >     Hi Mayuresh,
> >     >     >
> >     >     >     LGTM. Ive just made one small adjustment updating the
> wire
> >     > protocol to
> >     >     > show the magic byte bump.
> >     >     >
> >     >     >     Do we think we’re good to put to a vote? Is there any
> > other bits
> >     >     > needing discussion?
> >     >     >
> >     >     >     Cheers
> >     >     >     Mike
> >     >     >
> >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> >     > gharatmayuresh15@gmail.com>
> >     >     > wrote:
> >     >     >
> >     >     >         Hi Michael,
> >     >     >
> >     >     >         I have updated the migration section of the KIP. Can
> > you
> >     > please
> >     >     > take a look?
> >     >     >
> >     >     >         Thanks,
> >     >     >
> >     >     >         Mayuresh
> >     >     >
> >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
> >     >     > gharatmayuresh15@gmail.com
> >     >     >         > wrote:
> >     >     >
> >     >     >         > Hi Michael,
> >     >     >         >
> >     >     >         > That whilst sending tombstone and non null value,
> the
> >     > consumer
> >     >     > can expect
> >     >     >         > only to receive the non-null message only in step
> > (3) is
> >     > this
> >     >     > correct?
> >     >     >         > ---> I do agree with you here.
> >     >     >         >
> >     >     >         > Becket, Ismael : can you guys review the migration
> > plan
> >     > listed
> >     >     > above using
> >     >     >         > magic byte?
> >     >     >         >
> >     >     >         > Thanks,
> >     >     >         >
> >     >     >         > Mayuresh
> >     >     >         >
> >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
> >     >     > Michael.Pearce@ig.com>
> >     >     >         > wrote:
> >     >     >         >
> >     >     >         >> Many thanks for this Mayuresh. I don't have any
> >     > objections.
> >     >     >         >>
> >     >     >         >> I assume we should state:
> >     >     >         >>
> >     >     >         >> That whilst sending tombstone and non null value,
> > the
> >     > consumer
> >     >     > can expect
> >     >     >         >> only to receive the non-null message only in step
> > (3) is
> >     > this
> >     >     > correct?
> >     >     >         >>
> >     >     >         >> Cheers
> >     >     >         >> Mike
> >     >     >         >>
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> Sent using OWA for iPhone
> >     >     >         >> ________________________________________
> >     >     >         >> From: Mayuresh Gharat <gharatmayuresh15@gmail.com
> >
> >     >     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
> >     >     >         >> To: dev@kafka.apache.org
> >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> > Tombstone
> >     > Flag
> >     >     >         >>
> >     >     >         >> Hi Ismael,
> >     >     >         >>
> >     >     >         >> Thanks for the explanation.
> >     >     >         >> Specially I like this part where in you mentioned
> > we can
> >     > get
> >     >     > rid of the
> >     >     >         >> older null value support for log compaction later
> > on,
> >     > here :
> >     >     >         >> We can't change semantics of the message format
> > without
> >     > having
> >     >     > a long
> >     >     >         >> transition period. And we can't rely
> >     >     >         >> on people reading documentation or acting on a
> > warning for
> >     >     > something so
> >     >     >         >> fundamental. As such, my take is that we need to
> > bump the
> >     > magic
> >     >     > byte. The
> >     >     >         >> good news is
> >     >     >         >> that we don't have to support all versions
> forever.
> > We
> >     > have
> >     >     > said that we
> >     >     >         >> will support direct upgrades for 2 years. That
> > means that
> >     >     > message format
> >     >     >         >> version n could, in theory, be removed 2 years
> > after the
> >     > it's
> >     >     > introduced.
> >     >     >         >>
> >     >     >         >> Just a heads up, I would like to mention that even
> > without
> >     >     > bumping magic
> >     >     >         >> byte, we will *NOT* loose zero copy as in the
> > client(x+1)
> >     > in my
> >     >     >         >> explanation
> >     >     >         >> above will convert internally a null value to
> have a
> >     > tombstone
> >     >     > bit set and
> >     >     >         >> a tombstone bit set to have a null value
> > automatically
> >     >     > internally and by
> >     >     >         >> the time we move to version (x+2), the clients
> > would have
> >     >     > upgraded.
> >     >     >         >> Obviously if we support a request from
> consumer(x),
> > we
> >     > will
> >     >     > loose zero
> >     >     >         >> copy
> >     >     >         >> but that is the same case with magic byte.
> >     >     >         >>
> >     >     >         >> But if magic byte bump makes life easier for
> > transition
> >     > for the
> >     >     > above
> >     >     >         >> reasons that you explained, I am OK with it since
> > we are
> >     > going
> >     >     > to meet the
> >     >     >         >> end goal down the road :)
> >     >     >         >>
> >     >     >         >> On a side note can we update the doc here on magic
> > byte
> >     > to say
> >     >     > that "*it
> >     >     >         >> should be bumped whenever the message format is
> > changed
> >     > or the
> >     >     >         >> interpretation of message format (usage of the
> > reserved
> >     > bits as
> >     >     > well) is
> >     >     >         >> changed*".
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> Hi Michael,
> >     >     >         >>
> >     >     >         >> Here is the update plan that we discussed offline
> >     > yesterday :
> >     >     >         >>
> >     >     >         >> Currently the magic-byte which corresponds to the
> >     >     > "message.format.version"
> >     >     >         >> is set to 1.
> >     >     >         >>
> >     >     >         >> 1) On broker it will be set to 1 initially.
> >     >     >         >>
> >     >     >         >> 2) When a producer client sends a message with
> > magic-byte
> >     > = 2,
> >     >     > since the
> >     >     >         >> broker is on magic-byte = 1, we will down convert
> > it,
> >     > which
> >     >     > means if the
> >     >     >         >> tombstone bit is set, the value will be set to
> > null. A
> >     > consumer
> >     >     >         >> understanding magic-byte = 1, will still work with
> > this. A
> >     >     > consumer
> >     >     >         >> working
> >     >     >         >> with magic-byte =2 will also be able to understand
> > this,
> >     > since
> >     >     > it
> >     >     >         >> understands the tombstone.
> >     >     >         >> Now there is still the question of supporting a
> >     > non-tombstone
> >     >     > and null
> >     >     >         >> value from producer client with magic-byte = 2.*
> (I
> > am
> >     > not sure
> >     >     > if we
> >     >     >         >> should support this. Ismael/Becket can comment
> > here)*
> >     >     >         >>
> >     >     >         >> 3) When almost all the clients have upgraded, the
> >     >     > message.format.version
> >     >     >         >> on
> >     >     >         >> the broker can be changed to 2, where in the down
> >     > conversion in
> >     >     > the above
> >     >     >         >> step will not happen. If at this point we get a
> > consumer
> >     >     > request from a
> >     >     >         >> older consumer, we might have to down convert
> where
> > in we
> >     > loose
> >     >     > zero copy,
> >     >     >         >> but these cases should be rare.
> >     >     >         >>
> >     >     >         >> Becket can you review this plan and add more
> > details if I
> >     > have
> >     >     >         >> missed/wronged something, before we put it on KIP.
> >     >     >         >>
> >     >     >         >> Thanks,
> >     >     >         >>
> >     >     >         >> Mayuresh
> >     >     >         >>
> >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
> >     >     > Michael.Pearce@ig.com>
> >     >     >         >> wrote:
> >     >     >         >>
> >     >     >         >> > Thanks guys, for discussing this offline and
> > getting
> >     > some
> >     >     > consensus.
> >     >     >         >> >
> >     >     >         >> > So its clear for myself and others what is
> > proposed now
> >     > (i
> >     >     > think i
> >     >     >         >> > understand, but want to make sure)
> >     >     >         >> >
> >     >     >         >> > Could i ask either directly update the kip to
> > detail the
> >     >     > migration
> >     >     >         >> > strategy, or (re-)state your offline discussed
> and
> >     > agreed
> >     >     > migration
> >     >     >         >> > strategy based on a magic byte is in this
> thread.
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > The main original driver for the KIP was to
> > support
> >     >     > compaction where
> >     >     >         >> value
> >     >     >         >> > isn't null, based off the discussions on KIP-82
> > thread.
> >     >     >         >> >
> >     >     >         >> > We should be able to support non-tombstone +
> null
> > value
> >     > by the
> >     >     >         >> completion
> >     >     >         >> > of the KIP, as we noted when discussing this
> kip,
> > having
> >     >     > logic based on
> >     >     >         >> a
> >     >     >         >> > null value isn't very clean and also separates
> the
> >     > concerns.
> >     >     >         >> >
> >     >     >         >> > As discussed already though we can split this
> into
> >     > KIP-87a
> >     >     > and KIP-87b
> >     >     >         >> >
> >     >     >         >> > Where we look to deliver KIP-87a on a compacted
> > topic
> >     > (to
> >     >     > address the
> >     >     >         >> > immediate issues)
> >     >     >         >> > * tombstone + null value
> >     >     >         >> > * tombstone + non-null value
> >     >     >         >> > * non-tombstone + non-null value
> >     >     >         >> >
> >     >     >         >> > Then we can discuss once KIP-87a is completed
> > options
> >     > later
> >     >     > and how we
> >     >     >         >> > support the second part KIP-87b to deliver:
> >     >     >         >> > * non-tombstone + null value
> >     >     >         >> >
> >     >     >         >> > Cheers
> >     >     >         >> > Mike
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > ________________________________________
> >     >     >         >> > From: Becket Qin <becket.qin@gmail.com>
> >     >     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
> >     >     >         >> > To: dev@kafka.apache.org
> >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> >     > Tombstone Flag
> >     >     >         >> >
> >     >     >         >> > Renu, Mayuresh and I had an offline discussion,
> > and
> >     > following
> >     >     > is a brief
> >     >     >         >> > summary.
> >     >     >         >> >
> >     >     >         >> > 1. We agreed that not bumping up magic value may
> > result
> >     > in
> >     >     > losing zero
> >     >     >         >> copy
> >     >     >         >> > during migration.
> >     >     >         >> > 2. Given that bumping up magic value is almost
> > free and
> >     > has
> >     >     > benefit of
> >     >     >         >> > avoiding potential performance issue. It is
> > probably
> >     > worth
> >     >     > doing.
> >     >     >         >> >
> >     >     >         >> > One issue we still need to think about is
> whether
> > we
> >     > want to
> >     >     > support a
> >     >     >         >> > non-tombstone message with null value.
> >     >     >         >> > Currently it is not supported by Kafka. If we
> > allow a
> >     >     > non-tombstone null
> >     >     >         >> > value message to exist after KIP-87. The problem
> > is
> >     > that such
> >     >     > message
> >     >     >         >> will
> >     >     >         >> > not be supported by the consumers prior to
> KIP-87.
> >     > Because a
> >     >     > null value
> >     >     >         >> > will always be interpreted to a tombstone.
> >     >     >         >> >
> >     >     >         >> > One option is that we keep the current way, i.e.
> > do not
> >     >     > support such
> >     >     >         >> > message. It would be good to know if there is a
> >     > concrete use
> >     >     > case for
> >     >     >         >> such
> >     >     >         >> > message. If there is not, we can probably just
> not
> >     > support it.
> >     >     >         >> >
> >     >     >         >> > Thanks,
> >     >     >         >> >
> >     >     >         >> > JIangjie (Becket) Qin
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh
> Gharat <
> >     >     >         >> > gharatmayuresh15@gmail.com
> >     >     >         >> > > wrote:
> >     >     >         >> >
> >     >     >         >> > > Hi Ismael,
> >     >     >         >> > >
> >     >     >         >> > > This is something I can think of for migration
> > plan:
> >     >     >         >> > > So the migration plan can look something like
> > this,
> >     > with up
> >     >     >         >> conversion :
> >     >     >         >> > >
> >     >     >         >> > > 1) Currently lets say we have Broker at
> version
> > x.
> >     >     >         >> > > 2) Currently we have clients at version x.
> >     >     >         >> > > 3) a) We move the version to Broker(x+1) :
> > supports
> >     > both
> >     >     > tombstone and
> >     >     >         >> > null
> >     >     >         >> > > for log compaction.
> >     >     >         >> > >     b) We upgrade the client to version
> > client(x+1) :
> >     > if in
> >     >     > the
> >     >     >         >> producer
> >     >     >         >> > > client(x+1) the value is set to null, we will
> >     > automatically
> >     >     > set the
> >     >     >         >> > > Tombstone bit internally. If the producer
> > client(x+1)
> >     > sets
> >     >     > the
> >     >     >         >> tombstone
> >     >     >         >> > > itself, well and good. For producer client(x),
> > the
> >     > broker
> >     >     > will up
> >     >     >         >> convert
> >     >     >         >> > > to have the tombstone bit. Broker(x+1) is
> > supporting
> >     > both.
> >     >     > Consumer
> >     >     >         >> > > client(x+1) will be aware of this and should
> be
> > able
> >     > to
> >     >     > handle this.
> >     >     >         >> For
> >     >     >         >> > > consumer client(x) we will down convert the
> > message
> >     > on the
> >     >     > broker
> >     >     >         >> side.
> >     >     >         >> > >     c) At this point we will have to specify a
> >     > warning or
> >     >     > clearly
> >     >     >         >> specify
> >     >     >         >> > > in docs that this behavior is about to be
> > changed for
> >     > log
> >     >     > compaction.
> >     >     >         >> > > 4) a) In next release of the Broker(x+2), we
> > say that
> >     > only
> >     >     > Tombstone
> >     >     >         >> is
> >     >     >         >> > > used for log compaction on the Broker side.
> >     > Clients(x+1)
> >     >     > still is
> >     >     >         >> > > supported.
> >     >     >         >> > >     b) We upgrade the client to version
> > client(x+2) :
> >     > if
> >     >     > value is set
> >     >     >         >> to
> >     >     >         >> > > null, tombstone will not be set automatically.
> > The
> >     > client
> >     >     > will have to
> >     >     >         >> > call
> >     >     >         >> > > setTombstone() to actually set the tombstone.
> >     >     >         >> > >
> >     >     >         >> > > We should compare this migration plan with the
> >     > migration
> >     >     > plan for
> >     >     >         >> magic
> >     >     >         >> > > byte bump and do whatever looks good.
> >     >     >         >> > > I am just worried that if we go down magic
> byte
> > route,
> >     >     > unless I am
> >     >     >         >> > missing
> >     >     >         >> > > something, it sounds like kafka will be stuck
> > with
> >     >     > supporting both
> >     >     >         >> null
> >     >     >         >> > > value and tombstone bit for log compaction for
> > life
> >     > long,
> >     >     > which does
> >     >     >         >> not
> >     >     >         >> > > look like a good end state.
> >     >     >         >> > >
> >     >     >         >> > > Thanks,
> >     >     >         >> > >
> >     >     >         >> > > Mayuresh
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh
> > Gharat <
> >     >     >         >> > > gharatmayuresh15@gmail.com
> >     >     >         >> > > > wrote:
> >     >     >         >> > >
> >     >     >         >> > > > Hi Ismael,
> >     >     >         >> > > >
> >     >     >         >> > > > That's a very good point which I might have
> > not
> >     >     > considered earlier.
> >     >     >         >> > > >
> >     >     >         >> > > > Here is a plan that I can think of:
> >     >     >         >> > > >
> >     >     >         >> > > > Stage 1) The broker from now on, up converts
> > the
> >     > message
> >     >     > to have the
> >     >     >         >> > > > tombstone marker. The log compaction thread
> > does log
> >     >     > compaction
> >     >     >         >> based
> >     >     >         >> > on
> >     >     >         >> > > > both null and tombstone marker. This is our
> >     > transition
> >     >     > period.
> >     >     >         >> > > > Stage 2) The next release we only say that
> log
> >     > compaction
> >     >     > is based
> >     >     >         >> on
> >     >     >         >> > > > tombstone marker. (Open source kafka makes
> > this as a
> >     >     > policy). By
> >     >     >         >> this
> >     >     >         >> > > time,
> >     >     >         >> > > > the organization which is moving to this
> > release
> >     > will be
> >     >     > sure that
> >     >     >         >> they
> >     >     >         >> > > > have gone through the entire transition
> > period.
> >     >     >         >> > > >
> >     >     >         >> > > > My only goal of doing this is that Kafka
> > clearly
> >     >     > specifies the end
> >     >     >         >> > state
> >     >     >         >> > > > about what log compaction means (is it null
> > value
> >     > or a
> >     >     > tombstone
> >     >     >         >> > marker,
> >     >     >         >> > > > but not both).
> >     >     >         >> > > >
> >     >     >         >> > > > What do you think?
> >     >     >         >> > > >
> >     >     >         >> > > > Thanks,
> >     >     >         >> > > >
> >     >     >         >> > > > Mayuresh
> >     >     >         >> > > > .
> >     >     >         >> > > >
> >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael
> Juma <
> >     >     > ismael@juma.me.uk>
> >     >     >         >> > wrote:
> >     >     >         >> > > >
> >     >     >         >> > > >> One comment below.
> >     >     >         >> > > >>
> >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh
> > Gharat <
> >     >     >         >> > > >> gharatmayuresh15@gmail.com
> >     >     >         >> > > >> > wrote:
> >     >     >         >> > > >>
> >     >     >         >> > > >> >    - If we don't bump up the magic byte,
> > on the
> >     > broker
> >     >     > side, the
> >     >     >         >> > > broker
> >     >     >         >> > > >> >    will always have to look at both
> > tombstone
> >     > bit and
> >     >     > the value
> >     >     >         >> when
> >     >     >         >> > > do
> >     >     >         >> > > >> the
> >     >     >         >> > > >> >    compaction. Assuming we do not bump up
> > the
> >     > magic
> >     >     > byte,
> >     >     >         >> > > >> >    imagine the broker sees a message
> which
> > does
> >     > not
> >     >     > have a
> >     >     >         >> tombstone
> >     >     >         >> > > bit
> >     >     >         >> > > >> >    set. The broker does not know when the
> >     > message was
> >     >     > produced
> >     >     >         >> (i.e.
> >     >     >         >> > > >> > whether
> >     >     >         >> > > >> >    the message has been up converted or
> > not), it
> >     > has
> >     >     > to take a
> >     >     >         >> > further
> >     >     >         >> > > >> > look at
> >     >     >         >> > > >> >    the value to see if it is null or not
> in
> >     > order to
> >     >     > determine
> >     >     >         >> if it
> >     >     >         >> > > is
> >     >     >         >> > > >> a
> >     >     >         >> > > >> >    tombstone. The same logic has to be
> put
> > on the
> >     >     > consumer as
> >     >     >         >> well
> >     >     >         >> > > >> because
> >     >     >         >> > > >> > the
> >     >     >         >> > > >> >    consumer does not know if the message
> > has
> >     > been up
> >     >     > converted or
> >     >     >         >> > not.
> >     >     >         >> > > >> >       - If we upconvert while appending,
> > this is
> >     > not
> >     >     > the case,
> >     >     >         >> > right?
> >     >     >         >> > > >>
> >     >     >         >> > > >>
> >     >     >         >> > > >> If I understand you correctly, this is not
> >     > sufficient
> >     >     > because the
> >     >     >         >> log
> >     >     >         >> > > may
> >     >     >         >> > > >> have messages appended before it was
> > upgraded to
> >     > include
> >     >     > KIP-87.
> >     >     >         >> > > >>
> >     >     >         >> > > >> Ismael
> >     >     >         >> > > >>
> >     >     >         >> > > >
> >     >     >         >> > > >
> >     >     >         >> > > >
> >     >     >         >> > > > --
> >     >     >         >> > > > -Regards,
> >     >     >         >> > > > Mayuresh R. Gharat
> >     >     >         >> > > > (862) 250-7125
> >     >     >         >> > > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > > --
> >     >     >         >> > > -Regards,
> >     >     >         >> > > Mayuresh R. Gharat
> >     >     >         >> > > (862) 250-7125
> >     >     >         >> > >
> >     >     >         >> > The information contained in this email is
> > strictly
> >     >     > confidential and for
> >     >     >         >> > the use of the addressee only, unless otherwise
> >     > indicated. If
> >     >     > you are
> >     >     >         >> not
> >     >     >         >> > the intended recipient, please do not read,
> copy,
> > use or
> >     >     > disclose to
> >     >     >         >> others
> >     >     >         >> > this message or any attachment. Please also
> > notify the
> >     > sender
> >     >     > by
> >     >     >         >> replying
> >     >     >         >> > to this email or by telephone (+44(020 7896
> 0011)
> > and
> >     > then
> >     >     > delete the
> >     >     >         >> email
> >     >     >         >> > and any copies of it. Opinions, conclusion (etc)
> > that
> >     > do not
> >     >     > relate to
> >     >     >         >> the
> >     >     >         >> > official business of this company shall be
> > understood as
> >     >     > neither given
> >     >     >         >> nor
> >     >     >         >> > endorsed by it. IG is a trading name of IG
> Markets
> >     > Limited (a
> >     >     > company
> >     >     >         >> > registered in England and Wales, company number
> >     > 04008957) and
> >     >     > IG Index
> >     >     >         >> > Limited (a company registered in England and
> > Wales,
> >     > company
> >     >     > number
> >     >     >         >> > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     >     > Dowgate Hill,
> >     >     >         >> > London EC4R 2YA. Both IG Markets Limited
> (register
> >     > number
> >     >     > 195355) and IG
> >     >     >         >> > Index Limited (register number 114059) are
> > authorised
> >     > and
> >     >     > regulated by
> >     >     >         >> the
> >     >     >         >> > Financial Conduct Authority.
> >     >     >         >> >
> >     >     >         >>
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> --
> >     >     >         >> -Regards,
> >     >     >         >> Mayuresh R. Gharat
> >     >     >         >> (862) 250-7125
> >     >     >         >> The information contained in this email is
> strictly
> >     >     > confidential and for
> >     >     >         >> the use of the addressee only, unless otherwise
> >     > indicated. If
> >     >     > you are not
> >     >     >         >> the intended recipient, please do not read, copy,
> > use or
> >     >     > disclose to others
> >     >     >         >> this message or any attachment. Please also notify
> > the
> >     > sender
> >     >     > by replying
> >     >     >         >> to this email or by telephone (+44(020 7896 0011)
> > and then
> >     >     > delete the email
> >     >     >         >> and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     >     > relate to the
> >     >     >         >> official business of this company shall be
> > understood as
> >     >     > neither given nor
> >     >     >         >> endorsed by it. IG is a trading name of IG Markets
> >     > Limited (a
> >     >     > company
> >     >     >         >> registered in England and Wales, company number
> > 04008957)
> >     > and
> >     >     > IG Index
> >     >     >         >> Limited (a company registered in England and
> Wales,
> >     > company
> >     >     > number
> >     >     >         >> 01190902). Registered address at Cannon Bridge
> > House, 25
> >     >     > Dowgate Hill,
> >     >     >         >> London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     >     > 195355) and IG
> >     >     >         >> Index Limited (register number 114059) are
> > authorised and
> >     >     > regulated by the
> >     >     >         >> Financial Conduct Authority.
> >     >     >         >>
> >     >     >         >
> >     >     >         >
> >     >     >         >
> >     >     >         > --
> >     >     >         > -Regards,
> >     >     >         > Mayuresh R. Gharat
> >     >     >         > (862) 250-7125
> >     >     >         >
> >     >     >
> >     >     >
> >     >     >
> >     >     >         --
> >     >     >         -Regards,
> >     >     >         Mayuresh R. Gharat
> >     >     >         (862) 250-7125
> >     >     >
> >     >     >
> >     >     >     The information contained in this email is strictly
> > confidential
> >     > and
> >     >     > for the use of the addressee only, unless otherwise
> indicated.
> > If
> >     > you are
> >     >     > not the intended recipient, please do not read, copy, use or
> >     > disclose to
> >     >     > others this message or any attachment. Please also notify the
> > sender
> >     > by
> >     >     > replying to this email or by telephone (+44(020 7896 0011)
> and
> > then
> >     > delete
> >     >     > the email and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     >     > relate to the official business of this company shall be
> > understood
> >     > as
> >     >     > neither given nor endorsed by it. IG is a trading name of IG
> > Markets
> >     >     > Limited (a company registered in England and Wales, company
> > number
> >     >     > 04008957) and IG Index Limited (a company registered in
> > England and
> >     > Wales,
> >     >     > company number 01190902). Registered address at Cannon Bridge
> > House,
> >     > 25
> >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register
> >     > number
> >     >     > 195355) and IG Index Limited (register number 114059) are
> > authorised
> >     > and
> >     >     > regulated by the Financial Conduct Authority.
> >     >     >
> >     >     >
> >     >     >
> >     >
> >     >
> >     > The information contained in this email is strictly confidential
> and
> > for
> >     > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > the intended recipient, please do not read, copy, use or disclose
> to
> > others
> >     > this message or any attachment. Please also notify the sender by
> > replying
> >     > to this email or by telephone (+44(020 7896 0011) and then delete
> > the email
> >     > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to the
> >     > official business of this company shall be understood as neither
> > given nor
> >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >     > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > Limited (a company registered in England and Wales, company number
> >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > Index Limited (register number 114059) are authorised and regulated
> > by the
> >     > Financial Conduct Authority.
> >     >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message