kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Becket Qin <becket....@gmail.com>
Subject Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
Date Wed, 09 Nov 2016 18:49:30 GMT
I am not sure if we need the second stage. Wouldn't it be enough to say
that a message is a tombstone if one of the following is true?
1. tombstone flag is set.
2. value is null.

If we go to stage 2, the only difference is that we can theoretically
support a null value non-tombstone message in a log compacted topic, but I
am not sure if that has any use case.

Thanks,

Jiangjie (Becket) Qin


On Wed, Nov 9, 2016 at 9:23 AM, Mayuresh Gharat <gharatmayuresh15@gmail.com>
wrote:

> I think it will be a good idea. +1
>
> Thanks,
>
> Mayuresh
>
> On Wed, Nov 9, 2016 at 9:13 AM, Michael Pearce <Michael.Pearce@ig.com>
> wrote:
>
> > +1 Mayuresh, I think this is a good solution/strategy.
> >
> > Shall we update the KIP with this? Becket/Jun/Joel any comments to add
> > before we do?
> >
> > On 08/11/2016, 17:29, "Mayuresh Gharat" <gharatmayuresh15@gmail.com>
> > wrote:
> >
> >     I think the migration can be done in 2 stages :
> >
> >     1) In first stage the broker should understand the attribute flag as
> > well
> >     as Null for the value for log compaction.
> >     2) In second stage we move on to supporting only the attribute flag
> > for log
> >     compaction.
> >
> >     I agree with Becket that for older clients (consumers) the broker
> might
> >     have to down convert a message that has the attribute flag set for
> log
> >     compacting but has a non null value. But this should be in first
> stage.
> >     Once all the clients have upgraded (clients start recognizing the
> > attribute
> >     flag), we can move the broker to stage 2.
> >
> >     Thanks,
> >
> >     Mayuresh
> >
> >     On Tue, Nov 8, 2016 at 12:06 AM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> >     wrote:
> >
> >     > Also we can add further guidance:
> >     >
> >     > To  avoid the below caveat to organisations by promoting of
> > upgrading all
> >     > consumers first before relying on producing tombstone messages with
> > data
> >     >
> >     > Sent using OWA for iPhone
> >     > ________________________________________
> >     > From: Michael Pearce
> >     > Sent: Tuesday, November 8, 2016 8:03:32 AM
> >     > To: dev@kafka.apache.org
> >     > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> >     >
> >     > Thanks Jun on the feedback, I think I understand the issue/point
> now.
> >     >
> >     > We def can add that on older client version if tombstone marker
> make
> > the
> >     > value null to preserve behaviour.
> >     >
> >     > There is one caveats to this:
> >     >
> >     > * we have to be clear that data is lost if reading via old
> > client/message
> >     > format - I don't think this is a big issue as mostly the idea/use
> > case is
> >     > around meta data transport as such would only be as bad as current
> > situation
> >     >
> >     > Re having configurable broker this was to handle cases like you
> > described
> >     > but in another way by allowing organisation choose the behaviour of
> > the
> >     > compaction per broker or per topic so they could manage their
> > transition to
> >     > using tombstone markers.
> >     >
> >     > On hind sight it maybe easier to just upgrade and downgrade the
> > messages
> >     > on version as you propose.
> >     >
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > Sent using OWA for iPhone
> >     > ________________________________________
> >     > From: Jun Rao <jun@confluent.io>
> >     > Sent: Tuesday, November 8, 2016 12:34:41 AM
> >     > To: dev@kafka.apache.org
> >     > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> >     >
> >     > For the use case, one potential use case is for schema
> registration.
> > For
> >     > example, in Avro, a null value corresponds to a Null schema. So, if
> > you
> >     > want to be able to keep the schema id in a delete message, the
> value
> > can't
> >     > be null. We could get around this issue by specializing null value
> > during
> >     > schema registration though.
> >     >
> >     > Now for the proposed changes. We probably should preserve client
> >     > compatibility. If a client application is sending a null value to a
> >     > compacted topic, ideally, it should work the same after the client
> >     > upgrades.
> >     >
> >     > I am not sure about making the tombstone marker configurable,
> > especially at
> >     > the topic level. Should we allow users to change the config values
> > back and
> >     > forth, and what would be the implication?
> >     >
> >     > Thanks,
> >     >
> >     > Jun
> >     >
> >     > On Mon, Nov 7, 2016 at 10:48 AM, Becket Qin <becket.qin@gmail.com>
> > wrote:
> >     >
> >     > > Hi Michael,
> >     > >
> >     > > Yes, changing the logic in the log cleaner makes sense. There
> > could be
> >     > some
> >     > > other thing worth thinking (e.g. the message size change after
> >     > conversion),
> >     > > though.
> >     > >
> >     > > The scenario I was thinking is the following:
> >     > > Imagine a distributed caching system built on top of Kafka. A
> user
> > is
> >     > > consuming from a topic and it is guaranteed that if the user
> > consume to
> >     > the
> >     > > log end it will get the latest value for all the keys. Currently
> > if the
> >     > > consumer sees a null value it knows the key has been removed. Now
> > let's
> >     > say
> >     > > we rolled out this change. And the producer applies a message
> with
> > the
> >     > > tombstone flag set, but the value was not null. When we append
> that
> >     > message
> >     > > to the log I suppose we will not do the down conversion if the
> > broker has
> >     > > set the message.format.version to the latest. Because the log
> > cleaner
> >     > won't
> >     > > touch the active log segment, so that message will be sitting in
> > the
> >     > active
> >     > > segment as is. Now when a consumer that hasn't upgraded yet
> > consumes that
> >     > > tombstone message in the active segment, it seems that the broker
> > will
> >     > need
> >     > > to down convert that message to remove the value, right? In this
> > case, we
> >     > > cannot wait for the log cleaner to do the down conversion because
> > that
> >     > > message may have already been consumed before the log compaction
> > happens.
> >     > >
> >     > > Thanks,
> >     > >
> >     > > Jiangjie (Becket) Qin
> >     > >
> >     > >
> >     > >
> >     > > On Mon, Nov 7, 2016 at 9:59 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> >     > > wrote:
> >     > >
> >     > > > Hi Becket,
> >     > > >
> >     > > > We were thinking more about having the logic that’s in the
> method
> >     > > > shouldRetainMessage configurable via http://kafka.apache.org/
> >     > > > documentation.html#brokerconfigs  at a broker/topic level. And
> > then
> >     > > scrap
> >     > > > auto converting the message, and allow organisations to manage
> > the
> >     > > rollout
> >     > > > of enabling of the feature.
> >     > > > (this isn’t in documentation but in response to the discussion
> > thread
> >     > as
> >     > > > an alternative approach to roll out the feature)
> >     > > >
> >     > > > Does this make any more sense?
> >     > > >
> >     > > > Thanks
> >     > > > Mike
> >     > > >
> >     > > > On 11/3/16, 2:27 PM, "Becket Qin" <becket.qin@gmail.com>
> wrote:
> >     > > >
> >     > > >     Hi Michael,
> >     > > >
> >     > > >     Do you mean using a new configuration it is just the
> exiting
> >     > > >     message.format.version config? It seems the
> > message.format.version
> >     > > > config
> >     > > >     is enough in this case. And the default value would always
> > be the
> >     > > > latest
> >     > > >     version.
> >     > > >
> >     > > >     > Message version migration would be handled as like in
> > KIP-32
> >     > > >
> >     > > >     Also just want to confirm on this. Today if an old consumer
> >     > consumes
> >     > > a
> >     > > > log
> >     > > >     compacted topic and sees an empty value, it knows that is a
> >     > > tombstone.
> >     > > >     After we start to use the attribute bit, a tombstone
> message
> > can
> >     > > have a
> >     > > >     non-empty value. So by "like in KIP-32" you mean we will
> > remove the
> >     > > > value
> >     > > >     to down convert the message if the consumer version is old,
> > right?
> >     > > >
> >     > > >     Thanks.
> >     > > >
> >     > > >     Jiangjie (Becket) Qin
> >     > > >
> >     > > >     On Wed, Nov 2, 2016 at 1:37 AM, Michael Pearce <
> >     > > Michael.Pearce@ig.com>
> >     > > >     wrote:
> >     > > >
> >     > > >     > Hi Joel , et al.
> >     > > >     >
> >     > > >     > Any comments on the below idea to handle roll out /
> > compatibility
> >     > > of
> >     > > > this
> >     > > >     > feature, using a configuration?
> >     > > >     >
> >     > > >     > Does it make sense/clear?
> >     > > >     > Does it add value?
> >     > > >     > Do we want to enforce flag by default, or value by
> > default, or
> >     > > both?
> >     > > >     >
> >     > > >     > Cheers
> >     > > >     > Mike
> >     > > >     >
> >     > > >     >
> >     > > >     > On 10/27/16, 4:47 PM, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> >     > > > wrote:
> >     > > >     >
> >     > > >     >     Thanks, James, I think this is a really good addition
> > to the
> >     > > KIP
> >     > > >     > details, please feel free to amend the wiki/add the use
> > cases,
> >     > also
> >     > > > if any
> >     > > >     > others you think of. I definitely think its worthwhile
> >     > documenting.
> >     > > > If you
> >     > > >     > can’t let me know ill add them next week (just leaving
> for
> > a long
> >     > > > weekend
> >     > > >     > off)
> >     > > >     >
> >     > > >     >     Re Joel and others comments about upgrade and
> > compatibility.
> >     > > >     >
> >     > > >     >     Rather than trying to auto manage this.
> >     > > >     >
> >     > > >     >     Actually maybe we make a configuration option, both
> at
> > server
> >     > > > and per
> >     > > >     > topic level to control the behavior of how the server
> logic
> >     > should
> >     > > > work out
> >     > > >     > if the record, is a tombstone record .
> >     > > >     >
> >     > > >     >     e.g.
> >     > > >     >
> >     > > >     >     key = compation.tombstone.marker
> >     > > >     >
> >     > > >     >     value options:
> >     > > >     >
> >     > > >     >     value   (continues to use null value as tombstone
> > marker)
> >     > > >     >     flag (expects to use the tombstone flag)
> >     > > >     >     value_or_flag (if either is true it treats the record
> > as a
> >     > > > tombstone)
> >     > > >     >
> >     > > >     >     This way on upgrade users can keep current behavior,
> > and
> >     > slowly
> >     > > >     > migrate to the new. Having a transition period of using
> >     > > > value_or_flag,
> >     > > >     > finally having flag only if an organization wishes to use
> > null
> >     > > values
> >     > > >     > without it being treated as a tombstone marker (use case
> > noted
> >     > > below)
> >     > > >     >
> >     > > >     >     Having it both global broker level and topic override
> > also
> >     > > > allows some
> >     > > >     > flexibility here.
> >     > > >     >
> >     > > >     >     Cheers
> >     > > >     >     Mike
> >     > > >     >
> >     > > >     >
> >     > > >     >
> >     > > >     >
> >     > > >     >
> >     > > >     >
> >     > > >     >     On 10/27/16, 8:03 AM, "James Cheng" <
> > wushujames@gmail.com>
> >     > > > wrote:
> >     > > >     >
> >     > > >     >         This KIP would definitely address a gap in the
> > current
> >     > > >     > functionality, where you currently can't have a tombstone
> > with
> >     > any
> >     > > >     > associated content.
> >     > > >     >
> >     > > >     >         That said, I'd like to talk about use cases, to
> > make sure
> >     > > > that
> >     > > >     > this is in fact useful. The KIP should be updated with
> > whatever
> >     > use
> >     > > > cases
> >     > > >     > we come up with.
> >     > > >     >
> >     > > >     >         First of all, an observation: When we speak about
> > log
> >     > > > compaction,
> >     > > >     > we typically think of "the latest message for a key is
> > retained".
> >     > > In
> >     > > > that
> >     > > >     > respect, a delete tombstone (i.e. a message with a null
> > payload)
> >     > is
> >     > > > treated
> >     > > >     > the same as any other Kafka message: the latest message
> is
> >     > > retained.
> >     > > > It
> >     > > >     > doesn't matter whether the latest message is null, or if
> > the
> >     > latest
> >     > > > message
> >     > > >     > has actual content. In all cases, the last message is
> > retained.
> >     > > >     >
> >     > > >     >         The only way a delete tombstone is treated
> > differently
> >     > from
> >     > > > other
> >     > > >     > Kafka messages is that it automatically disappears after
> a
> > while.
> >     > > > The time
> >     > > >     > of deletion is specified using delete.retention.ms.
> >     > > >     >
> >     > > >     >         So what we're really talking about is, do we want
> > to
> >     > > support
> >     > > >     > messages in a log-compacted topic that auto-delete
> > themselves
> >     > after
> >     > > > a while?
> >     > > >     >
> >     > > >     >         In a thread from 2015, there was a discussion on
> >     > > first-class
> >     > > >     > support of headers between Roger Hoover, Felix GV, Jun
> > Rao, and
> >     > I.
> >     > > > See
> >     > > >     > thread at https://groups.google.com/d/
> > msg/confluent-platform/
> >     > > >     > 8xPbjyUE_7E/yQ1AeCufL_gJ <https://groups.google.com/d/
> >     > > >     > msg/confluent-platform/8xPbjyUE_7E/yQ1AeCufL_gJ> . In
> that
> >     > thread,
> >     > > > Jun
> >     > > >     > raised a good question that I didn't have a good answer
> > for at
> >     > the
> >     > > > time: If
> >     > > >     > a message is going to auto-delete itself after a while,
> how
> >     > > > important was
> >     > > >     > the message? That is, what information did the message
> > contain
> >     > that
> >     > > > was
> >     > > >     > important *for a while* but not so important that it
> > needed to be
> >     > > > kept
> >     > > >     > around forever?
> >     > > >     >
> >     > > >     >         Some use cases that I can think of:
> >     > > >     >
> >     > > >     >         1) Tracability. I would like to know who issued
> > this
> >     > delete
> >     > > >     > tombstone. It might include the hostname, IP of the
> > producer of
> >     > the
> >     > > > delete.
> >     > > >     >         2) Timestamps. I would like to know when this
> > delete was
> >     > > > issued.
> >     > > >     > This use case is already addressed by the availability of
> >     > > per-message
> >     > > >     > timestamps that came in 0.10.0
> >     > > >     >         3) Data provenance. I hope I'm using this phrase
> >     > correctly,
> >     > > > but
> >     > > >     > what I mean is, where did this delete come from? What
> > processing
> >     > > job
> >     > > >     > emitted it? What input to the processing job caused this
> > delete
> >     > to
> >     > > be
> >     > > >     > produced? For example, if a record in topic A was
> > processed and
> >     > > > caused a
> >     > > >     > delete tombstone to be emitted to topic B, I might like
> the
> >     > offset
> >     > > > of the
> >     > > >     > topic A message to be attached to the topic B message.
> >     > > >     >         4) Distributed tracing for stream topologies.
> This
> > might
> >     > > be a
> >     > > >     > slight repeat of the above use cases. In the
> microservices
> > world,
> >     > > we
> >     > > > can
> >     > > >     > generate call-graphs of webservices using tools like
> > Zipkin/
> >     > > > opentracing.io
> >     > > >     > <http://opentracing.io/>, or something homegrown like
> >     > > >     > https://engineering.linkedin.
> com/distributed-service-call-
> >     > > >     > graph/real-time-distributed-tracing-website-performance-
> >     > > and-efficiency
> >     > > > <
> >     > > >     > https://engineering.linkedin.
> com/distributed-service-call-
> >     > > >     > graph/real-time-distributed-tracing-website-performance-
> >     > > > and-efficiency>.
> >     > > >     > I can imagine that you might want to do something similar
> > for
> >     > > stream
> >     > > >     > processing topologies, where stream processing jobs carry
> > along
> >     > and
> >     > > > forward
> >     > > >     > along a globally unique identifier, and a distributed
> > topology
> >     > > graph
> >     > > > is
> >     > > >     > generated.
> >     > > >     >         5) Cases where processing a delete requires data
> > that is
> >     > > not
> >     > > >     > available in the message key. I'm not sure I have a good
> > example
> >     > of
> >     > > > this,
> >     > > >     > though. One hand-wavy example might be where I am
> > publishing
> >     > > > documents into
> >     > > >     > Kafka where the documentId is the message key, and the
> text
> >     > > contents
> >     > > > of the
> >     > > >     > document are in the message body. And I have a consuming
> > job that
> >     > > > does some
> >     > > >     > analytics on the message body. If that document gets
> > deleted,
> >     > then
> >     > > > the
> >     > > >     > consuming job might need the original message body in
> > order to
> >     > > > "delete"
> >     > > >     > that message's impact from the analytics. But I'm not
> sure
> > that
> >     > is
> >     > > a
> >     > > > great
> >     > > >     > example. If the consumer was worried about that, the
> > consumer
> >     > would
> >     > > >     > probably keep the original message around, stored by
> > primary key.
> >     > > > And then
> >     > > >     > all it would need from a delete message would be the
> > primary key
> >     > of
> >     > > > the
> >     > > >     > message.
> >     > > >     >
> >     > > >     >         Do people think these are valid use cases?
> >     > > >     >
> >     > > >     >         What are other use cases that people can think
> of?
> >     > > >     >
> >     > > >     >         -James
> >     > > >     >
> >     > > >     >         > On Oct 26, 2016, at 3:46 PM, Mayuresh Gharat <
> >     > > >     > gharatmayuresh15@gmail.com> wrote:
> >     > > >     >         >
> >     > > >     >         > +1 @Joel.
> >     > > >     >         > I think a clear migration plan of upgrading and
> >     > > > downgrading of
> >     > > >     > server and
> >     > > >     >         > clients along with handling of issues that Joel
> >     > > mentioned,
> >     > > > on
> >     > > >     > the KIP would
> >     > > >     >         > be really great.
> >     > > >     >         >
> >     > > >     >         > Thanks,
> >     > > >     >         >
> >     > > >     >         > Mayuresh
> >     > > >     >         >
> >     > > >     >         > On Wed, Oct 26, 2016 at 3:31 PM, Joel Koshy <
> >     > > > jjkoshy.w@gmail.com>
> >     > > >     > wrote:
> >     > > >     >         >
> >     > > >     >         >> I'm not sure why it would be useful, but it
> > should be
> >     > > >     > theoretically
> >     > > >     >         >> possible if the attribute bit alone is enough
> > to mark
> >     > a
> >     > > >     > tombstone. OTOH, we
> >     > > >     >         >> could consider that as invalid if we wish.
> > These are
> >     > > > relevant
> >     > > >     > details that
> >     > > >     >         >> I think should be added to the KIP.
> >     > > >     >         >>
> >     > > >     >         >> Also, in the few odd scenarios that I
> mentioned
> > we
> >     > > should
> >     > > > also
> >     > > >     > consider
> >     > > >     >         >> that fetches could be coming from other
> >     > > yet-to-be-upgraded
> >     > > >     > brokers in a
> >     > > >     >         >> cluster that is being upgraded. So we would
> > probably
> >     > > want
> >     > > > to
> >     > > >     > continue to
> >     > > >     >         >> support nulls as tombstones or down-convert in
> > a way
> >     > > that
> >     > > > we
> >     > > >     > are sure works
> >     > > >     >         >> with least surprise to fetchers.
> >     > > >     >         >>
> >     > > >     >         >> There is a slightly vague statement under
> >     > > "Compatibility,
> >     > > >     > Deprecation, and
> >     > > >     >         >> Migration Plan" that could benefit more
> details:
> >     > *Logic
> >     > > > would
> >     > > >     > base on
> >     > > >     >         >> current behavior of null value or if tombstone
> > flag
> >     > set
> >     > > to
> >     > > >     > true, as such
> >     > > >     >         >> wouldn't impact any existing flows simply
> allow
> > new
> >     > > > producers
> >     > > >     > to make use
> >     > > >     >         >> of the feature*. It is unclear to me based on
> > that
> >     > > > whether you
> >     > > >     > would
> >     > > >     >         >> interpret null as a tombstone if the tombstone
> >     > attribute
> >     > > > bit is
> >     > > >     > off.
> >     > > >     >         >>
> >     > > >     >         >> On Wed, Oct 26, 2016 at 3:10 PM, Xavier
> Léauté <
> >     > > >     > xavier@confluent.io>
> >     > > >     >         >> wrote:
> >     > > >     >         >>
> >     > > >     >         >>> Does this mean that starting with V4 requests
> > we
> >     > would
> >     > > > allow
> >     > > >     > storing null
> >     > > >     >         >>> messages in compacted topics? The KIP should
> > probably
> >     > > > clarify
> >     > > >     > the
> >     > > >     >         >> behavior
> >     > > >     >         >>> for null messages where the tombstone flag is
> > not
> >     > net.
> >     > > >     >         >>>
> >     > > >     >         >>> On Wed, Oct 26, 2016 at 1:32 AM Magnus
> > Edenhill <
> >     > > >     > magnus@edenhill.se>
> >     > > >     >         >>> wrote:
> >     > > >     >         >>>
> >     > > >     >         >>>> 2016-10-25 21:36 GMT+02:00 Nacho Solis
> >     > > >     > <nsolis@linkedin.com.invalid>:
> >     > > >     >         >>>>
> >     > > >     >         >>>>> I think you probably require a MagicByte
> > bump if
> >     > you
> >     > > > expect
> >     > > >     > correct
> >     > > >     >         >>>>> behavior of the system as a whole.
> >     > > >     >         >>>>>
> >     > > >     >         >>>>> From a client perspective you want to make
> > sure
> >     > that
> >     > > > when you
> >     > > >     >         >> deliver a
> >     > > >     >         >>>>> message that the broker supports the
> feature
> > you're
> >     > > > expecting
> >     > > >     >         >>>>> (compaction).  So, depending on the
> behavior
> > of the
> >     > > > broker on
> >     > > >     >         >>>> encountering
> >     > > >     >         >>>>> a previously undefined bit flag I would
> > suggest
> >     > > making
> >     > > > some
> >     > > >     > change to
> >     > > >     >         >>>> make
> >     > > >     >         >>>>> certain that flag-based compaction is
> > supported.
> >     > I'm
> >     > > > going
> >     > > >     > to guess
> >     > > >     >         >>> that
> >     > > >     >         >>>>> the MagicByte would do this.
> >     > > >     >         >>>>>
> >     > > >     >         >>>>
> >     > > >     >         >>>> I dont believe this is needed since it is
> > already
> >     > > > attributed
> >     > > >     > through
> >     > > >     >         >> the
> >     > > >     >         >>>> request's API version.
> >     > > >     >         >>>>
> >     > > >     >         >>>> Producer:
> >     > > >     >         >>>> * if a client sends ProduceRequest V4 then
> >     > > > attributes.bit5
> >     > > >     > indicates a
> >     > > >     >         >>>> tombstone
> >     > > >     >         >>>> * if a clients sends ProduceRequest <V4 then
> >     > > > attributes.bit5
> >     > > >     > is
> >     > > >     >         >> ignored
> >     > > >     >         >>>> and value==null indicates a tombstone
> >     > > >     >         >>>> * in both cases the on-disk messages are
> > stored with
> >     > > >     > attributes.bit5
> >     > > >     >         >> (I
> >     > > >     >         >>>> assume?)
> >     > > >     >         >>>>
> >     > > >     >         >>>> Consumer:
> >     > > >     >         >>>> * if a clients sends FetchRequest V4
> messages
> > are
> >     > > >     > sendfile():ed
> >     > > >     >         >> directly
> >     > > >     >         >>>> from disk (with attributes.bit5)
> >     > > >     >         >>>> * if a client sends FetchRequest <V4
> messages
> > are
> >     > > > slowpathed
> >     > > >     > and
> >     > > >     >         >>>> translated from attributes.bit5 to
> value=null
> > as
> >     > > > required.
> >     > > >     >         >>>>
> >     > > >     >         >>>>
> >     > > >     >         >>>> That's my understanding anyway, please
> > correct me if
> >     > > I'm
> >     > > >     > wrong.
> >     > > >     >         >>>>
> >     > > >     >         >>>> /Magnus
> >     > > >     >         >>>>
> >     > > >     >         >>>>
> >     > > >     >         >>>>
> >     > > >     >         >>>>> On Tue, Oct 25, 2016 at 10:17 AM, Magnus
> > Edenhill <
> >     > > >     >         >> magnus@edenhill.se>
> >     > > >     >         >>>>> wrote:
> >     > > >     >         >>>>>
> >     > > >     >         >>>>>> It is safe to assume that a previously
> > undefined
> >     > > > attributes
> >     > > >     > bit
> >     > > >     >         >> will
> >     > > >     >         >>> be
> >     > > >     >         >>>>>> unset in protocol requests from existing
> > clients,
> >     > if
> >     > > > not,
> >     > > >     > such a
> >     > > >     >         >>> client
> >     > > >     >         >>>>> is
> >     > > >     >         >>>>>> already violating the protocol and needs
> to
> > be
> >     > > fixed.
> >     > > >     >         >>>>>>
> >     > > >     >         >>>>>> So I dont see a need for a MagicByte bump,
> > both
> >     > > > broker and
> >     > > >     > client
> >     > > >     >         >> has
> >     > > >     >         >>>> the
> >     > > >     >         >>>>>> information it needs to construct or parse
> > the
> >     > > message
> >     > > >     > according to
> >     > > >     >         >>>>> request
> >     > > >     >         >>>>>> version.
> >     > > >     >         >>>>>>
> >     > > >     >         >>>>>>
> >     > > >     >         >>>>>> 2016-10-25 18:48 GMT+02:00 Michael Pearce
> <
> >     > > >     > Michael.Pearce@ig.com>:
> >     > > >     >         >>>>>>
> >     > > >     >         >>>>>>> Hi Magnus,
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>> I was wondering if I even needed to
> change
> > those
> >     > > > also, as
> >     > > >     >         >>> technically
> >     > > >     >         >>>>>>> we’re just making use of a non used
> > attribute
> >     > bit,
> >     > > > but im
> >     > > >     > not
> >     > > >     >         >> 100%
> >     > > >     >         >>>> that
> >     > > >     >         >>>>>> it
> >     > > >     >         >>>>>>> be always false currently.
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>> If someone can say 100% it will already
> be
> > set
> >     > > false
> >     > > > with
> >     > > >     > current
> >     > > >     >         >>> and
> >     > > >     >         >>>>>>> historic bit wise masking techniques used
> > over
> >     > the
> >     > > > time,
> >     > > >     > we could
> >     > > >     >         >>> do
> >     > > >     >         >>>>> away
> >     > > >     >         >>>>>>> with both, and simply just start to use
> it.
> >     > > > Unfortunately
> >     > > >     > I don’t
> >     > > >     >         >>>> have
> >     > > >     >         >>>>>> that
> >     > > >     >         >>>>>>> historic knowledge so was hoping it would
> > be
> >     > > flagged
> >     > > > up in
> >     > > >     > this
> >     > > >     >         >>>>>> discussion
> >     > > >     >         >>>>>>> thread ?
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>> Cheers
> >     > > >     >         >>>>>>> Mike
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>> On 10/25/16, 5:36 PM, "Magnus Edenhill" <
> >     > > >     > magnus@edenhill.se>
> >     > > >     >         >>> wrote:
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>>    Hi Michael,
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>>    With the version bumps for Produce and
> > Fetch
> >     > > > requests,
> >     > > >     > do you
> >     > > >     >         >>>>> really
> >     > > >     >         >>>>>>> need
> >     > > >     >         >>>>>>>    to bump MagicByte too?
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>>    Regards,
> >     > > >     >         >>>>>>>    Magnus
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>>    2016-10-25 18:09 GMT+02:00 Michael
> > Pearce <
> >     > > >     >         >>> Michael.Pearce@ig.com
> >     > > >     >         >>>>> :
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>>> Hi All,
> >     > > >     >         >>>>>>>>
> >     > > >     >         >>>>>>>> I would like to discuss the following
> KIP
> >     > > proposal:
> >     > > >     >         >>>>>>>> https://cwiki.apache.org/
> >     > > > confluence/display/KAFKA/KIP-
> >     > > >     >         >>>>>>>> 87+-+Add+Compaction+Tombstone+Flag
> >     > > >     >         >>>>>>>>
> >     > > >     >         >>>>>>>> This is off the back of the discussion
> on
> > KIP-82
> >     > > /
> >     > > > KIP
> >     > > >     >         >>> meeting
> >     > > >     >         >>>>>>> where it
> >     > > >     >         >>>>>>>> was agreed to separate this issue and
> > feature.
> >     > > See:
> >     > > >     >         >>>>>>>> http://mail-archives.apache.
> >     > > > org/mod_mbox/kafka-dev/201610
> >     > > >     > .
> >     > > >     >         >>>>>>>> mbox/%3cCAJS3ho8OcR==
> > EcxsJ8OP99pD2hz=iiGecWsv-
> >     > > >     >         >>>>>>>> EZsBsNyDcKr=g@mail.gmail.com%3e
> >     > > >     >         >>>>>>>>
> >     > > >     >         >>>>>>>> Thanks
> >     > > >     >         >>>>>>>> Mike
> >     > > >     >         >>>>>>>>
> >     > > >     >         >>>>>>>> The information contained in this email
> is
> >     > > strictly
> >     > > >     >         >>>> confidential
> >     > > >     >         >>>>>> and
> >     > > >     >         >>>>>>> for
> >     > > >     >         >>>>>>>> the use of the addressee only, unless
> > otherwise
> >     > > > indicated.
> >     > > >     >         >> If
> >     > > >     >         >>>> you
> >     > > >     >         >>>>>>> are not
> >     > > >     >         >>>>>>>> the intended recipient, please do not
> > read,
> >     > copy,
> >     > > > use or
> >     > > >     >         >>>> disclose
> >     > > >     >         >>>>>> to
> >     > > >     >         >>>>>>> others
> >     > > >     >         >>>>>>>> this message or any attachment. Please
> > also
> >     > notify
> >     > > > the
> >     > > >     >         >> sender
> >     > > >     >         >>>> by
> >     > > >     >         >>>>>>> replying
> >     > > >     >         >>>>>>>> to this email or by telephone (+44(020
> > 7896
> >     > 0011)
> >     > > > and then
> >     > > >     >         >>>> delete
> >     > > >     >         >>>>>>> the email
> >     > > >     >         >>>>>>>> and any copies of it. Opinions,
> > conclusion (etc)
> >     > > > that do
> >     > > >     >         >> not
> >     > > >     >         >>>>> relate
> >     > > >     >         >>>>>>> to the
> >     > > >     >         >>>>>>>> official business of this company shall
> be
> >     > > > understood as
> >     > > >     >         >>>> neither
> >     > > >     >         >>>>>>> given nor
> >     > > >     >         >>>>>>>> endorsed by it. IG is a trading name of
> IG
> >     > Markets
> >     > > > Limited
> >     > > >     >         >> (a
> >     > > >     >         >>>>>> company
> >     > > >     >         >>>>>>>> registered in England and Wales, company
> > number
> >     > > > 04008957)
> >     > > >     >         >> and
> >     > > >     >         >>>> IG
> >     > > >     >         >>>>>>> Index
> >     > > >     >         >>>>>>>> Limited (a company registered in England
> > and
> >     > > Wales,
> >     > > >     > company
> >     > > >     >         >>>>> number
> >     > > >     >         >>>>>>>> 01190902). Registered address at Cannon
> > Bridge
> >     > > > House, 25
> >     > > >     >         >>>> Dowgate
> >     > > >     >         >>>>>>> Hill,
> >     > > >     >         >>>>>>>> London EC4R 2YA. Both IG Markets Limited
> >     > (register
> >     > > > number
> >     > > >     >         >>>> 195355)
> >     > > >     >         >>>>>>> and IG
> >     > > >     >         >>>>>>>> Index Limited (register number 114059)
> are
> >     > > > authorised and
> >     > > >     >         >>>>> regulated
> >     > > >     >         >>>>>>> by the
> >     > > >     >         >>>>>>>> Financial Conduct Authority.
> >     > > >     >         >>>>>>>>
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>> The information contained in this email
> is
> >     > strictly
> >     > > >     > confidential
> >     > > >     >         >>> and
> >     > > >     >         >>>>> for
> >     > > >     >         >>>>>>> the use of the addressee only, unless
> > otherwise
> >     > > > indicated.
> >     > > >     > If you
> >     > > >     >         >>> are
> >     > > >     >         >>>>> not
> >     > > >     >         >>>>>>> the intended recipient, please do not
> > read, copy,
> >     > > > use or
> >     > > >     > disclose
> >     > > >     >         >>> to
> >     > > >     >         >>>>>> others
> >     > > >     >         >>>>>>> this message or any attachment. Please
> also
> >     > notify
> >     > > > the
> >     > > >     > sender by
> >     > > >     >         >>>>> replying
> >     > > >     >         >>>>>>> to this email or by telephone (+44(020
> > 7896 0011)
> >     > > > and then
> >     > > >     > delete
> >     > > >     >         >>> the
> >     > > >     >         >>>>>> email
> >     > > >     >         >>>>>>> and any copies of it. Opinions,
> conclusion
> > (etc)
> >     > > > that do
> >     > > >     > not
> >     > > >     >         >> relate
> >     > > >     >         >>>> to
> >     > > >     >         >>>>>> the
> >     > > >     >         >>>>>>> official business of this company shall
> be
> >     > > > understood as
> >     > > >     > neither
> >     > > >     >         >>>> given
> >     > > >     >         >>>>>> nor
> >     > > >     >         >>>>>>> endorsed by it. IG is a trading name of
> IG
> >     > Markets
> >     > > > Limited
> >     > > >     > (a
> >     > > >     >         >>> company
> >     > > >     >         >>>>>>> registered in England and Wales, company
> > number
> >     > > > 04008957)
> >     > > >     > and IG
> >     > > >     >         >>>> Index
> >     > > >     >         >>>>>>> Limited (a company registered in England
> > and
> >     > Wales,
> >     > > > company
> >     > > >     >         >> number
> >     > > >     >         >>>>>>> 01190902). Registered address at Cannon
> > Bridge
> >     > > > House, 25
> >     > > >     > Dowgate
> >     > > >     >         >>>> Hill,
> >     > > >     >         >>>>>>> London EC4R 2YA. Both IG Markets Limited
> >     > (register
> >     > > > number
> >     > > >     > 195355)
> >     > > >     >         >>> and
> >     > > >     >         >>>>> IG
> >     > > >     >         >>>>>>> Index Limited (register number 114059)
> are
> >     > > > authorised and
> >     > > >     >         >> regulated
> >     > > >     >         >>>> by
> >     > > >     >         >>>>>> the
> >     > > >     >         >>>>>>> Financial Conduct Authority.
> >     > > >     >         >>>>>>>
> >     > > >     >         >>>>>>
> >     > > >     >         >>>>>
> >     > > >     >         >>>>>
> >     > > >     >         >>>>>
> >     > > >     >         >>>>> --
> >     > > >     >         >>>>> Nacho (Ignacio) Solis
> >     > > >     >         >>>>> Kafka
> >     > > >     >         >>>>> nsolis@linkedin.com
> >     > > >     >         >>>>>
> >     > > >     >         >>>>
> >     > > >     >         >>>
> >     > > >     >         >>
> >     > > >     >         >
> >     > > >     >         >
> >     > > >     >         >
> >     > > >     >         > --
> >     > > >     >         > -Regards,
> >     > > >     >         > Mayuresh R. Gharat
> >     > > >     >         > (862) 250-7125
> >     > > >     >
> >     > > >     >
> >     > > >     >
> >     > > >     >     The information contained in this email is strictly
> >     > > confidential
> >     > > > and
> >     > > >     > for the use of the addressee only, unless otherwise
> > indicated. If
> >     > > > you are
> >     > > >     > not the intended recipient, please do not read, copy, use
> > or
> >     > > > disclose to
> >     > > >     > others this message or any attachment. Please also notify
> > the
> >     > > sender
> >     > > > by
> >     > > >     > replying to this email or by telephone (+44(020 7896
> 0011)
> > and
> >     > then
> >     > > > delete
> >     > > >     > the email and any copies of it. Opinions, conclusion
> (etc)
> > that
> >     > do
> >     > > > not
> >     > > >     > relate to the official business of this company shall be
> >     > understood
> >     > > > as
> >     > > >     > neither given nor endorsed by it. IG is a trading name of
> > IG
> >     > > Markets
> >     > > >     > Limited (a company registered in England and Wales,
> company
> >     > number
> >     > > >     > 04008957) and IG Index Limited (a company registered in
> > England
> >     > and
> >     > > > Wales,
> >     > > >     > company number 01190902). Registered address at Cannon
> > Bridge
> >     > > House,
> >     > > > 25
> >     > > >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register
> >     > > > number
> >     > > >     > 195355) and IG Index Limited (register number 114059) are
> >     > > authorised
> >     > > > and
> >     > > >     > regulated by the Financial Conduct Authority.
> >     > > >     >
> >     > > >     >
> >     > > >     >
> >     > > >
> >     > > >
> >     > > > The information contained in this email is strictly
> confidential
> > and
> >     > for
> >     > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> >     > not
> >     > > > the intended recipient, please do not read, copy, use or
> > disclose to
> >     > > others
> >     > > > this message or any attachment. Please also notify the sender
> by
> >     > replying
> >     > > > to this email or by telephone (+44(020 7896 0011) and then
> > delete the
> >     > > email
> >     > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> >     > > the
> >     > > > official business of this company shall be understood as
> neither
> > given
> >     > > nor
> >     > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> >     > > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> >     > > > Limited (a company registered in England and Wales, company
> > number
> >     > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> >     > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> >     > IG
> >     > > > Index Limited (register number 114059) are authorised and
> > regulated by
> >     > > the
> >     > > > Financial Conduct Authority.
> >     > > >
> >     > >
> >     > The information contained in this email is strictly confidential
> and
> > for
> >     > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > the intended recipient, please do not read, copy, use or disclose
> to
> > others
> >     > this message or any attachment. Please also notify the sender by
> > replying
> >     > to this email or by telephone (+44(020 7896 0011) and then delete
> > the email
> >     > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to the
> >     > official business of this company shall be understood as neither
> > given nor
> >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >     > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > Limited (a company registered in England and Wales, company number
> >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > Index Limited (register number 114059) are authorised and regulated
> > by the
> >     > Financial Conduct Authority.
> >     >
> >
> >
> >
> >     --
> >     -Regards,
> >     Mayuresh R. Gharat
> >     (862) 250-7125
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message