kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renu Tewari <tewa...@gmail.com>
Subject Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
Date Wed, 26 Oct 2016 18:01:45 GMT
+1 @Magnus.
It is also in line with traditional use of the magic field to indicate a
change in the format of the message. Thus a change in the magic field
indicates a different "schema" which in this case would reflect adding a
new field, removing a field or changing the type of fields etc.

The version number bump does not change the message format but just the
interpretation of the values.

Going with what @Michael suggested on keeping it simple we don't need to a
add a new field to indicate a tombstone and therefore require a change in
the magic byte field. The attribute bit is sufficient for indicating the
new interpretation of attribute.bit5 to indicate a tombstone.

Bumping the version number and changing the attribute bit keeps it simple.


Thanks
Renu



On Wed, Oct 26, 2016 at 10:09 AM, Mayuresh Gharat <
gharatmayuresh15@gmail.com> wrote:

> I see the reasoning Magnus described.
> If you check the docs https://kafka.apache.org/documentation#messageformat
> ,
> it says :
>
> 1 byte "magic" identifier to allow format changes, value is 0 or 1
>
> Moreover as per comments in the code :
> /**
>    * The "magic" value
>    * When magic value is 0, the message uses absolute offset and does not
> have a timestamp field.
>    * When magic value is 1, the message uses relative offset and has a
> timestamp field.
>    */
>
> Since timeStamp was added as an actual field we bumped the the magic byte
> to 1 for this change.
> But since we are not adding an actual field, we can do away with bumping up
> the magic byte.
>
> If we really want to go the standard route of bumping up the magic byte for
> any change to message format we should actually add a new field for
> handling log compaction instead of just using the attribute field, which
> might sound like an overkill.
>
> Thanks,
>
> Mayuresh
>
>
>
>
>
>
> On Wed, Oct 26, 2016 at 1:32 AM, Magnus Edenhill <magnus@edenhill.se>
> wrote:
>
> > 2016-10-25 21:36 GMT+02:00 Nacho Solis <nsolis@linkedin.com.invalid>:
> >
> > > I think you probably require a MagicByte bump if you expect correct
> > > behavior of the system as a whole.
> > >
> > > From a client perspective you want to make sure that when you deliver a
> > > message that the broker supports the feature you're expecting
> > > (compaction).  So, depending on the behavior of the broker on
> > encountering
> > > a previously undefined bit flag I would suggest making some change to
> > make
> > > certain that flag-based compaction is supported.  I'm going to guess
> that
> > > the MagicByte would do this.
> > >
> >
> > I dont believe this is needed since it is already attributed through the
> > request's API version.
> >
> > Producer:
> >  * if a client sends ProduceRequest V4 then attributes.bit5 indicates a
> > tombstone
> >  * if a clients sends ProduceRequest <V4 then attributes.bit5 is ignored
> > and value==null indicates a tombstone
> >  * in both cases the on-disk messages are stored with attributes.bit5 (I
> > assume?)
> >
> > Consumer:
> >  * if a clients sends FetchRequest V4 messages are sendfile():ed directly
> > from disk (with attributes.bit5)
> >  * if a client sends FetchRequest <V4 messages are slowpathed and
> > translated from attributes.bit5 to value=null as required.
> >
> >
> > That's my understanding anyway, please correct me if I'm wrong.
> >
> > /Magnus
> >
> >
> >
> > > On Tue, Oct 25, 2016 at 10:17 AM, Magnus Edenhill <magnus@edenhill.se>
> > > wrote:
> > >
> > > > It is safe to assume that a previously undefined attributes bit will
> be
> > > > unset in protocol requests from existing clients, if not, such a
> client
> > > is
> > > > already violating the protocol and needs to be fixed.
> > > >
> > > > So I dont see a need for a MagicByte bump, both broker and client has
> > the
> > > > information it needs to construct or parse the message according to
> > > request
> > > > version.
> > > >
> > > >
> > > > 2016-10-25 18:48 GMT+02:00 Michael Pearce <Michael.Pearce@ig.com>:
> > > >
> > > > > Hi Magnus,
> > > > >
> > > > > I was wondering if I even needed to change those also, as
> technically
> > > > > we’re just making use of a non used attribute bit, but im not 100%
> > that
> > > > it
> > > > > be always false currently.
> > > > >
> > > > > If someone can say 100% it will already be set false with current
> and
> > > > > historic bit wise masking techniques used over the time, we could
> do
> > > away
> > > > > with both, and simply just start to use it. Unfortunately I don’t
> > have
> > > > that
> > > > > historic knowledge so was hoping it would be flagged up in this
> > > > discussion
> > > > > thread ☺
> > > > >
> > > > > Cheers
> > > > > Mike
> > > > >
> > > > > On 10/25/16, 5:36 PM, "Magnus Edenhill" <magnus@edenhill.se>
> wrote:
> > > > >
> > > > >     Hi Michael,
> > > > >
> > > > >     With the version bumps for Produce and Fetch requests, do you
> > > really
> > > > > need
> > > > >     to bump MagicByte too?
> > > > >
> > > > >     Regards,
> > > > >     Magnus
> > > > >
> > > > >
> > > > >     2016-10-25 18:09 GMT+02:00 Michael Pearce <
> Michael.Pearce@ig.com
> > >:
> > > > >
> > > > >     > Hi All,
> > > > >     >
> > > > >     > I would like to discuss the following KIP proposal:
> > > > >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >     > 87+-+Add+Compaction+Tombstone+Flag
> > > > >     >
> > > > >     > This is off the back of the discussion on KIP-82  / KIP
> meeting
> > > > > where it
> > > > >     > was agreed to separate this issue and feature. See:
> > > > >     > http://mail-archives.apache.org/mod_mbox/kafka-dev/201610.
> > > > >     > mbox/%3cCAJS3ho8OcR==EcxsJ8OP99pD2hz=iiGecWsv-
> > > > >     > EZsBsNyDcKr=g@mail.gmail.com%3e
> > > > >     >
> > > > >     > Thanks
> > > > >     > Mike
> > > > >     >
> > > > >     > The information contained in this email is strictly
> > confidential
> > > > and
> > > > > for
> > > > >     > the use of the addressee only, unless otherwise indicated.
If
> > you
> > > > > are not
> > > > >     > the intended recipient, please do not read, copy, use or
> > disclose
> > > > to
> > > > > others
> > > > >     > this message or any attachment. Please also notify the sender
> > by
> > > > > replying
> > > > >     > to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > > > the email
> > > > >     > and any copies of it. Opinions, conclusion (etc) that do
not
> > > relate
> > > > > to the
> > > > >     > official business of this company shall be understood as
> > neither
> > > > > given nor
> > > > >     > endorsed by it. IG is a trading name of IG Markets Limited
(a
> > > > company
> > > > >     > registered in England and Wales, company number 04008957)
and
> > IG
> > > > > Index
> > > > >     > Limited (a company registered in England and Wales, company
> > > number
> > > > >     > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > > > Hill,
> > > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > > > and IG
> > > > >     > Index Limited (register number 114059) are authorised and
> > > regulated
> > > > > by the
> > > > >     > Financial Conduct Authority.
> > > > >     >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly confidential
> and
> > > for
> > > > > the use of the addressee only, unless otherwise indicated. If you
> are
> > > not
> > > > > the intended recipient, please do not read, copy, use or disclose
> to
> > > > others
> > > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to
> > > > the
> > > > > official business of this company shall be understood as neither
> > given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > > > registered in England and Wales, company number 04008957) and IG
> > Index
> > > > > Limited (a company registered in England and Wales, company number
> > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> > > IG
> > > > > Index Limited (register number 114059) are authorised and regulated
> > by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Nacho (Ignacio) Solis
> > > Kafka
> > > nsolis@linkedin.com
> > >
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message