kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From davek2 <...@git.apache.org>
Subject [GitHub] kafka pull request: Allocate 2 attribute bits to signal payload fo...
Date Tue, 24 May 2016 01:12:27 GMT
GitHub user davek2 opened a pull request:


    Allocate 2 attribute bits to signal payload format

    This documentation update proposes a mechanism to signal the serialization used for the
message payload, resolving issue https://issues.apache.org/jira/browse/KAFKA-3744.  No change
is made to the message structure; two previously-reserved bits in the attribute byte now have
defined values, and for one of four cases the key field is defined to be a JSON object.
    No change is required to messaging software.   No change is required to existing producer
and consumer clients that use pre-agreed payload serialization. 
    Misc notes:
    1) Only one attribute bit would be needed if serialization were always signalled using
the key field.  But it seems preferable to define two common serializations that do not have
any dependency on the key field.  Selection of the common formats is arbitrary; text and avro
seem reasonable but any two could be used instead.
    2) The compression attribute uses three bits but only two are defined.  If the intent
is to use all three bits for compression the undefined values should be listed as reserved;
if not, the timestamp attribute can slide down to bit 2 and serialization to bits 3~4, leaving
bits 5~7 reserved.
    3) It's unclear why message field 6 should be called "key" - a variable-length field is
more likely to be described as "attributes" or "metadata", and 1-byte field 3 would be called
"flags" instead of "attributes".
    4) Field 8 is called "payload" under message format and "value" under on-disk format.
 Changed to payload in both places.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/davek2/kafka trunk

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1419
commit 1d88b8d48cdfe67989bebf239f7588ca24e961b6
Author: Joe <davek2@hotmail.com>
Date:   2016-05-24T00:32:04Z

    Allocate 2 attribute bits for payload format


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message