kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From radai <radai.rosenbl...@gmail.com>
Subject Re: [DISCUSS] KIP-82 - Add Record Headers
Date Fri, 10 Feb 2017 22:57:24 GMT
yeah I'm ok with both strings and multi-maps

On Fri, Feb 10, 2017 at 2:37 PM, Gwen Shapira <gwen@confluent.io> wrote:

> Can we also compromise on a multi-map?
>
> One of my use-cases is tagging on multi-DC setup. It will be nice if
> each "hop" can add "Tag" header with its own DC-name without having to
> edit a single string that someone else wrote. For some financial
> use-cases it will be a requirement.
>
> Gwen
>
> On Fri, Feb 10, 2017 at 12:19 PM, Ignacio Solis <isolis@igso.net> wrote:
> > I would be willing to compromise on Strings as keys if that moves all
> > of us forward.
> >
> > Right now headers are blocking a few of the things we want to do.
> >
> > Nacho
> >
> > On Thu, Feb 9, 2017 at 2:14 PM, Jay Kreps <jay@confluent.io> wrote:
> >> I'm neutral positive on adding headers conceptually. I'm -1 on a
> proposal
> >> using ints for header names. I just don't think a global namespace with
> >> integer names is an acceptable end-user interface given that our goal
> is to
> >> encourage a federated ecosystem of header users. Not trying to be a
> pain in
> >> the ass, I just think it would be a bit of a wart.
> >>
> >> -Jay
> >>
> >> On Thu, Feb 9, 2017 at 12:12 AM, Michael Pearce <Michael.Pearce@ig.com>
> >> wrote:
> >>
> >>> As we'd like to start work on this to make it into the next release in
> a
> >>> few months.
> >>>
> >>> If we were to put this to the vote today who would actually give this
> a -1
> >>> vote?
> >>>
> >>> Based on the number of identified usecases (some stronger than others)
> >>>
> >>> Also if we were to keep with int keys based on all the discussion
> thread
> >>> for the pros vs cons who would give a -1 as it stands? Eg anyone
> believe it
> >>> simply wouldn't work.
> >>>
> >>> If not so we make progress would it be sensible to put this to vote?
> So we
> >>> can start work?
> >>>
> >>> Cheers
> >>> Mike
> >>>
> >>>
> >>> Sent using OWA for iPhone
> >>> ________________________________________
> >>> From: Jeroen van Disseldorp <jeroen@axual.io>
> >>> Sent: Wednesday, February 1, 2017 2:43:45 PM
> >>> To: dev@kafka.apache.org
> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>>
> >>> I'm lagging a bit here too, but want to respond to the usefulness of
> >>> this KIP for two use cases: bidirectional mirroring and smart cluster
> >>> switches.
> >>>
> >>> Bidirectional mirroring
> >>>
> >>> For a Dutch bank we have currently implemented bidirectional mirroring
> >>> using our own headers, which we encode in the Value using a custom
> >>> serializer. It encodes just one byte extra, containing a
> "copy/original"
> >>> bit. When applications serialize an avro object, the bit is unset by
> >>> default. Our mirroring mechanism subscribes to all topics, checks the
> >>> bit, and when unset the bit is set and the message is forwarded to the
> >>> other DC.
> >>>
> >>> The overhead of this is pretty low (1 byte) and it's cheap, since the
> >>> mirroring mechanism does not need to fully deserialize the value, but
> >>> just check+flip one bit in the byte[] value. On the receiving end, the
> >>> mirroring mechanism sees the message for the second time, but notices
> >>> the message is a copy already and ignores it.
> >>>
> >>> The mechanism has been in production since last year and it works
> really
> >>> well. Its downside however is the incompatibility with avro serialized
> >>> values, as the whole organization is mandated to use our own custom
> >>> serde and we were forced to create a wrapper client library around the
> >>> default kafka client. The callback we offer to applications is very
> >>> similar to that proposed in the KIP, as every message our users get
> >>> consists of <Metadata, K, V>.
> >>>
> >>> Additionally our header contains a message id and serialization
> >>> timestamp (invented before 0.10.0.0). These fields could also be done
> >>> away with when this KIP is implemented.
> >>>
> >>> Smart cluster switches
> >>>
> >>> Another case we are considering to use headers for is enabling "smart
> >>> cluster switches" for consumers and producers in a way that would avoid
> >>> duplicates and/or missed messages in most cases. This involves encoding
> >>> a producer id (app name or id), producer instance id (uuid generated
> >>> upon producer's start) and sequence number in every message. We expect
> >>> to use one sequence per partition.
> >>>
> >>> These fields enable our consumers to keep track of which messages were
> >>> seen without relying on offsets, which is a cluster-bound value.
> >>> Consumers keep track of seen messages by keeping a small administration
> >>> of ranges of sequence numbers per producer instance and partition.
> Since
> >>> sequence numbers are always increasing per producer and partition, the
> >>> memory consumption of such administration is pretty low, while being
> >>> meaningful  across clusters. After every poll() we can commit the
> >>> administration to a topic in much the same way as we do with offsets.
> >>> The difference is that this topic can be mirrored to the other cluster
> >>> and upon a switch of consumer, all instances can use the topic to
> >>> initialize themselves and skip all messages already processed.
> >>>
> >>> Hope these views help in the upcoming decision on this KIP.
> >>>
> >>> Regards
> >>>    Jeroen
> >>>
> >>>
> >>>
> >>> On 09/01/2017 20:40, Gwen Shapira wrote:
> >>> > Mirroring + ClusterIDs:
> >>> >
> >>> > Kafka clusters already have an automatically generated ID that is
> >>> > exposed to clients:
> >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 78%3A+Cluster+Id
> >>> >
> >>> > It sounds trivial to get MirrorMaker's consumer to grab this ID
> (which
> >>> > always exists) from the source cluster and add it to the message
> >>> > headers.
> >>> >
> >>> > Obviously adding stuff to the message makes it larger so this should
> >>> > be a feature you should be able to turn on/off depending on the
> >>> > replication requirements. If you need the feature - you pay the
> price.
> >>> > I don't think there is an immediate need to put an effort into
> >>> > shrinking the Cluster ID, but others can comment on that.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <jay@confluent.io> wrote:
> >>> >> Hey Michael,
> >>> >>
> >>> >> Sorry for lagging, I've been out for the holiday break. What would
> help
> >>> me
> >>> >> would be to get to a consolidated list of use cases we've worked
> out and
> >>> >> think add real value. I think getting down to the details really
> matters
> >>> >> for these.
> >>> >>
> >>> >> The two lists I'm aware of are the one in the KIP and the one you
> gave
> >>> in
> >>> >> this email thread. Both seem to be more at the sketch level. Let me
> >>> review
> >>> >> the use cases and see if I have it right. If so let's get it in the
> >>> design
> >>> >> proposal.
> >>> >>
> >>> >> Your use cases:
> >>> >>
> >>> >>     1. Tracing IDs -- I get this use case and think I see how it
> would
> >>> work.
> >>> >>     I think because the tracing is from a third party system there
> is
> >>> some
> >>> >>     value to having a header versus a custom format to make this
> work
> >>> fully
> >>> >>     transparently.
> >>> >>     2. Mirroring -- I'm a bit unsure how this will work, I'll
> explain my
> >>> >>     thinking below
> >>> >>     3. End-to-end encryption -- I think what you are saying is that
> if
> >>> you
> >>> >>     want "routing' fields that are accessible in transit without
> >>> decrypting you
> >>> >>     need a scheme to leave this decrypted.I think the idea is to
> avoid
> >>> >>     decrypting to get a header, but I'm not sure if I get how this
> is
> >>> better
> >>> >>     than a custom message format that doesn't encrypt the things you
> >>> don't want
> >>> >>     encrypted since you have to be aware of the scheme anyway.
> >>> >>     4. EOS -- I'm not convinced this is a good approach for EOS.
> >>> >>
> >>> >> Linked from KIP:
> >>> >>
> >>> >>     1. Schema-Registry-aware avro serialization -- not clear how
> this is
> >>> >>     better than keeping the schema id in the value
> >>> >>     2. Polyglot kafka topics -- marginally better than a container
> >>> format,
> >>> >>     but of course you still have to all agree on the the header and
> >>> >>     header=>serialization mapping so not sure if it's really better
> >>> than a
> >>> >>     container format. Also not wild about polyglot topics :-)
> >>> >>     3. End-to-End routing trail including mirroring --- I'd put this
> >>> under
> >>> >>     tracing ids
> >>> >>     4. Cross-service process tracing -- ditto
> >>> >>     5. Signing -- I like this use case but it seems to depend on a
> >>> feature
> >>> >>     we don't have and aren't proposing adding, right?
> >>> >>     6. Per-record custom compression -- why do you want this?
> >>> compressing
> >>> >>     per message won't work well compared to batch compression,
> right?
> >>> >>     7. Large Message Support -- I think we can agree that although
> this
> >>> >>     might work it's clearly not the right way to support large
> messages
> >>> in Kafka
> >>> >>
> >>> >> Matthias
> >>> >>
> >>> >>     1. Control messages --- Matthias, not sure if I get the
> motivation
> >>> in
> >>> >>     streams for this. If we did want to do it, wouldn't it be
> easier to
> >>> >>     implement and by allowing users to send control messages (in the
> >>> EOS sense)
> >>> >>     and adding a consumer config to allow either passing these on or
> >>> >>     suppressing them?
> >>> >>
> >>> >> So I if I were to summarize I'd list the use cases as follows:
> >>> >>
> >>> >>     - Much better with headers: tracing IDs
> >>> >>     - Somewhat better: end-to-end encryption, polyglot topics
> >>> >>     - Don't buy: Mirroring (though perhaps could be made to work
> with
> >>> >>     changes to the proposal), EOS, large messages, compression
> >>> >>     - Not sure: Control messages
> >>> >>
> >>> >> Does this match where other people are at? From my point of view for
> >>> >> something like this where we are changing the core data model we
> should
> >>> >> ideally get a handful of use cases to the point where they are
> really
> >>> well
> >>> >> thought out and we know they can use the feature. I think we have
> one
> >>> solid
> >>> >> one with tracing IDs, and I think maybe mirroring could be another
> with
> >>> >> some work. I'm less sure about the others.
> >>> >>
> >>> >> Bi-directional Mirroring
> >>> >> I think bidirectional mirroring could potentially become a
> compelling
> >>> use
> >>> >> case if we an find a way to make it work. If I understand it, the
> >>> proposal
> >>> >> is to add a header like "origin-cluster" with the cluster id of the
> >>> Kafka
> >>> >> cluster. Then modify the mirroring tool to disallow mirroring a
> message
> >>> >> back to it's own origin cluster. This basic idea is definitely
> workable
> >>> and
> >>> >> useful, but can we really get to a good implementation using
> headers?
> >>> >> I imagine the approach would be to add this header in the producer
> if
> >>> some
> >>> >> config is set. There are two problems:
> >>> >>
> >>> >>     - You add a pretty big honking cluster ID to each message. For
> small
> >>> >>     messages this maybe larger than the contents itself.
> >>> >>     - What happens if someone somewhere forgets to set that config
> on
> >>> some
> >>> >>     client somewhere?
> >>> >>
> >>> >> If someone forgets to set the config in some client somewhere I
> guess
> >>> you
> >>> >> have various options in mirror maker:
> >>> >>
> >>> >>     1. drop the data (pretty bad, now your data is out of sync)
> >>> >>     2. crash (pretty bad--how do you recover)
> >>> >>     3. insert some default (pretty bad since would also lead to data
> >>> loss if
> >>> >>     that's not right)
> >>> >>
> >>> >> Basically although I think a cluster tag could be a good way to do
> >>> >> bi-directional mirroring, I'm not sure this naive approach I
> described
> >>> >> would get us to a usable implementation. This could be fixable. I
> think
> >>> if
> >>> >> there was some scheme to check required headers on the server that
> could
> >>> >> alleviate the missing header problem. Not really sure what to do
> about
> >>> the
> >>> >> message bloat problem, but maybe there could be some way to shrink
> the
> >>> >> cluster id?
> >>> >>
> >>> >> Cheers,
> >>> >>
> >>> >> -Jay
> >>> >>
> >>> >>
> >>> >> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <
> Michael.Pearce@ig.com
> >>> >
> >>> >> wrote:
> >>> >>
> >>> >>> @Jay, Ismael and a few others.
> >>> >>>
> >>> >>> Based on the discussion on kip-87, and the additional use case by
> >>> Matthias.
> >>> >>>
> >>> >>> Just for clarity do we now have consensus on the earlier points (1)
> >>> >>> headers are useful (2) headers should be native in Kafka?
> >>> >>>
> >>> >>> Cheers
> >>> >>> Mike
> >>> >>> ________________________________________
> >>> >>> From: radai <radai.rosenblatt@gmail.com>
> >>> >>> Sent: Monday, December 19, 2016 5:01 PM
> >>> >>> To: dev@kafka.apache.org
> >>> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>> >>>
> >>> >>> having said that im perfectly fine with something like a "no
> payload"
> >>> flag
> >>> >>> (as opposed to null payload), which would mean such a msg is not
> >>> returned
> >>> >>> from poll() unless the user code specifically "opts in" and would
> serve
> >>> >>> only to carry headers (making it a control message)
> >>> >>>
> >>> >>> On Mon, Dec 19, 2016 at 8:58 AM, radai <radai.rosenblatt@gmail.com
> >
> >>> wrote:
> >>> >>>
> >>> >>>> if all you want is to tag "end of stream" wouldnt a header on the
> last
> >>> >>> msg
> >>> >>>> in the stream be enough?
> >>> >>>>
> >>> >>>> On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
> >>> matthias@confluent.io>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>> The details about headers for control messages are still to
> define.
> >>> But
> >>> >>>>> yes, the idea is to have some common default behavior that
> clients
> >>> would
> >>> >>>>> need to implement.
> >>> >>>>>
> >>> >>>>> The point is, that "regular headers" add meta data to regular
> >>> messages.
> >>> >>>>> Thus, those messages will be returned to the user via .poll().
> And
> >>> after
> >>> >>>>> the message is received the user can check if meta data is
> present
> >>> and
> >>> >>>>> read it.
> >>> >>>>>
> >>> >>>>> For control messages, we do not want those to pop up via .poll()
> as
> >>> >>>>> those are no regular messages. A client would need to opt-in to
> see
> >>> >>>>> those messages (either via poll() or maybe a callback). Thus, we
> need
> >>> >>>>> some special (standardized) header IDs that indicate control
> messages
> >>> >>>>> that should not be returned to the user via poll() by default.
> >>> >>>>>
> >>> >>>>>
> >>> >>>>> -Matthias
> >>> >>>>>
> >>> >>>>> On 12/17/16 9:37 PM, Roger Hoover wrote:
> >>> >>>>>> Matthias,
> >>> >>>>>>
> >>> >>>>>> Thanks for your input.  I'm +1 on control messages as they seem
> to
> >>> be
> >>> >>>>> the
> >>> >>>>>> simplest way to implement watermarks (
> >>> >>>>>> https://www.oreilly.com/ideas/the-world-beyond-batch-
> streaming-102
> >>> ),
> >>> >>> a
> >>> >>>>>> feature that would add a lot of value to Kafka Streams IMHO.
> >>> >>>>>>
> >>> >>>>>> Your argument that the control-message indicator belongs in the
> >>> >>>>> client-only
> >>> >>>>>> section of the record format make sense.  Just to make sure I
> >>> >>>>> understand,
> >>> >>>>>> are you suggesting that control messages would be indicated by a
> >>> >>>>> standard
> >>> >>>>>> reserved header?  By standard, I mean that ALL Kafka consumers
> would
> >>> >>>>> know
> >>> >>>>>> to handle these messages differently (possibly just ignoring
> them).
> >>> >>>>> This
> >>> >>>>>> would need to be added to the specification of the consumer
> protocol
> >>> >>> so
> >>> >>>>>> that all Kafka clients implement it, right?  I think it's a good
> >>> idea
> >>> >>>>> but
> >>> >>>>>> just checking.
> >>> >>>>>>
> >>> >>>>>> Cheers,
> >>> >>>>>>
> >>> >>>>>> Roger
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
> >>> >>> matthias@confluent.io
> >>> >>>>>> wrote:
> >>> >>>>>>
> >>> >>>>>>> Yes and no. I did overload the term "control message".
> >>> >>>>>>>
> >>> >>>>>>> EOS control messages are for client-broker communication and
> thus
> >>> >>> never
> >>> >>>>>>> exposed to any application. And I think this is a good design
> >>> because
> >>> >>>>>>> broker needs to understand those control messages. Thus, this
> >>> should
> >>> >>> be
> >>> >>>>>>> a protocol change.
> >>> >>>>>>>
> >>> >>>>>>> The type of control messages I have in mind are for
> client-client
> >>> >>>>>>> (application-application) communication and the broker is
> agnostic
> >>> to
> >>> >>>>>>> them. Thus, it should not be a protocol change.
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> -Matthias
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> On 12/14/16 9:42 AM, radai wrote:
> >>> >>>>>>>> arent control messages getting pushed as their own top level
> >>> >>> protocol
> >>> >>>>>>>> change (and a fairly massive one) for the transactions KIP ?
> >>> >>>>>>>>
> >>> >>>>>>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
> >>> >>>>> matthias@confluent.io>
> >>> >>>>>>>> wrote:
> >>> >>>>>>>>
> >>> >>>>>>>>> Hi,
> >>> >>>>>>>>>
> >>> >>>>>>>>> I want to add a completely new angle to this discussion. For
> >>> this,
> >>> >>> I
> >>> >>>>>>>>> want to propose an extension for the headers feature that
> enables
> >>> >>> new
> >>> >>>>>>>>> uses cases -- and those new use cases might convince people
> to
> >>> >>>>> support
> >>> >>>>>>>>> headers (of course including the larger scoped proposal).
> >>> >>>>>>>>>
> >>> >>>>>>>>> Extended Proposal:
> >>> >>>>>>>>>
> >>> >>>>>>>>> Allow messages with a certain header key to be special
> "control
> >>> >>>>>>>>> messages" (w/ o w/o payload) that are not exposed to an
> >>> application
> >>> >>>>> via
> >>> >>>>>>>>> .poll().
> >>> >>>>>>>>>
> >>> >>>>>>>>> Thus, a consumer client would automatically skip over those
> >>> >>>>> messages. If
> >>> >>>>>>>>> an application knows about embedded control messages, it can
> >>> "sing
> >>> >>>>> up"
> >>> >>>>>>>>> to those messages by the consumer client and either get a
> >>> callback
> >>> >>> or
> >>> >>>>>>>>> the consumer auto-drop for this messages gets disabled
> (allowing
> >>> to
> >>> >>>>>>>>> consumer those messages via poll()).
> >>> >>>>>>>>>
> >>> >>>>>>>>> (The details need further considerations/discussion. I just
> want
> >>> to
> >>> >>>>>>>>> sketch the main idea.)
> >>> >>>>>>>>>
> >>> >>>>>>>>> Usage:
> >>> >>>>>>>>>
> >>> >>>>>>>>> There is a shared topic (ie, used by multiple applications)
> and a
> >>> >>>>>>>>> producer application wants to embed a special message in the
> >>> topic
> >>> >>>>> for a
> >>> >>>>>>>>> dedicated consumer application. Because only one application
> will
> >>> >>>>>>>>> understand this message, it cannot be a regular message as
> this
> >>> >>> would
> >>> >>>>>>>>> break all applications that do not understand this message.
> The
> >>> >>>>> producer
> >>> >>>>>>>>> application would set a special metadata key and no consumer
> >>> >>>>> application
> >>> >>>>>>>>> would see this control message by default because they did
> not
> >>> >>> enable
> >>> >>>>>>>>> their consumer client to return this message in poll() (and
> the
> >>> >>>>> client
> >>> >>>>>>>>> would just drop this message with special metadata key).
> Only the
> >>> >>>>> single
> >>> >>>>>>>>> application that should receive this message, will subscribe
> to
> >>> >>> this
> >>> >>>>>>>>> message on its consumer client and process it.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Concrete Use Case: Kafka Streams
> >>> >>>>>>>>>
> >>> >>>>>>>>> In Kafka Streams, we would like to propagate "control
> messages"
> >>> >>> from
> >>> >>>>>>>>> subtopology to subtopology. There are multiple scenarios for
> >>> which
> >>> >>>>> this
> >>> >>>>>>>>> would be useful. For example, currently we do not guarantee a
> >>> >>>>>>>>> "consistent shutdown" of an application. By this, I mean that
> >>> input
> >>> >>>>>>>>> records might not be completely processed by the whole
> topology
> >>> >>>>> because
> >>> >>>>>>>>> the application shutdown happens "in between" and an
> intermediate
> >>> >>>>> result
> >>> >>>>>>>>> topic gets "stock" in an intermediate topic. Thus, a user
> would
> >>> see
> >>> >>>>> an
> >>> >>>>>>>>> committed offset of the source topic of the application, but
> no
> >>> >>>>>>>>> corresponding result record in the output topic.
> >>> >>>>>>>>>
> >>> >>>>>>>>> Having "shutdown markers" would allow us, to first stop the
> >>> >>> upstream
> >>> >>>>>>>>> subtopology and write this marker into the intermediate
> topic and
> >>> >>> the
> >>> >>>>>>>>> downstream subtopology would only shut down itself after is
> sees
> >>> >>> the
> >>> >>>>>>>>> "shutdown marker". Thus, we can guarantee on shutdown, that
> no
> >>> >>>>>>>>> "in-flight" messages got stuck in intermediate topics.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> A similar usage would be for KIP-95 (Incremental Batch
> >>> Processing).
> >>> >>>>>>>>> There was a discussion about the proposed metadata topic,
> and we
> >>> >>>>> could
> >>> >>>>>>>>> avoid this metadata topic if we would have "control
> messages".
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Right now, we cannot insert an "application control message"
> >>> >>> because
> >>> >>>>>>>>> Kafka Streams does not own all topics it read/writes and thus
> >>> might
> >>> >>>>>>>>> break other consumer application (as described above) if we
> >>> inject
> >>> >>>>>>>>> random messages that are not understood by other apps.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Of course, one can work around "embedded control messaged" by
> >>> using
> >>> >>>>> an
> >>> >>>>>>>>> additional topic to propagate control messaged between
> >>> application
> >>> >>>>> (as
> >>> >>>>>>>>> suggestion in KIP-95 via a metadata topic for Kafka
> Streams). But
> >>> >>>>> there
> >>> >>>>>>>>> are major concerns about adding this metadata topic in the
> KIP
> >>> and
> >>> >>>>> this
> >>> >>>>>>>>> shows that other application that need a similar pattern
> might
> >>> >>> profit
> >>> >>>>>>>>> from topic embedded "control messages", too.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> One last important consideration: those "control messages"
> are
> >>> used
> >>> >>>>> for
> >>> >>>>>>>>> client to client communication and are not understood by the
> >>> >>> broker.
> >>> >>>>>>>>> Thus, those messages should not be enabled within the message
> >>> >>> format
> >>> >>>>>>>>> (c.f. tombstone flag -- KIP-87). However, "client land"
> record
> >>> >>>>> headers
> >>> >>>>>>>>> would be a nice way to implement them. Because KIP-82 did
> >>> consider
> >>> >>>>> key
> >>> >>>>>>>>> namespaces for metatdata keys, this extension should not be
> an
> >>> own
> >>> >>>>> KIP
> >>> >>>>>>>>> but should be included in KIP-82 to reserve a namespace for
> >>> >>> "control
> >>> >>>>>>>>> message" in the first place.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Sorry for the long email... Looking forward to your feedback.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> -Matthias
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
> >>> >>>>>>>>>> Hi Jun
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 100) each time a transaction exits a jvm for a remote system
> >>> >>>>> (HTTP/JMS/
> >>> >>>>>>>>> Hopefully one day kafka) the APM tools stich in a unique id
> >>> >>> (though I
> >>> >>>>>>>>> believe it contains the end2end uuid embedded in this id), on
> >>> >>>>> receiving
> >>> >>>>>>> the
> >>> >>>>>>>>> message at the receiving JVM the apm code takes this out, and
> >>> >>>>> continues
> >>> >>>>>>> its
> >>> >>>>>>>>> tracing on the that new thread. Both JVM's (and other
> languages
> >>> the
> >>> >>>>> APM
> >>> >>>>>>>>> tool supports) send this data async back to the central
> >>> controllers
> >>> >>>>>>> where
> >>> >>>>>>>>> the stiching togeather occurs. For this they need some header
> >>> space
> >>> >>>>> for
> >>> >>>>>>>>> them to put this id.
> >>> >>>>>>>>>> 101) Yes indeed we have a business transaction Id in the
> >>> payload.
> >>> >>>>>>> Though
> >>> >>>>>>>>> this is a system level tracing, that we need to have marry
> up.
> >>> Also
> >>> >>>>> as
> >>> >>>>>>> per
> >>> >>>>>>>>> note on end2end encryption we'd be unable to prove the flow
> if
> >>> the
> >>> >>>>>>> payload
> >>> >>>>>>>>> is encrypted as we'd not have access to this at certain
> points of
> >>> >>> the
> >>> >>>>>>> flow
> >>> >>>>>>>>> through the infrastructure/platform.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 103) As said we use this mechanism in IG very successfully,
> as
> >>> >>>>> stated
> >>> >>>>>>>>> per key we guarantee the transaction producing app to handle
> the
> >>> >>>>>>>>> transaction of a key at one DC unless at point of critical
> >>> failure
> >>> >>>>>>> where we
> >>> >>>>>>>>> have to flip processing to another. We care about key
> ordering.
> >>> >>>>>>>>>> I disagree on the offset comment for the partition solution
> >>> unless
> >>> >>>>> you
> >>> >>>>>>>>> do full ISR, or expensive full XA transactions even with
> >>> partitions
> >>> >>>>> you
> >>> >>>>>>>>> cannot fully guarantee offsets would match.
> >>> >>>>>>>>>> 105) Very much so, I need to have access at the platform
> level
> >>> to
> >>> >>>>> the
> >>> >>>>>>>>> other meta data all mentioned, without having to need to have
> >>> >>> access
> >>> >>>>> to
> >>> >>>>>>> the
> >>> >>>>>>>>> encryption keys of the payload.
> >>> >>>>>>>>>> 106)
> >>> >>>>>>>>>> Techincally yes for AZ/Region/Cluster, but then we'd need to
> >>> have
> >>> >>> a
> >>> >>>>>>>>> global producerId register which would be very hard to
> >>> >>>>> enforce/ensure is
> >>> >>>>>>>>> current and correct, just to understand the message origins
> of
> >>> its
> >>> >>>>>>>>> region/az/cluster for routing.
> >>> >>>>>>>>>> The client wrapper version, producerId can be the same, as
> >>> >>> obviously
> >>> >>>>>>> the
> >>> >>>>>>>>> producer could upgrade its wrapper, as such we need to know
> what
> >>> >>>>> wrapper
> >>> >>>>>>>>> version the message is created with.
> >>> >>>>>>>>>> Likewise the IP address, as stated we can have our producer
> >>> move,
> >>> >>>>> where
> >>> >>>>>>>>> its IP would change.
> >>> >>>>>>>>>> 107)
> >>> >>>>>>>>>> UUID is set on the message by interceptors before actual
> >>> producer
> >>> >>>>>>>>> transport send. This is for platform level message dedupe
> >>> >>> guarantee,
> >>> >>>>> the
> >>> >>>>>>>>> business payload should be agnostic to this. Please see
> >>> >>>>>>>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
> >>> >>>>> ection.html
> >>> >>>>>>>>> note this is not touching business payloads.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> On 06/12/2016, 18:22, "Jun Rao" <jun@confluent.io> wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      Hi, Michael,
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      Thanks for the reply. I find it very helpful.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      Data lineage:
> >>> >>>>>>>>>>      100. I'd like to understand the APM use case a bit
> more. It
> >>> >>>>> sounds
> >>> >>>>>>>>> like
> >>> >>>>>>>>>>      that those APM plugins can generate a transaction id
> that
> >>> we
> >>> >>>>> could
> >>> >>>>>>>>>>      potentially put in the header of every message. How
> would
> >>> you
> >>> >>>>>>>>> typically
> >>> >>>>>>>>>>      make use of such transaction ids? Are there other
> metadata
> >>> >>>>>>>>> associated with
> >>> >>>>>>>>>>      the transaction id and if so, how are they propagated
> >>> >>>>> downstream?
> >>> >>>>>>>>>>      101. For the finance use case, if the concept of
> >>> transaction
> >>> >>> is
> >>> >>>>>>>>> important,
> >>> >>>>>>>>>>      wouldn't it be typically included in the message
> payload
> >>> >>>>> instead of
> >>> >>>>>>>>> as an
> >>> >>>>>>>>>>      optional header field?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      102. The data lineage that Altas and Navigator support
> >>> seems
> >>> >>> to
> >>> >>>>> be
> >>> >>>>>>>>> at the
> >>> >>>>>>>>>>      dataset level, not per record level? So, not sure if
> per
> >>> >>> message
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>>      are relevant there.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      Mirroring:
> >>> >>>>>>>>>>      103. The benefit of using separate partitions is that
> it
> >>> >>>>>>> potentially
> >>> >>>>>>>>> makes
> >>> >>>>>>>>>>      it easy to preserve offsets during mirroring. This will
> >>> make
> >>> >>> it
> >>> >>>>>>>>> easier for
> >>> >>>>>>>>>>      consumer to switch clusters. Currently, the consumers
> can
> >>> >>> switch
> >>> >>>>>>>>> clusters
> >>> >>>>>>>>>>      by using the timestampToOffset() api, but it has to
> deal
> >>> with
> >>> >>>>>>>>> duplicates.
> >>> >>>>>>>>>>      Good point on the issue with log compact and I am not
> sure
> >>> how
> >>> >>>>> to
> >>> >>>>>>>>> address
> >>> >>>>>>>>>>      this. However, even if we mirror into the existing
> >>> partitions,
> >>> >>>>> the
> >>> >>>>>>>>> ordering
> >>> >>>>>>>>>>      for messages generated from different clusters seems
> >>> >>>>>>>>> non-deterministic
> >>> >>>>>>>>>>      anyway. So, it seems that the consumers already have to
> >>> deal
> >>> >>>>> with
> >>> >>>>>>>>> that? If
> >>> >>>>>>>>>>      a topic is compacted, does that mean which messages are
> >>> >>>>> preserved
> >>> >>>>>>> is
> >>> >>>>>>>>> also
> >>> >>>>>>>>>>      non-deterministic across clusters?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      104. Good point on partition key.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      End-to-end encryption:
> >>> >>>>>>>>>>      105. So, it seems end-to-end encryption is useful. Are
> >>> headers
> >>> >>>>>>>>> useful there?
> >>> >>>>>>>>>>      Auditing:
> >>> >>>>>>>>>>      106. It seems other than the UUID, all other metadata
> are
> >>> per
> >>> >>>>>>>>> producer?
> >>> >>>>>>>>>>      EOS:
> >>> >>>>>>>>>>      107. How are those UUIDs generated? I am not sure if
> they
> >>> can
> >>> >>> be
> >>> >>>>>>>>> generated
> >>> >>>>>>>>>>      in the producer library. An application may send
> messages
> >>> >>>>> through a
> >>> >>>>>>>>> load
> >>> >>>>>>>>>>      balancer and on retry, the same message could be
> routed to
> >>> a
> >>> >>>>>>>>> different
> >>> >>>>>>>>>>      producer instance. So, it seems that the application
> has to
> >>> >>>>>>> generate
> >>> >>>>>>>>> the
> >>> >>>>>>>>>>      UUIDs. In that case, shouldn't the application just
> put the
> >>> >>>>> UUID in
> >>> >>>>>>>>> the
> >>> >>>>>>>>>>      payload?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      Thanks,
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      Jun
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
> >>> >>>>>>>>> Michael.Pearce@ig.com>
> >>> >>>>>>>>>>      wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>      > Hi Jun.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Per Transaction Tracing / Data Lineage.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > As Stated in the KIP this has the first use case of
> how
> >>> many
> >>> >>>>> APM
> >>> >>>>>>>>> tools now
> >>> >>>>>>>>>>      > work.
> >>> >>>>>>>>>>      > I would find it impossible for any one to argue this
> is
> >>> not
> >>> >>>>>>>>> important or a
> >>> >>>>>>>>>>      > niche market as it has its own gartner report for
> this
> >>> >>> space.
> >>> >>>>>>> Such
> >>> >>>>>>>>>>      > companies as Appdynamics, NewRelic, Dynatrace,
> Hawqular
> >>> are
> >>> >>>>> but a
> >>> >>>>>>>>> few.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Likewise these APM tools can help very rapidly track
> down
> >>> >>>>> issues
> >>> >>>>>>>>> and
> >>> >>>>>>>>>>      > automatically capture metrics, perform actions based
> on
> >>> >>>>>>> unexpected
> >>> >>>>>>>>> behavior
> >>> >>>>>>>>>>      > to auto recover services.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Before mentioning looking at aggregated stats, in
> these
> >>> >>> cases
> >>> >>>>>>> where
> >>> >>>>>>>>>>      > actually on critical flows we cannot afford to have
> >>> >>> aggregated
> >>> >>>>>>>>> rolled up
> >>> >>>>>>>>>>      > stats only.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > With the APM tool we use its actually able to detect
> a
> >>> >>> single
> >>> >>>>>>>>> transaction
> >>> >>>>>>>>>>      > failure and capture the thread traces in the JVM
> where it
> >>> >>>>> failed
> >>> >>>>>>>>> and
> >>> >>>>>>>>>>      > everything for us, to the point it sends us alerts
> where
> >>> we
> >>> >>>>> have
> >>> >>>>>>>>> this
> >>> >>>>>>>>>>      > giving the line number of the code that caused it,
> the
> >>> >>>>>>> transaction
> >>> >>>>>>>>> trace
> >>> >>>>>>>>>>      > through all the services and endpoints (supported)
> upto
> >>> the
> >>> >>>>> point
> >>> >>>>>>>>> of
> >>> >>>>>>>>>>      > failure, it can also capture the data in and out (so
> we
> >>> can
> >>> >>>>>>>>> replay).
> >>> >>>>>>>>>>      > Because atm Kafka doesn't support us being able to
> stich
> >>> in
> >>> >>>>> these
> >>> >>>>>>>>> tracing
> >>> >>>>>>>>>>      > transaction ids natively, we cannot get these
> benefits as
> >>> >>>>> such is
> >>> >>>>>>>>> limiting
> >>> >>>>>>>>>>      > our ability support apps and monitor them to the same
> >>> >>>>> standards
> >>> >>>>>>> we
> >>> >>>>>>>>> come to
> >>> >>>>>>>>>>      > expect when on a kafka flow.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > This actually ties in with Data Lineage, as the same
> >>> tracing
> >>> >>>>> can
> >>> >>>>>>>>> be used
> >>> >>>>>>>>>>      > to back stich this. Essentially many times due to the
> >>> sums
> >>> >>> of
> >>> >>>>>>> money
> >>> >>>>>>>>>>      > involved there are disputes, and typically as a
> financial
> >>> >>>>>>>>> institute the
> >>> >>>>>>>>>>      > easiest and cleanest way to prove when disputes
> arise is
> >>> to
> >>> >>>>>>>>> present the
> >>> >>>>>>>>>>      > actual flow and processes involved in a transaction.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Likewise as Hadoop matures its evident this case is
> >>> >>>>> important, as
> >>> >>>>>>>>> tools
> >>> >>>>>>>>>>      > such as Atlas (Hortonworks led) and Navigator
> (cloudera
> >>> led)
> >>> >>>>> are
> >>> >>>>>>>>> evident
> >>> >>>>>>>>>>      > also I believe the importance here is very much NOT
> just
> >>> a
> >>> >>>>>>>>> financial issue.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > From a MDM point of view any company wanting to care
> >>> about
> >>> >>>>> Data
> >>> >>>>>>>>> Quality
> >>> >>>>>>>>>>      > and Data Governance - Data Lineage is a key piece in
> this
> >>> >>>>> puzzle.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > RE Mirroring,
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > As per the KIP in-fact this is exactly what we do re
> >>> cluster
> >>> >>>>> id,
> >>> >>>>>>>>> to mirror
> >>> >>>>>>>>>>      > a network of clusters between AZ's / Regions. We
> know a
> >>> >>>>>>>>> transaction for a
> >>> >>>>>>>>>>      > key will be done within a  AZ/Region, as such we
> know the
> >>> >>>>> write
> >>> >>>>>>> to
> >>> >>>>>>>>> kafka
> >>> >>>>>>>>>>      > would be ordered per key. But we need eventual view
> of
> >>> that
> >>> >>>>>>> across
> >>> >>>>>>>>> in our
> >>> >>>>>>>>>>      > other regions/az's. When we have complete AZ or
> Region
> >>> >>>>> failure we
> >>> >>>>>>>>> know
> >>> >>>>>>>>>>      > there will be a brief interruption whilst those
> >>> transactions
> >>> >>>>> are
> >>> >>>>>>>>> moved to
> >>> >>>>>>>>>>      > another region but we expect after it to continue.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > As mentioned having separate Partions to do this
> starts
> >>> to
> >>> >>> get
> >>> >>>>>>>>>>      > ugly/complicated for us:
> >>> >>>>>>>>>>      > how would I do compaction where a key is in two
> >>> partitions?
> >>> >>>>>>>>>>      > How do we balance consumers so where multiple
> partitions
> >>> >>> with
> >>> >>>>> the
> >>> >>>>>>>>> same key
> >>> >>>>>>>>>>      > goto the same consumer
> >>> >>>>>>>>>>      > What do you do if cluster 1 has 5 partitions but
> cluster
> >>> 20
> >>> >>>>> has
> >>> >>>>>>> 10
> >>> >>>>>>>>> because
> >>> >>>>>>>>>>      > its larger kit in our more core DC's, as such key to
> >>> >>> partition
> >>> >>>>>>>>> mappings for
> >>> >>>>>>>>>>      > consumers get even more complicated.
> >>> >>>>>>>>>>      > What do you do if we add or remove a complete region
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Where as simple mirror will work we just need to
> ensure
> >>> we
> >>> >>>>> don't
> >>> >>>>>>>>> have a
> >>> >>>>>>>>>>      > cycle which we can do with clusterId.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > We even have started to look at shortest path mirror
> >>> routing
> >>> >>>>>>> based
> >>> >>>>>>>>> on
> >>> >>>>>>>>>>      > clusterId, if we also had the region and az info on
> the
> >>> >>>>>>> originating
> >>> >>>>>>>>>>      > message, this we have not implemented but some ideas
> come
> >>> >>> from
> >>> >>>>>>>>> network
> >>> >>>>>>>>>>      > routing, and also the dispatcher router in apache
> qpid.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Also we need to have data perimeters e.g. certain
> data
> >>> >>> cannot
> >>> >>>>>>> leave
> >>> >>>>>>>>>>      > certain countries borders. We want this all
> automated so
> >>> >>> that
> >>> >>>>> at
> >>> >>>>>>>>> the
> >>> >>>>>>>>>>      > platform level without having to touch or look at the
> >>> >>> business
> >>> >>>>>>>>> data inside
> >>> >>>>>>>>>>      > we can have headers we can put tags into so that we
> can
> >>> >>> ensure
> >>> >>>>>>>>> this doesn't
> >>> >>>>>>>>>>      > occur when we mirror. (actually links in to data
> lineage
> >>> /
> >>> >>>>>>> tracing
> >>> >>>>>>>>> as again
> >>> >>>>>>>>>>      > we need to tag messages at a platform level) Examples
> >>> are we
> >>> >>>>> are
> >>> >>>>>>>>> not
> >>> >>>>>>>>>>      > allowed Private customer details to leave
> Switzerland,
> >>> yet
> >>> >>> we
> >>> >>>>>>> need
> >>> >>>>>>>>> those
> >>> >>>>>>>>>>      > systems integrated.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Lastly around mirroring we have a partionKey field,
> as
> >>> the
> >>> >>> key
> >>> >>>>>>>>> used for
> >>> >>>>>>>>>>      > portioning logic != compaction key all the time but
> we
> >>> want
> >>> >>> to
> >>> >>>>>>>>> preserve it
> >>> >>>>>>>>>>      > for when we mirror so that if source cluster
> partition
> >>> count
> >>> >>>>> !=
> >>> >>>>>>>>> destination
> >>> >>>>>>>>>>      > cluster partition count we can honour the same
> >>> partitioning
> >>> >>>>>>> logic.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > RE End 2 End encryption
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > As I believe mentioned just before, the solution you
> >>> mention
> >>> >>>>> just
> >>> >>>>>>>>> doesn't
> >>> >>>>>>>>>>      > cut the mustard these days with many regulators. An
> >>> >>> operations
> >>> >>>>>>>>> person with
> >>> >>>>>>>>>>      > access to the box should not be able to have access
> to
> >>> the
> >>> >>>>> data.
> >>> >>>>>>>>> Many now
> >>> >>>>>>>>>>      > actually impose quite literally the implementation
> >>> expected
> >>> >>>>> being
> >>> >>>>>>>>> end2end
> >>> >>>>>>>>>>      > encryption for certain data (Singapore for us is one
> >>> that I
> >>> >>> am
> >>> >>>>>>>>> most aware
> >>> >>>>>>>>>>      > of). In fact we're even now needing encrypt the data
> and
> >>> >>> store
> >>> >>>>>>> the
> >>> >>>>>>>>> keys in
> >>> >>>>>>>>>>      > HSM modules.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Likewise the performance penalty on encrypting
> >>> decrypting as
> >>> >>>>> you
> >>> >>>>>>>>> produce
> >>> >>>>>>>>>>      > over wire, then again encrypt decrypt as the data is
> >>> stored
> >>> >>> on
> >>> >>>>>>> the
> >>> >>>>>>>>> brokers
> >>> >>>>>>>>>>      > disks and back again, then again encrypted and
> decrypted
> >>> >>> back
> >>> >>>>>>> over
> >>> >>>>>>>>> the wire
> >>> >>>>>>>>>>      > each time for each consumer all adds up, ignoring
> this
> >>> >>>>> doubling
> >>> >>>>>>>>> with mirror
> >>> >>>>>>>>>>      > makers etc. simply encrypting the value once on
> write by
> >>> the
> >>> >>>>>>>>> client and
> >>> >>>>>>>>>>      > again decrypting on consume by the consumer is far
> more
> >>> >>>>>>>>> performant, but
> >>> >>>>>>>>>>      > then the routing and platform meta data needs to be
> >>> separate
> >>> >>>>>>> (thus
> >>> >>>>>>>>> headers)
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > RE Auditing:
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Our Auditing needs are:
> >>> >>>>>>>>>>      > Producer Id,
> >>> >>>>>>>>>>      > Origin Cluster Id that message first produced into
> >>> >>>>>>>>>>      > Origin AZ - agreed we can derive this if we have
> cluster
> >>> id,
> >>> >>>>> but
> >>> >>>>>>>>> it makes
> >>> >>>>>>>>>>      > resolving this for audit reporting a lot easier.
> >>> >>>>>>>>>>      > Origin Region - agreed we can derive this if we have
> >>> cluster
> >>> >>>>> id,
> >>> >>>>>>>>> but it
> >>> >>>>>>>>>>      > makes resolving this for audit reporting a lot
> easier.
> >>> >>>>>>>>>>      > Unique Message Identification (this is not the same
> as
> >>> >>>>>>> transaction
> >>> >>>>>>>>>>      > tracing) - note offset and partition are not the
> same, as
> >>> >>>>> when we
> >>> >>>>>>>>> mirror or
> >>> >>>>>>>>>>      > have for what ever system failure duplicate send,
> >>> >>>>>>>>>>      > Custom Client wrapper version (where organizations
> have
> >>> to
> >>> >>>>> wrap
> >>> >>>>>>>>> the kafka
> >>> >>>>>>>>>>      > client for added features) so we know what version
> of the
> >>> >>>>> wrapper
> >>> >>>>>>>>> is used
> >>> >>>>>>>>>>      > Producer IP address (in case of clients being in our
> >>> vm/open
> >>> >>>>>>> stack
> >>> >>>>>>>>> infra
> >>> >>>>>>>>>>      > where they can move around, producer id will stay the
> >>> same
> >>> >>> but
> >>> >>>>>>>>> this would
> >>> >>>>>>>>>>      > change)
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > RE Once and only once delivery case
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Using the same Message UUID for auditing we can
> achieve
> >>> this
> >>> >>>>>>> quite
> >>> >>>>>>>>> simply.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > As per how some other brokers do this (cough qpid,
> >>> artemis)
> >>> >>>>>>>>> message uuid
> >>> >>>>>>>>>>      > are used to dedupe where message is sent and
> produced but
> >>> >>> the
> >>> >>>>>>>>> client didn't
> >>> >>>>>>>>>>      > receive the ack, and there for replays the send, by
> >>> having a
> >>> >>>>>>>>> unique message
> >>> >>>>>>>>>>      > id per message, this can be filtered out, on
> consumers
> >>> where
> >>> >>>>>>>>> message
> >>> >>>>>>>>>>      > delivery may occur twice for what ever reasons a
> message
> >>> >>> uuid
> >>> >>>>> can
> >>> >>>>>>>>> be used
> >>> >>>>>>>>>>      > to remove duplicates being deliverd , like wise we
> can do
> >>> >>>>> this in
> >>> >>>>>>>>> the
> >>> >>>>>>>>>>      > mirrormakers so if we detect a dupe message we can
> avoid
> >>> >>>>>>>>> replicating it.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > Cheers
> >>> >>>>>>>>>>      > Mike
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > On 02/12/2016, 22:09, "Jun Rao" <jun@confluent.io>
> >>> wrote:
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >     Since this KIP affects message format, wire
> protocol,
> >>> >>>>> apis, I
> >>> >>>>>>>>> think
> >>> >>>>>>>>>>      > it's
> >>> >>>>>>>>>>      >     worth spending a bit more time to nail down the
> >>> concrete
> >>> >>>>> use
> >>> >>>>>>>>> cases. It
> >>> >>>>>>>>>>      >     would be bad if we add this feature, but when
> start
> >>> >>>>>>>>> implementing it
> >>> >>>>>>>>>>      > for say
> >>> >>>>>>>>>>      >     mirroring, we then realize that header is not the
> >>> best
> >>> >>>>>>>>> approach.
> >>> >>>>>>>>>>      > Initially,
> >>> >>>>>>>>>>      >     I thought I was convinced of the use cases of
> headers
> >>> >>> and
> >>> >>>>> was
> >>> >>>>>>>>> trying to
> >>> >>>>>>>>>>      >     write down a few use cases to convince others.
> That's
> >>> >>>>> when I
> >>> >>>>>>>>> became
> >>> >>>>>>>>>>      > less
> >>> >>>>>>>>>>      >     certain. For me to be convinced, I just want to
> see
> >>> two
> >>> >>>>>>> strong
> >>> >>>>>>>>> use
> >>> >>>>>>>>>>      > cases
> >>> >>>>>>>>>>      >     (instead of 10 maybe use cases) in the
> third-party
> >>> >>> space.
> >>> >>>>> The
> >>> >>>>>>>>> reason is
> >>> >>>>>>>>>>      >     that when we discussed the use cases within a
> >>> company,
> >>> >>>>> often
> >>> >>>>>>>>> it ends
> >>> >>>>>>>>>>      > with
> >>> >>>>>>>>>>      >     "we can't force everyone to use this standard
> since
> >>> we
> >>> >>> may
> >>> >>>>>>>>> have to
> >>> >>>>>>>>>>      >     integrate with third-party tools".
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >     At present, I am not sure why headers are useful
> for
> >>> >>>>> things
> >>> >>>>>>>>> like
> >>> >>>>>>>>>>      > schemaId
> >>> >>>>>>>>>>      >     or encryption. In order to do anything useful to
> the
> >>> >>>>> value,
> >>> >>>>>>>>> one needs
> >>> >>>>>>>>>>      > to
> >>> >>>>>>>>>>      >     know the schemaId or how data is encrypted, but
> >>> header
> >>> >>> is
> >>> >>>>>>>>> optional.
> >>> >>>>>>>>>>      > But, I
> >>> >>>>>>>>>>      >     can be convinced if someone (Radai, Sean, Todd?)
> >>> >>> provides
> >>> >>>>>>> more
> >>> >>>>>>>>> details
> >>> >>>>>>>>>>      > on
> >>> >>>>>>>>>>      >     the argument.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >     I am not very sure header is the best approach
> for
> >>> >>>>> mirroring
> >>> >>>>>>>>> either. If
> >>> >>>>>>>>>>      >     someone has thought about this more, I'd be
> happy to
> >>> >>> hear.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >     I can see the data lineage use case. I am just
> not
> >>> sure
> >>> >>>>> how
> >>> >>>>>>>>> widely
> >>> >>>>>>>>>>      >     applicable this is. If someone familiar with this
> >>> space
> >>> >>>>> can
> >>> >>>>>>>>> justify
> >>> >>>>>>>>>>      > this is
> >>> >>>>>>>>>>      >     a significant use case, say in the finance
> industry,
> >>> >>> this
> >>> >>>>>>>>> would be a
> >>> >>>>>>>>>>      > strong
> >>> >>>>>>>>>>      >     use case.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >     I can see the auditing use case. I am just not
> sure
> >>> if a
> >>> >>>>>>> native
> >>> >>>>>>>>>>      > producer id
> >>> >>>>>>>>>>      >     solves that problem. If there are additional
> metadata
> >>> >>>>> that's
> >>> >>>>>>>>> worth
> >>> >>>>>>>>>>      >     collecting but not covered by the producer id,
> that
> >>> >>> would
> >>> >>>>>>> make
> >>> >>>>>>>>> this a
> >>> >>>>>>>>>>      >     strong use case.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >     Thanks,
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >     Jun
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >     On Fri, Dec 2, 2016 at 1:41 PM, radai <
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>>      > wrote:
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >     > this KIP is about enabling headers, nothing
> more
> >>> >>> nothing
> >>> >>>>>>>>> less - so
> >>> >>>>>>>>>>      > no,
> >>> >>>>>>>>>>      >     > broker-side use of headers is not in the KIP
> scope.
> >>> >>>>>>>>>>      >     >
> >>> >>>>>>>>>>      >     > obviously though, once you have headers
> potential
> >>> use
> >>> >>>>> cases
> >>> >>>>>>>>> could
> >>> >>>>>>>>>>      > include
> >>> >>>>>>>>>>      >     > broker-side header-aware interceptors (which
> would
> >>> be
> >>> >>>>> the
> >>> >>>>>>>>> topic of
> >>> >>>>>>>>>>      > other
> >>> >>>>>>>>>>      >     > future KIPs).
> >>> >>>>>>>>>>      >     >
> >>> >>>>>>>>>>      >     > a trivially clear use case (to me) would be
> using
> >>> such
> >>> >>>>>>>>> broker-side
> >>> >>>>>>>>>>      >     > interceptors to enforce compliance with
> >>> organizational
> >>> >>>>>>>>> policies - it
> >>> >>>>>>>>>>      > would
> >>> >>>>>>>>>>      >     > make our SREs lives much easier if instead of
> >>> >>>>> retroactively
> >>> >>>>>>>>>>      > discovering
> >>> >>>>>>>>>>      >     > "rogue" topics/users those messages would have
> been
> >>> >>>>>>> rejected
> >>> >>>>>>>>>>      > up-front.
> >>> >>>>>>>>>>      >     >
> >>> >>>>>>>>>>      >     > the kafka broker code is lacking any such
> >>> >>> extensibility
> >>> >>>>>>>>> support
> >>> >>>>>>>>>>      > (beyond
> >>> >>>>>>>>>>      >     > maybe authorizer) which is why these use cases
> were
> >>> >>> left
> >>> >>>>>>> out
> >>> >>>>>>>>> of the
> >>> >>>>>>>>>>      > "case
> >>> >>>>>>>>>>      >     > for headers" doc - broker extensibility is a
> >>> separate
> >>> >>>>>>>>> discussion.
> >>> >>>>>>>>>>      >     >
> >>> >>>>>>>>>>      >     > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
> >>> >>>>>>>>> gwen@confluent.io>
> >>> >>>>>>>>>>      > wrote:
> >>> >>>>>>>>>>      >     >
> >>> >>>>>>>>>>      >     > > Woah, I wasn't aware this is something we'll
> do.
> >>> It
> >>> >>>>>>> wasn't
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>>      > KIP,
> >>> >>>>>>>>>>      >     > > right?
> >>> >>>>>>>>>>      >     > >
> >>> >>>>>>>>>>      >     > > I guess we could do it the same way ACLs
> >>> currently
> >>> >>>>> work.
> >>> >>>>>>>>>>      >     > > I had in mind something that will allow
> admins to
> >>> >>>>> apply
> >>> >>>>>>>>> rules to
> >>> >>>>>>>>>>      > the
> >>> >>>>>>>>>>      >     > > new create/delete/config topic APIs. So Todd
> can
> >>> >>>>> decide
> >>> >>>>>>> to
> >>> >>>>>>>>> reject
> >>> >>>>>>>>>>      >     > > "create topic" requests that ask for more
> than 40
> >>> >>>>>>>>> partitions, or
> >>> >>>>>>>>>>      >     > > require exactly 3 replicas, or no more than
> 50GB
> >>> >>>>>>> partition
> >>> >>>>>>>>> size,
> >>> >>>>>>>>>>      > etc.
> >>> >>>>>>>>>>      >     > >
> >>> >>>>>>>>>>      >     > > ACLs were added a bit ad-hoc, if we are
> planning
> >>> to
> >>> >>>>> apply
> >>> >>>>>>>>> more
> >>> >>>>>>>>>>      > rules
> >>> >>>>>>>>>>      >     > > to requests (and I think we should), we may
> want
> >>> a
> >>> >>> bit
> >>> >>>>>>>>> more generic
> >>> >>>>>>>>>>      >     > > design around that.
> >>> >>>>>>>>>>      >     > >
> >>> >>>>>>>>>>      >     > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>>      >     > wrote:
> >>> >>>>>>>>>>      >     > > > "wouldn't you be in the business of making
> sure
> >>> >>>>>>> everyone
> >>> >>>>>>>>> uses
> >>> >>>>>>>>>>      > them
> >>> >>>>>>>>>>      >     > > > properly?"
> >>> >>>>>>>>>>      >     > > >
> >>> >>>>>>>>>>      >     > > > thats where a broker-side plugin would come
> >>> handy
> >>> >>> -
> >>> >>>>> any
> >>> >>>>>>>>> incoming
> >>> >>>>>>>>>>      >     > message
> >>> >>>>>>>>>>      >     > > > that does not conform to org policy (read -
> >>> does
> >>> >>> not
> >>> >>>>>>>>> have the
> >>> >>>>>>>>>>      > proper
> >>> >>>>>>>>>>      >     > > > headers) gets thrown out (with an error
> >>> returned
> >>> >>> to
> >>> >>>>>>> user)
> >>> >>>>>>>>>>      >     > > >
> >>> >>>>>>>>>>      >     > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd
> Palino <
> >>> >>>>>>>>> tpalino@gmail.com>
> >>> >>>>>>>>>>      > wrote:
> >>> >>>>>>>>>>      >     > > >
> >>> >>>>>>>>>>      >     > > >> Come on, I've done at least 2 talks on
> this
> >>> one
> >>> >>> :)
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >> Producing counts to a topic is part of
> it, but
> >>> >>>>> that's
> >>> >>>>>>>>> only
> >>> >>>>>>>>>>      > part. So
> >>> >>>>>>>>>>      >     > you
> >>> >>>>>>>>>>      >     > > >> count you have 100 messages in topic A.
> When
> >>> you
> >>> >>>>>>> mirror
> >>> >>>>>>>>> topic A
> >>> >>>>>>>>>>      > to
> >>> >>>>>>>>>>      >     > > another
> >>> >>>>>>>>>>      >     > > >> cluster, you have 99 messages. Where was
> your
> >>> >>>>> problem?
> >>> >>>>>>>>> Or
> >>> >>>>>>>>>>      > worse, you
> >>> >>>>>>>>>>      >     > > have
> >>> >>>>>>>>>>      >     > > >> 100 messages, but one producer duplicated
> >>> >>> messages
> >>> >>>>> and
> >>> >>>>>>>>> another
> >>> >>>>>>>>>>      > one
> >>> >>>>>>>>>>      >     > lost
> >>> >>>>>>>>>>      >     > > >> messages. You need details about where the
> >>> >>> message
> >>> >>>>>>> came
> >>> >>>>>>>>> from in
> >>> >>>>>>>>>>      > order
> >>> >>>>>>>>>>      >     > to
> >>> >>>>>>>>>>      >     > > >> pinpoint problems when they happen. Source
> >>> >>> producer
> >>> >>>>>>>>> info, where
> >>> >>>>>>>>>>      > it was
> >>> >>>>>>>>>>      >     > > >> produced into your infrastructure, and
> when it
> >>> >>> was
> >>> >>>>>>>>> produced.
> >>> >>>>>>>>>>      > This
> >>> >>>>>>>>>>      >     > > requires
> >>> >>>>>>>>>>      >     > > >> you to add the information to the message.
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >> And yes, you still need to maintain your
> >>> clients.
> >>> >>>>> So
> >>> >>>>>>>>> maybe my
> >>> >>>>>>>>>>      > original
> >>> >>>>>>>>>>      >     > > >> example was not the best. My thoughts on
> not
> >>> >>>>> wanting
> >>> >>>>>>> to
> >>> >>>>>>>>> be
> >>> >>>>>>>>>>      > responsible
> >>> >>>>>>>>>>      >     > > for
> >>> >>>>>>>>>>      >     > > >> message formats stands, because that's
> very
> >>> much
> >>> >>>>>>>>> separate from
> >>> >>>>>>>>>>      > the
> >>> >>>>>>>>>>      >     > > client.
> >>> >>>>>>>>>>      >     > > >> As you know, we have our own internal
> client
> >>> >>>>> library
> >>> >>>>>>>>> that can
> >>> >>>>>>>>>>      > insert
> >>> >>>>>>>>>>      >     > the
> >>> >>>>>>>>>>      >     > > >> right headers, and right now inserts the
> right
> >>> >>>>> audit
> >>> >>>>>>>>>>      > information into
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> message fields. If they exist, and
> assuming
> >>> the
> >>> >>>>>>> message
> >>> >>>>>>>>> is Avro
> >>> >>>>>>>>>>      >     > encoded.
> >>> >>>>>>>>>>      >     > > >> What if someone wants to use JSON instead
> for
> >>> a
> >>> >>>>> good
> >>> >>>>>>>>> reason?
> >>> >>>>>>>>>>      > What if
> >>> >>>>>>>>>>      >     > > user X
> >>> >>>>>>>>>>      >     > > >> wants to encrypt messages, but user Y does
> >>> not?
> >>> >>>>>>>>> Maintaining the
> >>> >>>>>>>>>>      > client
> >>> >>>>>>>>>>      >     > > >> library is still much easier than
> maintaining
> >>> the
> >>> >>>>>>>>> message
> >>> >>>>>>>>>>      > formats.
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >> -Todd
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen
> Shapira <
> >>> >>>>>>>>> gwen@confluent.io
> >>> >>>>>>>>>>      > >
> >>> >>>>>>>>>>      >     > wrote:
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >> > Based on your last sentence, consider me
> >>> >>>>> convinced
> >>> >>>>>>> :)
> >>> >>>>>>>>>>      >     > > >> >
> >>> >>>>>>>>>>      >     > > >> > I get why headers are critical for
> Mirroring
> >>> >>> (you
> >>> >>>>>>>>> need tags to
> >>> >>>>>>>>>>      >     > prevent
> >>> >>>>>>>>>>      >     > > >> > loops and sometimes to route messages
> to the
> >>> >>>>> correct
> >>> >>>>>>>>>>      > destination).
> >>> >>>>>>>>>>      >     > > >> > But why do you need headers to audit?
> We are
> >>> >>>>>>> auditing
> >>> >>>>>>>>> by
> >>> >>>>>>>>>>      > producing
> >>> >>>>>>>>>>      >     > > >> > counts to a side topic (and I was under
> the
> >>> >>>>>>>>> impression you do
> >>> >>>>>>>>>>      > the
> >>> >>>>>>>>>>      >     > > >> > same), so we never need to modify the
> >>> message.
> >>> >>>>>>>>>>      >     > > >> >
> >>> >>>>>>>>>>      >     > > >> > Another thing - after we added headers,
> >>> >>> wouldn't
> >>> >>>>> you
> >>> >>>>>>>>> be in the
> >>> >>>>>>>>>>      >     > > >> > business of making sure everyone uses
> them
> >>> >>>>> properly?
> >>> >>>>>>>>> Making
> >>> >>>>>>>>>>      > sure
> >>> >>>>>>>>>>      >     > > >> > everyone includes the right headers you
> >>> need,
> >>> >>> not
> >>> >>>>>>>>> using the
> >>> >>>>>>>>>>      > header
> >>> >>>>>>>>>>      >     > > >> > names you intend to use, etc. I don't
> think
> >>> the
> >>> >>>>>>>>> "policing"
> >>> >>>>>>>>>>      > business
> >>> >>>>>>>>>>      >     > > >> > will ever go away.
> >>> >>>>>>>>>>      >     > > >> >
> >>> >>>>>>>>>>      >     > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd
> Palino
> >>> <
> >>> >>>>>>>>>>      > tpalino@gmail.com>
> >>> >>>>>>>>>>      >     > > wrote:
> >>> >>>>>>>>>>      >     > > >> > > Got it. As an ops guy, I'm not very
> happy
> >>> >>> with
> >>> >>>>> the
> >>> >>>>>>>>>>      > workaround.
> >>> >>>>>>>>>>      >     > Avro
> >>> >>>>>>>>>>      >     > > >> means
> >>> >>>>>>>>>>      >     > > >> > > that I have to be concerned with the
> >>> format
> >>> >>> of
> >>> >>>>> the
> >>> >>>>>>>>> messages
> >>> >>>>>>>>>>      > in
> >>> >>>>>>>>>>      >     > > order to
> >>> >>>>>>>>>>      >     > > >> > run
> >>> >>>>>>>>>>      >     > > >> > > the infrastructure (audit, mirroring,
> >>> etc.).
> >>> >>>>> That
> >>> >>>>>>>>> means
> >>> >>>>>>>>>>      > that I
> >>> >>>>>>>>>>      >     > have
> >>> >>>>>>>>>>      >     > > to
> >>> >>>>>>>>>>      >     > > >> > > handle the schemas, and I have to
> enforce
> >>> >>> rules
> >>> >>>>>>>>> about good
> >>> >>>>>>>>>>      >     > formats.
> >>> >>>>>>>>>>      >     > > >> This
> >>> >>>>>>>>>>      >     > > >> > is
> >>> >>>>>>>>>>      >     > > >> > > not something I want to be in the
> business
> >>> >>> of,
> >>> >>>>>>>>> because I
> >>> >>>>>>>>>>      > should be
> >>> >>>>>>>>>>      >     > > able
> >>> >>>>>>>>>>      >     > > >> > to
> >>> >>>>>>>>>>      >     > > >> > > run a service infrastructure without
> >>> needing
> >>> >>>>> to be
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>>      > weeds of
> >>> >>>>>>>>>>      >     > > >> > dealing
> >>> >>>>>>>>>>      >     > > >> > > with customer data formats.
> >>> >>>>>>>>>>      >     > > >> > >
> >>> >>>>>>>>>>      >     > > >> > > Trust me, a sizable portion of my
> support
> >>> >>> time
> >>> >>>>> is
> >>> >>>>>>>>> spent
> >>> >>>>>>>>>>      > dealing
> >>> >>>>>>>>>>      >     > with
> >>> >>>>>>>>>>      >     > > >> > schema
> >>> >>>>>>>>>>      >     > > >> > > issues. I really would like to get
> away
> >>> from
> >>> >>>>> that.
> >>> >>>>>>>>> Maybe
> >>> >>>>>>>>>>      > I'd have
> >>> >>>>>>>>>>      >     > > more
> >>> >>>>>>>>>>      >     > > >> > time
> >>> >>>>>>>>>>      >     > > >> > > for other hobbies. Like writing. ;)
> >>> >>>>>>>>>>      >     > > >> > >
> >>> >>>>>>>>>>      >     > > >> > > -Todd
> >>> >>>>>>>>>>      >     > > >> > >
> >>> >>>>>>>>>>      >     > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen
> >>> Shapira <
> >>> >>>>>>>>>>      > gwen@confluent.io>
> >>> >>>>>>>>>>      >     > > wrote:
> >>> >>>>>>>>>>      >     > > >> > >
> >>> >>>>>>>>>>      >     > > >> > >> I'm pretty satisfied with the current
> >>> >>>>> workarounds
> >>> >>>>>>>>> (Avro
> >>> >>>>>>>>>>      > container
> >>> >>>>>>>>>>      >     > > >> > >> format), so I'm not too excited
> about the
> >>> >>>>> extra
> >>> >>>>>>>>> work
> >>> >>>>>>>>>>      > required to
> >>> >>>>>>>>>>      >     > do
> >>> >>>>>>>>>>      >     > > >> > >> headers in Kafka. I absolutely don't
> >>> mind it
> >>> >>>>> if
> >>> >>>>>>>>> you do
> >>> >>>>>>>>>>      > it...
> >>> >>>>>>>>>>      >     > > >> > >> I think the Apache convention for
> "good
> >>> >>> idea,
> >>> >>>>> but
> >>> >>>>>>>>> not
> >>> >>>>>>>>>>      > willing to
> >>> >>>>>>>>>>      >     > > put
> >>> >>>>>>>>>>      >     > > >> > >> any work toward it" is +0.5? anyway,
> >>> that's
> >>> >>>>> what
> >>> >>>>>>> I
> >>> >>>>>>>>> was
> >>> >>>>>>>>>>      > trying to
> >>> >>>>>>>>>>      >     > > >> > >> convey :)
> >>> >>>>>>>>>>      >     > > >> > >>
> >>> >>>>>>>>>>      >     > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd
> >>> Palino
> >>> >>> <
> >>> >>>>>>>>>>      > tpalino@gmail.com>
> >>> >>>>>>>>>>      >     > > >> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> > Well I guess my question for you,
> >>> then, is
> >>> >>>>> what
> >>> >>>>>>>>> is
> >>> >>>>>>>>>>      > holding you
> >>> >>>>>>>>>>      >     > > back
> >>> >>>>>>>>>>      >     > > >> > from
> >>> >>>>>>>>>>      >     > > >> > >> > full support for headers? What's
> the
> >>> bit
> >>> >>>>> that
> >>> >>>>>>>>> you're
> >>> >>>>>>>>>>      > missing
> >>> >>>>>>>>>>      >     > that
> >>> >>>>>>>>>>      >     > > >> has
> >>> >>>>>>>>>>      >     > > >> > you
> >>> >>>>>>>>>>      >     > > >> > >> > under a full +1?
> >>> >>>>>>>>>>      >     > > >> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> > -Todd
> >>> >>>>>>>>>>      >     > > >> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM,
> Gwen
> >>> >>>>> Shapira <
> >>> >>>>>>>>>>      >     > gwen@confluent.io>
> >>> >>>>>>>>>>      >     > > >> > wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> I know why people who support
> headers
> >>> >>>>> support
> >>> >>>>>>>>> them, and
> >>> >>>>>>>>>>      > I've
> >>> >>>>>>>>>>      >     > > seen
> >>> >>>>>>>>>>      >     > > >> > what
> >>> >>>>>>>>>>      >     > > >> > >> >> the discussion is like.
> >>> >>>>>>>>>>      >     > > >> > >> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> This is why I'm asking people who
> are
> >>> >>>>> against
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>>      >     > > (especially
> >>> >>>>>>>>>>      >     > > >> > >> >> committers) what will make them
> change
> >>> >>>>> their
> >>> >>>>>>>>> mind - so
> >>> >>>>>>>>>>      > we can
> >>> >>>>>>>>>>      >     > > get
> >>> >>>>>>>>>>      >     > > >> > this
> >>> >>>>>>>>>>      >     > > >> > >> >> part over one way or another.
> >>> >>>>>>>>>>      >     > > >> > >> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> If I sound frustrated it is not at
> >>> Radai,
> >>> >>>>> Jun
> >>> >>>>>>>>> or you
> >>> >>>>>>>>>>      > (Todd)...
> >>> >>>>>>>>>>      >     > > I am
> >>> >>>>>>>>>>      >     > > >> > >> >> just looking for something
> concrete we
> >>> >>> can
> >>> >>>>> do
> >>> >>>>>>>>> to move
> >>> >>>>>>>>>>      > the
> >>> >>>>>>>>>>      >     > > >> discussion
> >>> >>>>>>>>>>      >     > > >> > >> >> along to the yummy design details
> >>> (which
> >>> >>> is
> >>> >>>>>>> the
> >>> >>>>>>>>>>      > argument I
> >>> >>>>>>>>>>      >     > > really
> >>> >>>>>>>>>>      >     > > >> am
> >>> >>>>>>>>>>      >     > > >> > >> >> looking forward to).
> >>> >>>>>>>>>>      >     > > >> > >> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM,
> Todd
> >>> >>>>> Palino <
> >>> >>>>>>>>>>      >     > tpalino@gmail.com>
> >>> >>>>>>>>>>      >     > > >> > wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> > So, Gwen, to your question (even
> >>> though
> >>> >>>>> I'm
> >>> >>>>>>>>> not a
> >>> >>>>>>>>>>      >     > > committer)...
> >>> >>>>>>>>>>      >     > > >> > >> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> > I have always been a strong
> >>> supporter
> >>> >>> of
> >>> >>>>>>>>> introducing
> >>> >>>>>>>>>>      > the
> >>> >>>>>>>>>>      >     > > concept
> >>> >>>>>>>>>>      >     > > >> > of an
> >>> >>>>>>>>>>      >     > > >> > >> >> > envelope to messages, which
> headers
> >>> >>>>>>>>> accomplishes. The
> >>> >>>>>>>>>>      >     > message
> >>> >>>>>>>>>>      >     > > key
> >>> >>>>>>>>>>      >     > > >> > is
> >>> >>>>>>>>>>      >     > > >> > >> >> > already an example of a piece of
> >>> >>> envelope
> >>> >>>>>>>>>>      > information. By
> >>> >>>>>>>>>>      >     > > >> > providing a
> >>> >>>>>>>>>>      >     > > >> > >> >> means
> >>> >>>>>>>>>>      >     > > >> > >> >> > to do this within Kafka itself,
> and
> >>> not
> >>> >>>>>>>>> relying on
> >>> >>>>>>>>>>      > use-case
> >>> >>>>>>>>>>      >     > > >> > specific
> >>> >>>>>>>>>>      >     > > >> > >> >> > implementations, you make it
> much
> >>> >>> easier
> >>> >>>>> for
> >>> >>>>>>>>>>      > components to
> >>> >>>>>>>>>>      >     > > >> > >> interoperate.
> >>> >>>>>>>>>>      >     > > >> > >> >> It
> >>> >>>>>>>>>>      >     > > >> > >> >> > simplifies development of all
> these
> >>> >>>>> things
> >>> >>>>>>>>> (message
> >>> >>>>>>>>>>      > routing,
> >>> >>>>>>>>>>      >     > > >> > auditing,
> >>> >>>>>>>>>>      >     > > >> > >> >> > encryption, etc.) because each
> one
> >>> does
> >>> >>>>> not
> >>> >>>>>>>>> have to
> >>> >>>>>>>>>>      > reinvent
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > >> wheel.
> >>> >>>>>>>>>>      >     > > >> > >> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> > It also makes it much easier
> from a
> >>> >>>>> client
> >>> >>>>>>>>> point of
> >>> >>>>>>>>>>      > view if
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > >> headers
> >>> >>>>>>>>>>      >     > > >> > >> >> are
> >>> >>>>>>>>>>      >     > > >> > >> >> > defined as part of the protocol
> >>> and/or
> >>> >>>>>>>>> message format
> >>> >>>>>>>>>>      > in
> >>> >>>>>>>>>>      >     > > general
> >>> >>>>>>>>>>      >     > > >> > >> because
> >>> >>>>>>>>>>      >     > > >> > >> >> > you can easily produce and
> consume
> >>> >>>>> messages
> >>> >>>>>>>>> without
> >>> >>>>>>>>>>      > having
> >>> >>>>>>>>>>      >     > to
> >>> >>>>>>>>>>      >     > > >> take
> >>> >>>>>>>>>>      >     > > >> > >> into
> >>> >>>>>>>>>>      >     > > >> > >> >> > account specific cases. For
> >>> example, I
> >>> >>>>> want
> >>> >>>>>>>>> to route
> >>> >>>>>>>>>>      >     > messages,
> >>> >>>>>>>>>>      >     > > >> but
> >>> >>>>>>>>>>      >     > > >> > >> >> client A
> >>> >>>>>>>>>>      >     > > >> > >> >> > doesn't support the way audit
> >>> >>> implemented
> >>> >>>>>>>>> headers, and
> >>> >>>>>>>>>>      >     > client
> >>> >>>>>>>>>>      >     > > B
> >>> >>>>>>>>>>      >     > > >> > >> doesn't
> >>> >>>>>>>>>>      >     > > >> > >> >> > support the way encryption or
> >>> routing
> >>> >>>>>>>>> implemented
> >>> >>>>>>>>>>      > headers,
> >>> >>>>>>>>>>      >     > so
> >>> >>>>>>>>>>      >     > > now
> >>> >>>>>>>>>>      >     > > >> > my
> >>> >>>>>>>>>>      >     > > >> > >> >> > application has to create some
> >>> really
> >>> >>>>>>> fragile
> >>> >>>>>>>>> (my
> >>> >>>>>>>>>>      >     > autocorrect
> >>> >>>>>>>>>>      >     > > >> just
> >>> >>>>>>>>>>      >     > > >> > >> tried
> >>> >>>>>>>>>>      >     > > >> > >> >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> > make that "tragic", which is
> >>> probably
> >>> >>>>>>>>> appropriate
> >>> >>>>>>>>>>      > too) code
> >>> >>>>>>>>>>      >     > to
> >>> >>>>>>>>>>      >     > > >> > strip
> >>> >>>>>>>>>>      >     > > >> > >> >> > everything off, rather than just
> >>> >>>>> consuming
> >>> >>>>>>> the
> >>> >>>>>>>>>>      > messages,
> >>> >>>>>>>>>>      >     > > picking
> >>> >>>>>>>>>>      >     > > >> > out
> >>> >>>>>>>>>>      >     > > >> > >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> 1
> >>> >>>>>>>>>>      >     > > >> > >> >> > or 2 headers it's interested
> in, and
> >>> >>>>>>>>> performing its
> >>> >>>>>>>>>>      >     > function.
> >>> >>>>>>>>>>      >     > > >> > >> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> > Honestly, this discussion has
> been
> >>> >>> going
> >>> >>>>> on
> >>> >>>>>>>>> for a
> >>> >>>>>>>>>>      > long time,
> >>> >>>>>>>>>>      >     > > and
> >>> >>>>>>>>>>      >     > > >> > it's
> >>> >>>>>>>>>>      >     > > >> > >> >> > always "Oh, you came up with 2
> use
> >>> >>> cases,
> >>> >>>>>>> and
> >>> >>>>>>>>> yeah,
> >>> >>>>>>>>>>      > those
> >>> >>>>>>>>>>      >     > use
> >>> >>>>>>>>>>      >     > > >> cases
> >>> >>>>>>>>>>      >     > > >> > >> are
> >>> >>>>>>>>>>      >     > > >> > >> >> > real things that someone would
> want
> >>> to
> >>> >>>>> do.
> >>> >>>>>>>>> Here's an
> >>> >>>>>>>>>>      >     > alternate
> >>> >>>>>>>>>>      >     > > >> way
> >>> >>>>>>>>>>      >     > > >> > to
> >>> >>>>>>>>>>      >     > > >> > >> >> > implement them so let's not do
> >>> >>> headers."
> >>> >>>>> If
> >>> >>>>>>>>> we have a
> >>> >>>>>>>>>>      > few
> >>> >>>>>>>>>>      >     > use
> >>> >>>>>>>>>>      >     > > >> cases
> >>> >>>>>>>>>>      >     > > >> > >> that
> >>> >>>>>>>>>>      >     > > >> > >> >> we
> >>> >>>>>>>>>>      >     > > >> > >> >> > actually came up with, you can
> be
> >>> sure
> >>> >>>>> that
> >>> >>>>>>>>> over the
> >>> >>>>>>>>>>      > next
> >>> >>>>>>>>>>      >     > year
> >>> >>>>>>>>>>      >     > > >> > >> there's a
> >>> >>>>>>>>>>      >     > > >> > >> >> > dozen others that we didn't
> think of
> >>> >>> that
> >>> >>>>>>>>> someone
> >>> >>>>>>>>>>      > would like
> >>> >>>>>>>>>>      >     > > to
> >>> >>>>>>>>>>      >     > > >> > do. I
> >>> >>>>>>>>>>      >     > > >> > >> >> > really think it's time to stop
> >>> >>> rehashing
> >>> >>>>>>> this
> >>> >>>>>>>>>>      > discussion and
> >>> >>>>>>>>>>      >     > > >> > instead
> >>> >>>>>>>>>>      >     > > >> > >> >> focus
> >>> >>>>>>>>>>      >     > > >> > >> >> > on a workable standard that we
> can
> >>> >>> adopt.
> >>> >>>>>>>>>>      >     > > >> > >> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> > -Todd
> >>> >>>>>>>>>>      >     > > >> > >> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM,
> Todd
> >>> >>>>> Palino
> >>> >>>>>>> <
> >>> >>>>>>>>>>      >     > > tpalino@gmail.com>
> >>> >>>>>>>>>>      >     > > >> > >> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >> C. per message encryption
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> One drawback of this approach
> is
> >>> that
> >>> >>>>> this
> >>> >>>>>>>>>>      > significantly
> >>> >>>>>>>>>>      >     > > reduce
> >>> >>>>>>>>>>      >     > > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> effectiveness of compression,
> >>> which
> >>> >>>>>>> happens
> >>> >>>>>>>>> on a
> >>> >>>>>>>>>>      > set of
> >>> >>>>>>>>>>      >     > > >> > serialized
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> messages. An alternative is to
> >>> enable
> >>> >>>>> SSL
> >>> >>>>>>>>> for wire
> >>> >>>>>>>>>>      >     > > encryption
> >>> >>>>>>>>>>      >     > > >> and
> >>> >>>>>>>>>>      >     > > >> > >> rely
> >>> >>>>>>>>>>      >     > > >> > >> >> on
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> the storage system (e.g. LUKS)
> >>> for at
> >>> >>>>> rest
> >>> >>>>>>>>>>      > encryption.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >> Jun, this is not sufficient.
> While
> >>> >>> this
> >>> >>>>>>> does
> >>> >>>>>>>>> cover
> >>> >>>>>>>>>>      > the case
> >>> >>>>>>>>>>      >     > > of
> >>> >>>>>>>>>>      >     > > >> > >> removing
> >>> >>>>>>>>>>      >     > > >> > >> >> a
> >>> >>>>>>>>>>      >     > > >> > >> >> >> drive from the system, it will
> not
> >>> >>>>> satisfy
> >>> >>>>>>>>> most
> >>> >>>>>>>>>>      > compliance
> >>> >>>>>>>>>>      >     > > >> > >> requirements
> >>> >>>>>>>>>>      >     > > >> > >> >> for
> >>> >>>>>>>>>>      >     > > >> > >> >> >> encryption of data as whoever
> has
> >>> >>>>> access to
> >>> >>>>>>>>> the
> >>> >>>>>>>>>>      > broker
> >>> >>>>>>>>>>      >     > itself
> >>> >>>>>>>>>>      >     > > >> > still
> >>> >>>>>>>>>>      >     > > >> > >> has
> >>> >>>>>>>>>>      >     > > >> > >> >> >> access to the unencrypted
> data. For
> >>> >>>>>>>>> end-to-end
> >>> >>>>>>>>>>      > encryption
> >>> >>>>>>>>>>      >     > you
> >>> >>>>>>>>>>      >     > > >> > need to
> >>> >>>>>>>>>>      >     > > >> > >> >> >> encrypt at the producer,
> before it
> >>> >>>>> enters
> >>> >>>>>>> the
> >>> >>>>>>>>>>      > system, and
> >>> >>>>>>>>>>      >     > > >> decrypt
> >>> >>>>>>>>>>      >     > > >> > at
> >>> >>>>>>>>>>      >     > > >> > >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >> consumer, after it exits the
> >>> system.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >> -Todd
> >>> >>>>>>>>>>      >     > > >> > >> >> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM,
> >>> radai
> >>> >>> <
> >>> >>>>>>>>>>      >     > > >> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>>      >     > > >> > >
> >>> >>>>>>>>>>      >     > > >> > >> >> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> another big plus of headers
> in the
> >>> >>>>>>> protocol
> >>> >>>>>>>>> is that
> >>> >>>>>>>>>>      > it
> >>> >>>>>>>>>>      >     > would
> >>> >>>>>>>>>>      >     > > >> > enable
> >>> >>>>>>>>>>      >     > > >> > >> >> rapid
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> iteration on ideas outside of
> core
> >>> >>>>> kafka
> >>> >>>>>>>>> and would
> >>> >>>>>>>>>>      > reduce
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > >> number of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> future wire format changes
> >>> required.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> a lot of what is currently a
> KIP
> >>> >>>>>>> represents
> >>> >>>>>>>>> use
> >>> >>>>>>>>>>      > cases that
> >>> >>>>>>>>>>      >     > > are
> >>> >>>>>>>>>>      >     > > >> > not
> >>> >>>>>>>>>>      >     > > >> > >> 100%
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> relevant to all users, and
> some of
> >>> >>> them
> >>> >>>>>>>>> require
> >>> >>>>>>>>>>      > rather
> >>> >>>>>>>>>>      >     > > invasive
> >>> >>>>>>>>>>      >     > > >> > wire
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> protocol changes. a thing a
> good
> >>> >>> recent
> >>> >>>>>>>>> example of
> >>> >>>>>>>>>>      > this is
> >>> >>>>>>>>>>      >     > > >> > kip-98.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> tx-utilizing traffic is
> expected
> >>> to
> >>> >>> be
> >>> >>>>> a
> >>> >>>>>>>>> very small
> >>> >>>>>>>>>>      >     > > fraction of
> >>> >>>>>>>>>>      >     > > >> > >> total
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> traffic and yet the changes
> are
> >>> >>>>> invasive.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> every such wire format change
> >>> >>>>> translates
> >>> >>>>>>>>> into
> >>> >>>>>>>>>>      > painful and
> >>> >>>>>>>>>>      >     > > slow
> >>> >>>>>>>>>>      >     > > >> > >> >> adoption of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> new versions.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> i think a lot of functionality
> >>> >>>>> currently
> >>> >>>>>>> in
> >>> >>>>>>>>> KIPs
> >>> >>>>>>>>>>      > could be
> >>> >>>>>>>>>>      >     > > "spun
> >>> >>>>>>>>>>      >     > > >> > out"
> >>> >>>>>>>>>>      >     > > >> > >> >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> implemented as opt-in plugins
> >>> >>>>> transmitting
> >>> >>>>>>>>> data over
> >>> >>>>>>>>>>      >     > > headers.
> >>> >>>>>>>>>>      >     > > >> > this
> >>> >>>>>>>>>>      >     > > >> > >> >> would
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> keep the core wire format
> >>> stable(r),
> >>> >>>>> core
> >>> >>>>>>>>> codebase
> >>> >>>>>>>>>>      >     > smaller,
> >>> >>>>>>>>>>      >     > > and
> >>> >>>>>>>>>>      >     > > >> > >> avoid
> >>> >>>>>>>>>>      >     > > >> > >> >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> "burden of proof" thats
> sometimes
> >>> >>>>> required
> >>> >>>>>>>>> to prove
> >>> >>>>>>>>>>      > a
> >>> >>>>>>>>>>      >     > > certain
> >>> >>>>>>>>>>      >     > > >> > >> feature
> >>> >>>>>>>>>>      >     > > >> > >> >> is
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> useful enough for a
> wide-enough
> >>> >>>>> audience
> >>> >>>>>>> to
> >>> >>>>>>>>> warrant
> >>> >>>>>>>>>>      > a wire
> >>> >>>>>>>>>>      >     > > >> format
> >>> >>>>>>>>>>      >     > > >> > >> >> change
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> and code complexity additions.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> (to be clear - kip-98 goes
> beyond
> >>> >>>>> "mere"
> >>> >>>>>>>>> wire format
> >>> >>>>>>>>>>      >     > changes
> >>> >>>>>>>>>>      >     > > >> and
> >>> >>>>>>>>>>      >     > > >> > im
> >>> >>>>>>>>>>      >     > > >> > >> not
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> saying it could have been
> >>> completely
> >>> >>>>> done
> >>> >>>>>>>>> with
> >>> >>>>>>>>>>      > headers,
> >>> >>>>>>>>>>      >     > but
> >>> >>>>>>>>>>      >     > > >> > >> >> exactly-once
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> delivery certainly could)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20
> AM,
> >>> Gwen
> >>> >>>>>>>>> Shapira <
> >>> >>>>>>>>>>      >     > > >> gwen@confluent.io
> >>> >>>>>>>>>>      >     > > >> > >
> >>> >>>>>>>>>>      >     > > >> > >> >> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > On Thu, Dec 1, 2016 at
> 10:24 AM,
> >>> >>>>> radai <
> >>> >>>>>>>>>>      >     > > >> > >> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > "For use cases within an
> >>> >>>>> organization,
> >>> >>>>>>>>> one could
> >>> >>>>>>>>>>      >     > always
> >>> >>>>>>>>>>      >     > > use
> >>> >>>>>>>>>>      >     > > >> > >> other
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > approaches such as
> >>> company-wise
> >>> >>>>>>>>> containers"
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > this is what linkedin has
> >>> >>>>>>> traditionally
> >>> >>>>>>>>> done
> >>> >>>>>>>>>>      > but there
> >>> >>>>>>>>>>      >     > > are
> >>> >>>>>>>>>>      >     > > >> > now
> >>> >>>>>>>>>>      >     > > >> > >> >> cases
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > (read
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > - topics) where this is
> not
> >>> >>>>>>> acceptable.
> >>> >>>>>>>>> this
> >>> >>>>>>>>>>      > makes
> >>> >>>>>>>>>>      >     > > headers
> >>> >>>>>>>>>>      >     > > >> > >> useful
> >>> >>>>>>>>>>      >     > > >> > >> >> even
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > within single orgs for
> cases
> >>> >>> where
> >>> >>>>>>>>>>      >     > > one-container-fits-all
> >>> >>>>>>>>>>      >     > > >> > cannot
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> apply.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > as for the particular use
> >>> cases
> >>> >>>>>>> listed,
> >>> >>>>>>>>> i dont
> >>> >>>>>>>>>>      > want
> >>> >>>>>>>>>>      >     > > this to
> >>> >>>>>>>>>>      >     > > >> > >> devolve
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> to a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > discussion of particular
> use
> >>> >>> cases
> >>> >>>>> - i
> >>> >>>>>>>>> think its
> >>> >>>>>>>>>>      >     > enough
> >>> >>>>>>>>>>      >     > > >> that
> >>> >>>>>>>>>>      >     > > >> > >> some
> >>> >>>>>>>>>>      >     > > >> > >> >> of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> them
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > I think a main point of
> >>> contention
> >>> >>> is
> >>> >>>>>>>>> that: We
> >>> >>>>>>>>>>      >     > identified
> >>> >>>>>>>>>>      >     > > few
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > use-cases where headers are
> >>> useful,
> >>> >>>>> do
> >>> >>>>>>> we
> >>> >>>>>>>>> want
> >>> >>>>>>>>>>      > Kafka to
> >>> >>>>>>>>>>      >     > > be a
> >>> >>>>>>>>>>      >     > > >> > >> system
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > that supports those
> use-cases?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > For example, Jun said:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > "Not sure how widely useful
> >>> >>>>> record-level
> >>> >>>>>>>>> lineage
> >>> >>>>>>>>>>      > is
> >>> >>>>>>>>>>      >     > though
> >>> >>>>>>>>>>      >     > > >> > since
> >>> >>>>>>>>>>      >     > > >> > >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > overhead could
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > be significant."
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > We know NiFi supports record
> >>> level
> >>> >>>>>>>>> lineage. I
> >>> >>>>>>>>>>      > don't
> >>> >>>>>>>>>>      >     > think
> >>> >>>>>>>>>>      >     > > it
> >>> >>>>>>>>>>      >     > > >> > was
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > developed for lols, I think
> it
> >>> is
> >>> >>>>> safe
> >>> >>>>>>> to
> >>> >>>>>>>>> assume
> >>> >>>>>>>>>>      > that
> >>> >>>>>>>>>>      >     > the
> >>> >>>>>>>>>>      >     > > NSA
> >>> >>>>>>>>>>      >     > > >> > >> needed
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > that functionality. We also
> know
> >>> >>> that
> >>> >>>>>>>>> certain
> >>> >>>>>>>>>>      > financial
> >>> >>>>>>>>>>      >     > > >> > institutes
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > need to track tampering with
> >>> >>> records
> >>> >>>>> at
> >>> >>>>>>> a
> >>> >>>>>>>>> record
> >>> >>>>>>>>>>      > level
> >>> >>>>>>>>>>      >     > and
> >>> >>>>>>>>>>      >     > > >> > there
> >>> >>>>>>>>>>      >     > > >> > >> are
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > federal regulations that
> >>> absolutely
> >>> >>>>>>>>> require
> >>> >>>>>>>>>>      > this.  They
> >>> >>>>>>>>>>      >     > > also
> >>> >>>>>>>>>>      >     > > >> > need
> >>> >>>>>>>>>>      >     > > >> > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > prove that routing apps that
> >>> >>>>> "touches"
> >>> >>>>>>> the
> >>> >>>>>>>>>>      > messages and
> >>> >>>>>>>>>>      >     > > >> either
> >>> >>>>>>>>>>      >     > > >> > >> reads
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > or updates headers couldn't
> have
> >>> >>>>>>> possibly
> >>> >>>>>>>>>>      > modified the
> >>> >>>>>>>>>>      >     > > >> payload
> >>> >>>>>>>>>>      >     > > >> > >> >> itself.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > They use record level
> >>> encryption to
> >>> >>>>> do
> >>> >>>>>>>>> that -
> >>> >>>>>>>>>>      > apps can
> >>> >>>>>>>>>>      >     > > read
> >>> >>>>>>>>>>      >     > > >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > (sometimes) modify headers
> but
> >>> >>> can't
> >>> >>>>>>>>> touch the
> >>> >>>>>>>>>>      > payload.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > We can totally say "those
> are
> >>> >>> corner
> >>> >>>>>>>>> cases and
> >>> >>>>>>>>>>      > not worth
> >>> >>>>>>>>>>      >     > > >> adding
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > headers to Kafka for", they
> >>> should
> >>> >>>>> use a
> >>> >>>>>>>>> different
> >>> >>>>>>>>>>      >     > pubsub
> >>> >>>>>>>>>>      >     > > >> > message
> >>> >>>>>>>>>>      >     > > >> > >> for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > that (Nifi or one of the
> other
> >>> 1000
> >>> >>>>> that
> >>> >>>>>>>>> cater
> >>> >>>>>>>>>>      >     > > specifically
> >>> >>>>>>>>>>      >     > > >> to
> >>> >>>>>>>>>>      >     > > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > financial industry).
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > But this gets us into a
> catch
> >>> 22:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > If we discuss a specific
> >>> use-case,
> >>> >>>>>>>>> someone can
> >>> >>>>>>>>>>      > always
> >>> >>>>>>>>>>      >     > say
> >>> >>>>>>>>>>      >     > > it
> >>> >>>>>>>>>>      >     > > >> > isn't
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > interesting enough for
> Kafka.
> >>> If we
> >>> >>>>>>>>> discuss more
> >>> >>>>>>>>>>      > general
> >>> >>>>>>>>>>      >     > > >> > trends,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > others can say "well, we
> are not
> >>> >>> sure
> >>> >>>>>>> any
> >>> >>>>>>>>> of them
> >>> >>>>>>>>>>      > really
> >>> >>>>>>>>>>      >     > > >> needs
> >>> >>>>>>>>>>      >     > > >> > >> >> headers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > specifically. This is just
> hand
> >>> >>>>> waving
> >>> >>>>>>>>> and not
> >>> >>>>>>>>>>      >     > > interesting.".
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > I think discussing
> use-cases in
> >>> >>>>>>> specifics
> >>> >>>>>>>>> is super
> >>> >>>>>>>>>>      >     > > important
> >>> >>>>>>>>>>      >     > > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> decide
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > implementation details for
> >>> headers
> >>> >>>>> (my
> >>> >>>>>>>>> use-cases
> >>> >>>>>>>>>>      > lean
> >>> >>>>>>>>>>      >     > > toward
> >>> >>>>>>>>>>      >     > > >> > >> >> numerical
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > keys with namespaces and
> object
> >>> >>>>> values,
> >>> >>>>>>>>> others
> >>> >>>>>>>>>>      > differ),
> >>> >>>>>>>>>>      >     > > but I
> >>> >>>>>>>>>>      >     > > >> > >> think
> >>> >>>>>>>>>>      >     > > >> > >> >> we
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > need to answer the general
> "Are
> >>> we
> >>> >>>>> going
> >>> >>>>>>>>> to have
> >>> >>>>>>>>>>      >     > headers"
> >>> >>>>>>>>>>      >     > > >> > question
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > first.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > I'd love to hear from the
> other
> >>> >>>>>>>>> committers in the
> >>> >>>>>>>>>>      >     > > discussion:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > What would it take to
> convince
> >>> you
> >>> >>>>> that
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>>      > in Kafka
> >>> >>>>>>>>>>      >     > > are
> >>> >>>>>>>>>>      >     > > >> a
> >>> >>>>>>>>>>      >     > > >> > >> good
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > idea in general, so we can
> move
> >>> >>> ahead
> >>> >>>>>>> and
> >>> >>>>>>>>> try to
> >>> >>>>>>>>>>      > agree
> >>> >>>>>>>>>>      >     > on
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > >> >> details?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > I feel like we keep moving
> the
> >>> goal
> >>> >>>>>>> posts
> >>> >>>>>>>>> and
> >>> >>>>>>>>>>      > this is
> >>> >>>>>>>>>>      >     > > truly
> >>> >>>>>>>>>>      >     > > >> > >> >> exhausting.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > For the record, I mildly
> support
> >>> >>>>> adding
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>>      > to Kafka
> >>> >>>>>>>>>>      >     > > >> > (+0.5?).
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > The community can continue
> to
> >>> find
> >>> >>>>>>>>> workarounds to
> >>> >>>>>>>>>>      > the
> >>> >>>>>>>>>>      >     > > issue
> >>> >>>>>>>>>>      >     > > >> and
> >>> >>>>>>>>>>      >     > > >> > >> there
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > are some benefits to
> keeping the
> >>> >>>>> message
> >>> >>>>>>>>> format
> >>> >>>>>>>>>>      > and
> >>> >>>>>>>>>>      >     > > clients
> >>> >>>>>>>>>>      >     > > >> > >> simpler.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > But I see the usefulness of
> >>> headers
> >>> >>>>> to
> >>> >>>>>>>>> many
> >>> >>>>>>>>>>      > use-cases
> >>> >>>>>>>>>>      >     > and
> >>> >>>>>>>>>>      >     > > if
> >>> >>>>>>>>>>      >     > > >> we
> >>> >>>>>>>>>>      >     > > >> > >> can
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > find a good and generally
> useful
> >>> >>> way
> >>> >>>>> to
> >>> >>>>>>>>> add it to
> >>> >>>>>>>>>>      > Kafka,
> >>> >>>>>>>>>>      >     > > it
> >>> >>>>>>>>>>      >     > > >> > will
> >>> >>>>>>>>>>      >     > > >> > >> make
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > Kafka easier to use for
> many -
> >>> >>> worthy
> >>> >>>>>>>>> goal in my
> >>> >>>>>>>>>>      > eyes.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > are interesting/feasible,
> but:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > A+B. i think there are use
> >>> cases
> >>> >>>>> for
> >>> >>>>>>>>> polyglot
> >>> >>>>>>>>>>      > topics.
> >>> >>>>>>>>>>      >     > > >> > >> especially if
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> kafka
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > is being used to "trunk"
> >>> >>> something
> >>> >>>>>>> else.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > D. multiple topics would
> make
> >>> it
> >>> >>>>>>> harder
> >>> >>>>>>>>> to write
> >>> >>>>>>>>>>      >     > > portable
> >>> >>>>>>>>>>      >     > > >> > >> consumer
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> code.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > partition remapping would
> mess
> >>> >>> with
> >>> >>>>>>>>> locality of
> >>> >>>>>>>>>>      >     > > consumption
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> guarantees.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > E+F. a use case I see for
> >>> >>>>>>>>> lineage/metadata is
> >>> >>>>>>>>>>      >     > > >> > >> billing/chargeback.
> >>> >>>>>>>>>>      >     > > >> > >> >> for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > that
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > use case it is not enough
> to
> >>> >>> simply
> >>> >>>>>>>>> record the
> >>> >>>>>>>>>>      > point
> >>> >>>>>>>>>>      >     > of
> >>> >>>>>>>>>>      >     > > >> > origin,
> >>> >>>>>>>>>>      >     > > >> > >> but
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> every
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > replication stop (think
> mirror
> >>> >>>>> maker)
> >>> >>>>>>>>> must also
> >>> >>>>>>>>>>      > add a
> >>> >>>>>>>>>>      >     > > >> record
> >>> >>>>>>>>>>      >     > > >> > to
> >>> >>>>>>>>>>      >     > > >> > >> >> form a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > "transit log".
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > as for stream processing
> on
> >>> top
> >>> >>> of
> >>> >>>>>>>>> kafka - i
> >>> >>>>>>>>>>      > know
> >>> >>>>>>>>>>      >     > samza
> >>> >>>>>>>>>>      >     > > >> has a
> >>> >>>>>>>>>>      >     > > >> > >> >> metadata
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > map
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > which they carry around in
> >>> >>>>> addition to
> >>> >>>>>>>>> user
> >>> >>>>>>>>>>      > values.
> >>> >>>>>>>>>>      >     > > headers
> >>> >>>>>>>>>>      >     > > >> > are
> >>> >>>>>>>>>>      >     > > >> > >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > perfect
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > fit for these things.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at
> 6:50
> >>> PM,
> >>> >>>>> Jun
> >>> >>>>>>>>> Rao <
> >>> >>>>>>>>>>      >     > > jun@confluent.io
> >>> >>>>>>>>>>      >     > > >> >
> >>> >>>>>>>>>>      >     > > >> > >> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> Hi, Michael,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> In order to answer the
> first
> >>> two
> >>> >>>>>>>>> questions, it
> >>> >>>>>>>>>>      > would
> >>> >>>>>>>>>>      >     > be
> >>> >>>>>>>>>>      >     > > >> > helpful
> >>> >>>>>>>>>>      >     > > >> > >> >> if we
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > could
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> identify 1 or 2 strong
> use
> >>> cases
> >>> >>>>> for
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>>      > in the
> >>> >>>>>>>>>>      >     > > space
> >>> >>>>>>>>>>      >     > > >> > for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > third-party
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> vendors. For use cases
> >>> within an
> >>> >>>>>>>>> organization,
> >>> >>>>>>>>>>      > one
> >>> >>>>>>>>>>      >     > > could
> >>> >>>>>>>>>>      >     > > >> > always
> >>> >>>>>>>>>>      >     > > >> > >> >> use
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > other
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> approaches such as
> >>> company-wise
> >>> >>>>>>>>> containers to
> >>> >>>>>>>>>>      > get
> >>> >>>>>>>>>>      >     > > around
> >>> >>>>>>>>>>      >     > > >> w/o
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> headers. I
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> went through the use
> cases in
> >>> >>> the
> >>> >>>>> KIP
> >>> >>>>>>>>> and in
> >>> >>>>>>>>>>      > Radai's
> >>> >>>>>>>>>>      >     > > wiki
> >>> >>>>>>>>>>      >     > > >> (
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> https://cwiki.apache.org/confl
> >>> >>>>>>>>>>      > uence/display/KAFKA/A+
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > Case+for+Kafka+Headers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> ).
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> The following are the
> ones
> >>> that
> >>> >>>>> that
> >>> >>>>>>> I
> >>> >>>>>>>>>>      > understand and
> >>> >>>>>>>>>>      >     > > >> could
> >>> >>>>>>>>>>      >     > > >> > be
> >>> >>>>>>>>>>      >     > > >> > >> in
> >>> >>>>>>>>>>      >     > > >> > >> >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> third-party use case
> >>> category.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> A. content-type
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> It seems that in general,
> >>> >>>>>>> content-type
> >>> >>>>>>>>> should
> >>> >>>>>>>>>>      > be set
> >>> >>>>>>>>>>      >     > at
> >>> >>>>>>>>>>      >     > > >> the
> >>> >>>>>>>>>>      >     > > >> > >> topic
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> level.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> Not sure if mixing
> messages
> >>> with
> >>> >>>>>>>>> different
> >>> >>>>>>>>>>      > content
> >>> >>>>>>>>>>      >     > > types
> >>> >>>>>>>>>>      >     > > >> > >> should be
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> encouraged.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> B. schema id
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> Since the value is mostly
> >>> >>> useless
> >>> >>>>>>>>> without
> >>> >>>>>>>>>>      > schema id,
> >>> >>>>>>>>>>      >     > it
> >>> >>>>>>>>>>      >     > > >> > seems
> >>> >>>>>>>>>>      >     > > >> > >> that
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > storing
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> the schema id together
> with
> >>> >>>>>>> serialized
> >>> >>>>>>>>> bytes
> >>> >>>>>>>>>>      > in the
> >>> >>>>>>>>>>      >     > > value
> >>> >>>>>>>>>>      >     > > >> is
> >>> >>>>>>>>>>      >     > > >> > >> >> better?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> C. per message encryption
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> One drawback of this
> >>> approach is
> >>> >>>>> that
> >>> >>>>>>>>> this
> >>> >>>>>>>>>>      >     > > significantly
> >>> >>>>>>>>>>      >     > > >> > reduce
> >>> >>>>>>>>>>      >     > > >> > >> >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> effectiveness of
> compression,
> >>> >>>>> which
> >>> >>>>>>>>> happens on
> >>> >>>>>>>>>>      > a set
> >>> >>>>>>>>>>      >     > of
> >>> >>>>>>>>>>      >     > > >> > >> serialized
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> messages. An alternative
> is
> >>> to
> >>> >>>>> enable
> >>> >>>>>>>>> SSL for
> >>> >>>>>>>>>>      > wire
> >>> >>>>>>>>>>      >     > > >> > encryption
> >>> >>>>>>>>>>      >     > > >> > >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> rely
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > on
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> the storage system (e.g.
> >>> LUKS)
> >>> >>>>> for at
> >>> >>>>>>>>> rest
> >>> >>>>>>>>>>      >     > encryption.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> D. cluster ID for
> mirroring
> >>> >>> across
> >>> >>>>>>>>> Kafka
> >>> >>>>>>>>>>      > clusters
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> This is actually
> interesting.
> >>> >>>>> Today,
> >>> >>>>>>>>> to avoid
> >>> >>>>>>>>>>      >     > > introducing
> >>> >>>>>>>>>>      >     > > >> > >> cycles
> >>> >>>>>>>>>>      >     > > >> > >> >> when
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > doing
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> mirroring across data
> >>> centers,
> >>> >>> one
> >>> >>>>>>>>> would
> >>> >>>>>>>>>>      > either have
> >>> >>>>>>>>>>      >     > to
> >>> >>>>>>>>>>      >     > > >> set
> >>> >>>>>>>>>>      >     > > >> > up
> >>> >>>>>>>>>>      >     > > >> > >> two
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> Kafka
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> clusters (a local and an
> >>> >>>>> aggregate)
> >>> >>>>>>>>> per data
> >>> >>>>>>>>>>      > center
> >>> >>>>>>>>>>      >     > or
> >>> >>>>>>>>>>      >     > > >> > rename
> >>> >>>>>>>>>>      >     > > >> > >> >> topics.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> Neither is ideal. With
> >>> headers,
> >>> >>>>> the
> >>> >>>>>>>>> producer
> >>> >>>>>>>>>>      > could
> >>> >>>>>>>>>>      >     > tag
> >>> >>>>>>>>>>      >     > > >> each
> >>> >>>>>>>>>>      >     > > >> > >> >> message
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> with
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> the producing cluster ID
> in
> >>> the
> >>> >>>>>>> header.
> >>> >>>>>>>>>>      > MirrorMaker
> >>> >>>>>>>>>>      >     > > could
> >>> >>>>>>>>>>      >     > > >> > then
> >>> >>>>>>>>>>      >     > > >> > >> >> avoid
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> mirroring messages to a
> >>> cluster
> >>> >>> if
> >>> >>>>>>>>> they are
> >>> >>>>>>>>>>      > tagged
> >>> >>>>>>>>>>      >     > with
> >>> >>>>>>>>>>      >     > > >> the
> >>> >>>>>>>>>>      >     > > >> > >> same
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> cluster
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> id.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> However, an alternative
> >>> approach
> >>> >>>>> is
> >>> >>>>>>> to
> >>> >>>>>>>>>>      > introduce sth
> >>> >>>>>>>>>>      >     > > like
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> hierarchical
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> topic and store messages
> from
> >>> >>>>>>> different
> >>> >>>>>>>>>>      > clusters in
> >>> >>>>>>>>>>      >     > > >> > different
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> partitions
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> under the same topic.
> This
> >>> >>>>> approach
> >>> >>>>>>>>> avoids
> >>> >>>>>>>>>>      > filtering
> >>> >>>>>>>>>>      >     > > out
> >>> >>>>>>>>>>      >     > > >> > >> unneeded
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> data
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> makes offset preserving
> >>> easier
> >>> >>> to
> >>> >>>>>>>>> support. It
> >>> >>>>>>>>>>      > may
> >>> >>>>>>>>>>      >     > make
> >>> >>>>>>>>>>      >     > > >> > >> compaction
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > trickier
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> though since the same
> key may
> >>> >>>>> show up
> >>> >>>>>>>>> in
> >>> >>>>>>>>>>      > different
> >>> >>>>>>>>>>      >     > > >> > partitions.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> E. record-level lineage
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> For example, a source
> >>> connector
> >>> >>>>> could
> >>> >>>>>>>>> store in
> >>> >>>>>>>>>>      > the
> >>> >>>>>>>>>>      >     > > message
> >>> >>>>>>>>>>      >     > > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> metadata
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> (e.g. UUID) of the source
> >>> >>> record.
> >>> >>>>>>>>> Similarly,
> >>> >>>>>>>>>>      > if a
> >>> >>>>>>>>>>      >     > > stream
> >>> >>>>>>>>>>      >     > > >> job
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> transforms
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> messages from topic A to
> >>> topic
> >>> >>> B,
> >>> >>>>> the
> >>> >>>>>>>>> library
> >>> >>>>>>>>>>      > could
> >>> >>>>>>>>>>      >     > > >> include
> >>> >>>>>>>>>>      >     > > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> source
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> message offset in each
> of the
> >>> >>>>>>>>> transformed
> >>> >>>>>>>>>>      > message in
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > >> header.
> >>> >>>>>>>>>>      >     > > >> > >> >> Not
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > sure
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> how widely useful
> >>> record-level
> >>> >>>>>>> lineage
> >>> >>>>>>>>> is
> >>> >>>>>>>>>>      > though
> >>> >>>>>>>>>>      >     > since
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > >> >> overhead
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > could
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> be significant.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> F. auditing metadata
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> We could put things like
> >>> >>>>>>>>> clientId/host/user in
> >>> >>>>>>>>>>      > the
> >>> >>>>>>>>>>      >     > > header
> >>> >>>>>>>>>>      >     > > >> in
> >>> >>>>>>>>>>      >     > > >> > >> each
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > message
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> for auditing. These
> metadata
> >>> are
> >>> >>>>>>>>> really at the
> >>> >>>>>>>>>>      >     > producer
> >>> >>>>>>>>>>      >     > > >> > level
> >>> >>>>>>>>>>      >     > > >> > >> >> though.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > So, a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> more efficient way is to
> only
> >>> >>>>>>> include a
> >>> >>>>>>>>>>      > "producerId"
> >>> >>>>>>>>>>      >     > > per
> >>> >>>>>>>>>>      >     > > >> > >> message
> >>> >>>>>>>>>>      >     > > >> > >> >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > send
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> the producerId ->
> metadata
> >>> >>> mapping
> >>> >>>>>>>>>>      > independently.
> >>> >>>>>>>>>>      >     > > KIP-98
> >>> >>>>>>>>>>      >     > > >> is
> >>> >>>>>>>>>>      >     > > >> > >> >> actually
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> proposing including such
> a
> >>> >>>>> producerId
> >>> >>>>>>>>> natively
> >>> >>>>>>>>>>      > in the
> >>> >>>>>>>>>>      >     > > >> > message.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> So, overall, I not sure
> that
> >>> I
> >>> >>> am
> >>> >>>>>>> fully
> >>> >>>>>>>>>>      > convinced of
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > strong
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > third-party
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> use cases of headers yet.
> >>> >>> Perhaps
> >>> >>>>> we
> >>> >>>>>>>>> could
> >>> >>>>>>>>>>      > discuss a
> >>> >>>>>>>>>>      >     > > bit
> >>> >>>>>>>>>>      >     > > >> > more
> >>> >>>>>>>>>>      >     > > >> > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> make
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > one
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> or two really convincing
> use
> >>> >>>>> cases.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> Another orthogonal
> question
> >>> is
> >>> >>>>>>>>> whether header
> >>> >>>>>>>>>>      > should
> >>> >>>>>>>>>>      >     > > be
> >>> >>>>>>>>>>      >     > > >> > >> exposed
> >>> >>>>>>>>>>      >     > > >> > >> >> in
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > stream
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> processing systems such
> Kafka
> >>> >>>>> stream,
> >>> >>>>>>>>> Samza,
> >>> >>>>>>>>>>      > and
> >>> >>>>>>>>>>      >     > Spark
> >>> >>>>>>>>>>      >     > > >> > >> streaming.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> Currently, those systems
> just
> >>> >>> deal
> >>> >>>>>>> with
> >>> >>>>>>>>>>      > key/value
> >>> >>>>>>>>>>      >     > > pairs.
> >>> >>>>>>>>>>      >     > > >> > >> Should we
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > expose a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> third thing header there
> too
> >>> or
> >>> >>>>>>>>> somehow map
> >>> >>>>>>>>>>      > header to
> >>> >>>>>>>>>>      >     > > key
> >>> >>>>>>>>>>      >     > > >> or
> >>> >>>>>>>>>>      >     > > >> > >> >> value?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> Thanks,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> Jun
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at
> 3:35
> >>> AM,
> >>> >>>>>>>>> Michael
> >>> >>>>>>>>>>      > Pearce <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> Michael.Pearce@ig.com>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > I assume, that after a
> >>> period
> >>> >>>>> of a
> >>> >>>>>>>>> week,
> >>> >>>>>>>>>>      > that there
> >>> >>>>>>>>>>      >     > > is
> >>> >>>>>>>>>>      >     > > >> no
> >>> >>>>>>>>>>      >     > > >> > >> >> concerns
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> now
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > with points 1, and 2
> and
> >>> now
> >>> >>> we
> >>> >>>>>>> have
> >>> >>>>>>>>>>      > agreement that
> >>> >>>>>>>>>>      >     > > >> > headers
> >>> >>>>>>>>>>      >     > > >> > >> are
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> useful
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > needed in Kafka. As
> such if
> >>> >>> put
> >>> >>>>> to
> >>> >>>>>>> a
> >>> >>>>>>>>> KIP
> >>> >>>>>>>>>>      > vote, this
> >>> >>>>>>>>>>      >     > > >> > wouldn't
> >>> >>>>>>>>>>      >     > > >> > >> be
> >>> >>>>>>>>>>      >     > > >> > >> >> a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > reason
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > reject.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > @
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > Ignacio on point 4).
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > I think for purpose of
> >>> getting
> >>> >>>>> this
> >>> >>>>>>>>> KIP
> >>> >>>>>>>>>>      > moving past
> >>> >>>>>>>>>>      >     > > >> this,
> >>> >>>>>>>>>>      >     > > >> > we
> >>> >>>>>>>>>>      >     > > >> > >> can
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> state
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > key will be a 4 bytes
> space
> >>> >>> that
> >>> >>>>>>> can
> >>> >>>>>>>>> will be
> >>> >>>>>>>>>>      >     > > naturally
> >>> >>>>>>>>>>      >     > > >> > >> >> interpreted
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> as
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > an
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > Int32 (if namespacing
> is
> >>> later
> >>> >>>>>>>>> wanted you can
> >>> >>>>>>>>>>      >     > easily
> >>> >>>>>>>>>>      >     > > >> split
> >>> >>>>>>>>>>      >     > > >> > >> this
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> into
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > two
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > int16 spaces), from the
> >>> wire
> >>> >>>>>>> protocol
> >>> >>>>>>>>>>      >     > implementation
> >>> >>>>>>>>>>      >     > > >> this
> >>> >>>>>>>>>>      >     > > >> > >> makes
> >>> >>>>>>>>>>      >     > > >> > >> >> no
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > difference I don't
> >>> believe. Is
> >>> >>>>> this
> >>> >>>>>>>>>>      > reasonable to
> >>> >>>>>>>>>>      >     > > all?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > On 5) as per point 4
> >>> therefor
> >>> >>>>> happy
> >>> >>>>>>>>> we keep
> >>> >>>>>>>>>>      > with 32
> >>> >>>>>>>>>>      >     > > >> bits.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
> >>> >>>>>>>>>>      > ignacio.solis@gmail.com on
> >>> >>>>>>>>>>      >     > > >> behalf
> >>> >>>>>>>>>>      >     > > >> > of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> Ignacio
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > Solis" <
> >>> >>> ignacio.solis@gmail.com
> >>> >>>>> on
> >>> >>>>>>>>> behalf of
> >>> >>>>>>>>>>      >     > > >> > isolis@igso.net
> >>> >>>>>>>>>>      >     > > >> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     Summary:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     3) Yes - Header
> value
> >>> as
> >>> >>>>> byte[]
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     4a) Int,Int - No
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     4b) Int - Yes
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     4c) String -
> Reluctant
> >>> >>> maybe
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     5) I believe the
> header
> >>> >>>>> system
> >>> >>>>>>>>> should
> >>> >>>>>>>>>>      > take a
> >>> >>>>>>>>>>      >     > > single
> >>> >>>>>>>>>>      >     > > >> > >> int.  I
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> think
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > 32bits is
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     a good size, if you
> >>> want
> >>> >>> to
> >>> >>>>>>>>> interpret
> >>> >>>>>>>>>>      > this as
> >>> >>>>>>>>>>      >     > to
> >>> >>>>>>>>>>      >     > > >> 16bit
> >>> >>>>>>>>>>      >     > > >> > >> >> numbers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> in
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > layer
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     above go right
> ahead.
> >>> If
> >>> >>>>>>>>> somebody wants
> >>> >>>>>>>>>>      > to
> >>> >>>>>>>>>>      >     > argue
> >>> >>>>>>>>>>      >     > > >> for
> >>> >>>>>>>>>>      >     > > >> > 16
> >>> >>>>>>>>>>      >     > > >> > >> >> bits
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> or
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > 64
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > bits of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     header key space I
> >>> would
> >>> >>>>>>> listen.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     Discussion:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     Dividing the key
> space
> >>> >>> into
> >>> >>>>>>>>> sub_key_1 and
> >>> >>>>>>>>>>      >     > > sub_key_2
> >>> >>>>>>>>>>      >     > > >> > >> makes no
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > sense to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > me at
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     this layer.  Are we
> >>> going
> >>> >>> to
> >>> >>>>>>>>> start
> >>> >>>>>>>>>>      > providing
> >>> >>>>>>>>>>      >     > > APIs to
> >>> >>>>>>>>>>      >     > > >> > get
> >>> >>>>>>>>>>      >     > > >> > >> all
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     sub_key_1s? or all
> the
> >>> >>>>>>>>> sub_key_2s?  If
> >>> >>>>>>>>>>      > there is
> >>> >>>>>>>>>>      >     > > no
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> distinguishing
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > functions
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     that are applied to
> >>> each
> >>> >>> one
> >>> >>>>>>>>> then they
> >>> >>>>>>>>>>      > should
> >>> >>>>>>>>>>      >     > be
> >>> >>>>>>>>>>      >     > > a
> >>> >>>>>>>>>>      >     > > >> > single
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> value.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > At
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > this
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     layer all we're
> doing
> >>> is
> >>> >>>>>>>>> equality.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     If the above layer
> >>> wants
> >>> >>> to
> >>> >>>>>>>>> interpret
> >>> >>>>>>>>>>      > this as
> >>> >>>>>>>>>>      >     > 2,
> >>> >>>>>>>>>>      >     > > 3
> >>> >>>>>>>>>>      >     > > >> or
> >>> >>>>>>>>>>      >     > > >> > >> more
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> values
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > that's a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     different
> question.  I
> >>> >>>>>>>>> personally think
> >>> >>>>>>>>>>      > it's
> >>> >>>>>>>>>>      >     > all
> >>> >>>>>>>>>>      >     > > one
> >>> >>>>>>>>>>      >     > > >> > >> >> keyspace
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > that is
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     getting assigned
> using
> >>> >>> some
> >>> >>>>>>>>> structure,
> >>> >>>>>>>>>>      > but if
> >>> >>>>>>>>>>      >     > you
> >>> >>>>>>>>>>      >     > > >> > want to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > sub-assign
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > parts
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     of it then that's
> fine.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     The same discussion
> >>> >>> applies
> >>> >>>>> to
> >>> >>>>>>>>> strings.
> >>> >>>>>>>>>>      > If
> >>> >>>>>>>>>>      >     > > somebody
> >>> >>>>>>>>>>      >     > > >> > >> argued
> >>> >>>>>>>>>>      >     > > >> > >> >> for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > strings,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     would we be
> arguing to
> >>> >>>>> divide
> >>> >>>>>>> the
> >>> >>>>>>>>>>      > strings with
> >>> >>>>>>>>>>      >     > > dots
> >>> >>>>>>>>>>      >     > > >> > ('.')
> >>> >>>>>>>>>>      >     > > >> > >> >> as a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > requirement?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     Would we want them
> to
> >>> give
> >>> >>>>> us
> >>> >>>>>>> the
> >>> >>>>>>>>>>      > different
> >>> >>>>>>>>>>      >     > name
> >>> >>>>>>>>>>      >     > > >> > segments
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > separately?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     Would we be
> performing
> >>> any
> >>> >>>>>>>>> actions on
> >>> >>>>>>>>>>      > this key
> >>> >>>>>>>>>>      >     > > other
> >>> >>>>>>>>>>      >     > > >> > than
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > matching?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     Nacho
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     On Fri, Nov 18,
> 2016 at
> >>> >>> 9:30
> >>> >>>>>>> AM,
> >>> >>>>>>>>> Michael
> >>> >>>>>>>>>>      >     > Pearce <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> Michael.Pearce@ig.com
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > #jay #jun any
> >>> concerns
> >>> >>> on
> >>> >>>>> 1
> >>> >>>>>>>>> and 2
> >>> >>>>>>>>>>      > still?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > @all
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > To get this
> moving
> >>> >>> along a
> >>> >>>>>>> bit
> >>> >>>>>>>>> more
> >>> >>>>>>>>>>      > I'd also
> >>> >>>>>>>>>>      >     > > like
> >>> >>>>>>>>>>      >     > > >> to
> >>> >>>>>>>>>>      >     > > >> > >> ask
> >>> >>>>>>>>>>      >     > > >> > >> >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> get
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > clarity on
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > the below last
> >>> points:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > 3) I believe
> we're
> >>> all
> >>> >>>>>>> roughly
> >>> >>>>>>>>> happy
> >>> >>>>>>>>>>      > with the
> >>> >>>>>>>>>>      >     > > >> header
> >>> >>>>>>>>>>      >     > > >> > >> value
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > being a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > byte[]?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > 4) I believe
> >>> consensus
> >>> >>> has
> >>> >>>>>>>>> been for an
> >>> >>>>>>>>>>      >     > > namespace
> >>> >>>>>>>>>>      >     > > >> > based
> >>> >>>>>>>>>>      >     > > >> > >> int
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > approach
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > {int,int} for the
> >>> key.
> >>> >>> Any
> >>> >>>>>>>>> objections
> >>> >>>>>>>>>>      > if this
> >>> >>>>>>>>>>      >     > > is
> >>> >>>>>>>>>>      >     > > >> > what
> >>> >>>>>>>>>>      >     > > >> > >> we
> >>> >>>>>>>>>>      >     > > >> > >> >> go
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > with?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > 5) as we have if
> >>> >>>>> assumption
> >>> >>>>>>> in
> >>> >>>>>>>>> (4)  is
> >>> >>>>>>>>>>      >     > correct,
> >>> >>>>>>>>>>      >     > > >> > >> {int,int}
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> keys.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > Should both
> int's be
> >>> >>>>> int16 or
> >>> >>>>>>>>> int32?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > I'm for them
> being
> >>> >>> int16(2
> >>> >>>>>>>>> bytes) as
> >>> >>>>>>>>>>      > combined
> >>> >>>>>>>>>>      >     > > is
> >>> >>>>>>>>>>      >     > > >> > space
> >>> >>>>>>>>>>      >     > > >> > >> of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > 4bytes as
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > per
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > original and
> gives
> >>> >>> plenty
> >>> >>>>> of
> >>> >>>>>>>>>>      > combinations for
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> foreseeable,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > keeps
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > the overhead
> small.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > Do we see any
> >>> benefit in
> >>> >>>>>>>>> another kip
> >>> >>>>>>>>>>      > call to
> >>> >>>>>>>>>>      >     > > >> discuss
> >>> >>>>>>>>>>      >     > > >> > >> >> these at
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > all?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > Cheers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > Mike
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> >>>>>>> ______________________________
> >>> >>>>>>>>>>      > __________
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > From: K Burstev <
> >>> >>>>>>>>> k.burstev@yandex.com>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > Sent: Friday,
> >>> November
> >>> >>> 18,
> >>> >>>>>>> 2016
> >>> >>>>>>>>>>      > 7:07:07 AM
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > To:
> >>> >>> dev@kafka.apache.org
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > Subject: Re:
> >>> [DISCUSS]
> >>> >>>>> KIP-82
> >>> >>>>>>>>> - Add
> >>> >>>>>>>>>>      > Record
> >>> >>>>>>>>>>      >     > > Headers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > For what it is
> worth
> >>> >>> also
> >>> >>>>> i
> >>> >>>>>>>>> agree. As
> >>> >>>>>>>>>>      > a user:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >  1) Yes -
> Headers are
> >>> >>>>>>>>> worthwhile
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >  2) Yes - Headers
> >>> should
> >>> >>>>> be a
> >>> >>>>>>>>> top level
> >>> >>>>>>>>>>      >     > option
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > 14.11.2016,
> 21:15,
> >>> >>>>> "Ignacio
> >>> >>>>>>>>> Solis" <
> >>> >>>>>>>>>>      >     > > >> isolis@igso.net
> >>> >>>>>>>>>>      >     > > >> > >:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > > 1) Yes -
> Headers
> >>> are
> >>> >>>>>>>>> worthwhile
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > > 2) Yes -
> Headers
> >>> >>> should
> >>> >>>>> be
> >>> >>>>>>> a
> >>> >>>>>>>>> top
> >>> >>>>>>>>>>      > level
> >>> >>>>>>>>>>      >     > option
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > > On Mon, Nov 14,
> >>> 2016
> >>> >>> at
> >>> >>>>>>> 9:16
> >>> >>>>>>>>> AM,
> >>> >>>>>>>>>>      > Michael
> >>> >>>>>>>>>>      >     > > Pearce
> >>> >>>>>>>>>>      >     > > >> <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > > wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Hi Roger,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  The kip
> >>> >>>>> details/examples
> >>> >>>>>>>>> the
> >>> >>>>>>>>>>      > original
> >>> >>>>>>>>>>      >     > > proposal
> >>> >>>>>>>>>>      >     > > >> > for
> >>> >>>>>>>>>>      >     > > >> > >> key
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > spacing
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> ,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > not
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  new
> mentioned as
> >>> per
> >>> >>>>>>>>> discussion
> >>> >>>>>>>>>>      > namespace
> >>> >>>>>>>>>>      >     > > >> idea.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  We will need
> to
> >>> >>> update
> >>> >>>>>>> the
> >>> >>>>>>>>> kip,
> >>> >>>>>>>>>>      > when we
> >>> >>>>>>>>>>      >     > get
> >>> >>>>>>>>>>      >     > > >> > >> agreement
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> this
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > is a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > better
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  approach
> (which
> >>> >>> seems
> >>> >>>>> to
> >>> >>>>>>>>> be the
> >>> >>>>>>>>>>      > case if I
> >>> >>>>>>>>>>      >     > > have
> >>> >>>>>>>>>>      >     > > >> > >> >> understood
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > general
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  feeling in
> the
> >>> >>>>>>>>> conversation)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Re the
> variable
> >>> >>> ints,
> >>> >>>>> at
> >>> >>>>>>>>> very
> >>> >>>>>>>>>>      > early stage
> >>> >>>>>>>>>>      >     > > we
> >>> >>>>>>>>>>      >     > > >> did
> >>> >>>>>>>>>>      >     > > >> > >> think
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> about
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > this. I
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > think
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  the added
> >>> complexity
> >>> >>>>> for
> >>> >>>>>>>>> the
> >>> >>>>>>>>>>      > saving isn't
> >>> >>>>>>>>>>      >     > > >> worth
> >>> >>>>>>>>>>      >     > > >> > it.
> >>> >>>>>>>>>>      >     > > >> > >> >> I'd
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > rather
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> go
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > with, if
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  we want to
> reduce
> >>> >>>>>>>>> overheads and
> >>> >>>>>>>>>>      > size
> >>> >>>>>>>>>>      >     > int16
> >>> >>>>>>>>>>      >     > > >> > (2bytes)
> >>> >>>>>>>>>>      >     > > >> > >> >> keys
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> as
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > it
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > keeps it
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  simple.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  On the note
> of no
> >>> >>>>>>> headers,
> >>> >>>>>>>>> there
> >>> >>>>>>>>>>      > is as
> >>> >>>>>>>>>>      >     > per
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > kip
> >>> >>>>>>>>>>      >     > > >> > >> as
> >>> >>>>>>>>>>      >     > > >> > >> >> we
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > use an
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > attribute
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  bit to
> denote if
> >>> >>>>> headers
> >>> >>>>>>>>> are
> >>> >>>>>>>>>>      > present or
> >>> >>>>>>>>>>      >     > > not as
> >>> >>>>>>>>>>      >     > > >> > such
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > provides a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > zero
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  overhead
> >>> currently
> >>> >>> if
> >>> >>>>>>>>> headers are
> >>> >>>>>>>>>>      > not
> >>> >>>>>>>>>>      >     > used.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  I think as
> radai
> >>> >>>>> mentions
> >>> >>>>>>>>> would be
> >>> >>>>>>>>>>      > good
> >>> >>>>>>>>>>      >     > > first
> >>> >>>>>>>>>>      >     > > >> > if we
> >>> >>>>>>>>>>      >     > > >> > >> >> can
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> get
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > clarity if
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > do
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  we now have
> >>> general
> >>> >>>>>>>>> consensus that
> >>> >>>>>>>>>>      > (1)
> >>> >>>>>>>>>>      >     > > headers
> >>> >>>>>>>>>>      >     > > >> > are
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > worthwhile
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > useful,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  and (2) we
> want
> >>> it
> >>> >>> as
> >>> >>>>> a
> >>> >>>>>>>>> top level
> >>> >>>>>>>>>>      > entity.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Just to
> state the
> >>> >>>>>>> obvious i
> >>> >>>>>>>>>>      > believe (1)
> >>> >>>>>>>>>>      >     > > >> headers
> >>> >>>>>>>>>>      >     > > >> > are
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > worthwhile
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > and (2)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  agree as a
> top
> >>> level
> >>> >>>>>>>>> entity.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Cheers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Mike
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>> ______________________________
> >>> >>>>>>>>>>      > __________
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  From: Roger
> >>> Hoover <
> >>> >>>>>>>>>>      >     > roger.hoover@gmail.com
> >>> >>>>>>>>>>      >     > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Sent:
> Wednesday,
> >>> >>>>> November
> >>> >>>>>>>>> 9, 2016
> >>> >>>>>>>>>>      > 9:10:47
> >>> >>>>>>>>>>      >     > > PM
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  To:
> >>> >>>>> dev@kafka.apache.org
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Subject: Re:
> >>> >>> [DISCUSS]
> >>> >>>>>>>>> KIP-82 - Add
> >>> >>>>>>>>>>      >     > Record
> >>> >>>>>>>>>>      >     > > >> > Headers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Sorry for
> going a
> >>> >>>>> little
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>>      > weeds but
> >>> >>>>>>>>>>      >     > > >> thanks
> >>> >>>>>>>>>>      >     > > >> > >> for
> >>> >>>>>>>>>>      >     > > >> > >> >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> replies
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > regarding
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  varint.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Agreed that a
> >>> prefix
> >>> >>>>> and
> >>> >>>>>>>>> {int,
> >>> >>>>>>>>>>      > int} can
> >>> >>>>>>>>>>      >     > be
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > >> same.
> >>> >>>>>>>>>>      >     > > >> > >> >> It
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > doesn't
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > look
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > like
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  that's what
> the
> >>> KIP
> >>> >>> is
> >>> >>>>>>>>> saying the
> >>> >>>>>>>>>>      > "Open"
> >>> >>>>>>>>>>      >     > > >> > section.
> >>> >>>>>>>>>>      >     > > >> > >> The
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > example
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > shows
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  2100001
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  for New
> Relic and
> >>> >>>>> 210002
> >>> >>>>>>>>> for App
> >>> >>>>>>>>>>      > Dynamics
> >>> >>>>>>>>>>      >     > > >> > implying
> >>> >>>>>>>>>>      >     > > >> > >> >> that
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > New
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > Relic
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  organization
> will
> >>> >>> have
> >>> >>>>>>>>> only a
> >>> >>>>>>>>>>      > single
> >>> >>>>>>>>>>      >     > > header id
> >>> >>>>>>>>>>      >     > > >> > to
> >>> >>>>>>>>>>      >     > > >> > >> work
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > with. Or
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > is
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > 2100001
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  a prefix? The
> >>> main
> >>> >>>>> point
> >>> >>>>>>>>> of a
> >>> >>>>>>>>>>      > namespace
> >>> >>>>>>>>>>      >     > or
> >>> >>>>>>>>>>      >     > > >> > prefix
> >>> >>>>>>>>>>      >     > > >> > >> is
> >>> >>>>>>>>>>      >     > > >> > >> >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > reduce
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  overhead of
> >>> config
> >>> >>>>>>> mapping
> >>> >>>>>>>>> or
> >>> >>>>>>>>>>      >     > registration
> >>> >>>>>>>>>>      >     > > >> > >> depending
> >>> >>>>>>>>>>      >     > > >> > >> >> on
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> how
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> namespaces/prefixes
> >>> >>>>> are
> >>> >>>>>>>>> managed.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Would love to
> >>> hear
> >>> >>>>> more
> >>> >>>>>>>>> feedback
> >>> >>>>>>>>>>      > on the
> >>> >>>>>>>>>>      >     > > >> > >> higher-level
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > questions
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > though...
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Cheers,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Roger
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  On Wed, Nov
> 9,
> >>> 2016
> >>> >>> at
> >>> >>>>>>>>> 11:38 AM,
> >>> >>>>>>>>>>      > radai <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> radai.rosenblatt@gmail.com
> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > I think
> this
> >>> >>>>> discussion
> >>> >>>>>>>>> is
> >>> >>>>>>>>>>      > getting a
> >>> >>>>>>>>>>      >     > bit
> >>> >>>>>>>>>>      >     > > >> into
> >>> >>>>>>>>>>      >     > > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> weeds on
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > technical
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> implementation
> >>> >>>>> details.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > I'd liek to
> >>> step
> >>> >>>>> back a
> >>> >>>>>>>>> minute
> >>> >>>>>>>>>>      > and try
> >>> >>>>>>>>>>      >     > > and
> >>> >>>>>>>>>>      >     > > >> > >> establish
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > where we
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > are in
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > larger
> picture:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > (re-wording
> >>> >>> nacho's
> >>> >>>>>>> last
> >>> >>>>>>>>>>      > paragraph)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > 1. are we
> all
> >>> in
> >>> >>>>>>>>> agreement that
> >>> >>>>>>>>>>      > headers
> >>> >>>>>>>>>>      >     > > are
> >>> >>>>>>>>>>      >     > > >> a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> worthwhile
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > useful
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > addition to
> >>> have?
> >>> >>>>> this
> >>> >>>>>>>>> was
> >>> >>>>>>>>>>      > contested
> >>> >>>>>>>>>>      >     > > early
> >>> >>>>>>>>>>      >     > > >> on
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > 2. are we
> all
> >>> in
> >>> >>>>>>>>> agreement on
> >>> >>>>>>>>>>      > headers
> >>> >>>>>>>>>>      >     > as
> >>> >>>>>>>>>>      >     > > top
> >>> >>>>>>>>>>      >     > > >> > >> level
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> entity
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > vs
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > headers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> squirreled-away in
> >>> >>>>> V?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > if there
> are
> >>> still
> >>> >>>>>>>>> concerns
> >>> >>>>>>>>>>      > around
> >>> >>>>>>>>>>      >     > these
> >>> >>>>>>>>>>      >     > > #2
> >>> >>>>>>>>>>      >     > > >> > >> points
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> (#jay?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > #jun?)?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > (and now
> back
> >>> to
> >>> >>> our
> >>> >>>>>>>>> normal
> >>> >>>>>>>>>>      > programming
> >>> >>>>>>>>>>      >     > > ...)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > varints are
> >>> nice.
> >>> >>>>>>> having
> >>> >>>>>>>>> said
> >>> >>>>>>>>>>      > that, its
> >>> >>>>>>>>>>      >     > > >> adding
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> complexity
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> (see
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> >>>>>>>>> https://github.com/addthis/
> >>> >>>>>>>>>>      >     > > >> > >> >> stream-lib/blob/master/src/
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> >>>>>>>>> main/java/com/clearspring/
> >>> >>>>>>>>>>      >     > > >> > >> >> analytics/util/Varint.java
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > as 1st
> google
> >>> >>>>> result)
> >>> >>>>>>>>> and would
> >>> >>>>>>>>>>      > require
> >>> >>>>>>>>>>      >     > > >> anyone
> >>> >>>>>>>>>>      >     > > >> > >> >> writing
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > other
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > clients
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > (C?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > Python? Go?
> >>> Bash?
> >>> >>>>> ;-) )
> >>> >>>>>>>>> to
> >>> >>>>>>>>>>      >     > get/implement
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > >> same,
> >>> >>>>>>>>>>      >     > > >> > >> >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > relatively
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > little gain
> >>> (int
> >>> >>> vs
> >>> >>>>>>>>> string is
> >>> >>>>>>>>>>      > order of
> >>> >>>>>>>>>>      >     > > >> > magnitude,
> >>> >>>>>>>>>>      >     > > >> > >> >> this
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > isnt).
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > int
> >>> namespacing vs
> >>> >>>>>>> {int,
> >>> >>>>>>>>> int}
> >>> >>>>>>>>>>      >     > namespacing
> >>> >>>>>>>>>>      >     > > >> are
> >>> >>>>>>>>>>      >     > > >> > >> >> basically
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > same
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > thing -
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > youre just
> >>> >>>>> namespacing
> >>> >>>>>>>>> an int64
> >>> >>>>>>>>>>      > and
> >>> >>>>>>>>>>      >     > > giving
> >>> >>>>>>>>>>      >     > > >> > people
> >>> >>>>>>>>>>      >     > > >> > >> >> while
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > 2^32
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > ranges
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > at a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > time. the
> part
> >>> i
> >>> >>>>> like
> >>> >>>>>>>>> about this
> >>> >>>>>>>>>>      > is
> >>> >>>>>>>>>>      >     > > letting
> >>> >>>>>>>>>>      >     > > >> > >> people
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> have a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> large
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > swath of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > numbers
> with
> >>> one
> >>> >>>>>>>>> registration so
> >>> >>>>>>>>>>      > they
> >>> >>>>>>>>>>      >     > > dont
> >>> >>>>>>>>>>      >     > > >> > have
> >>> >>>>>>>>>>      >     > > >> > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> come
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > back
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > every
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > single
> >>> >>> plugin/header
> >>> >>>>>>>>> they want to
> >>> >>>>>>>>>>      >     > > "reserve".
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > On Wed,
> Nov 9,
> >>> >>> 2016
> >>> >>>>> at
> >>> >>>>>>>>> 11:01 AM,
> >>> >>>>>>>>>>      > Roger
> >>> >>>>>>>>>>      >     > > >> Hoover
> >>> >>>>>>>>>>      >     > > >> > <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> roger.hoover@gmail.com>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > Since
> some of
> >>> >>> the
> >>> >>>>>>>>> debate has
> >>> >>>>>>>>>>      > been
> >>> >>>>>>>>>>      >     > about
> >>> >>>>>>>>>>      >     > > >> > >> overhead +
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > performance, I'm
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> wondering if
> >>> we
> >>> >>>>> have
> >>> >>>>>>>>>>      > considered a
> >>> >>>>>>>>>>      >     > > varint
> >>> >>>>>>>>>>      >     > > >> > >> encoding
> >>> >>>>>>>>>>      >     > > >> > >> >> (
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> >>> >>>>>>>>> https://developers.google.com/
> >>> >>>>>>>>>>      >     > > >> > >> >> protocol-buffers/docs/
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > encoding#varints)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > the
> header
> >>> >>> length
> >>> >>>>>>>>> field (int32
> >>> >>>>>>>>>>      > in the
> >>> >>>>>>>>>>      >     > > >> > proposal)
> >>> >>>>>>>>>>      >     > > >> > >> >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > header
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > ids? If
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > you
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > don't use
> >>> >>> headers,
> >>> >>>>>>> the
> >>> >>>>>>>>>>      > overhead would
> >>> >>>>>>>>>>      >     > > be a
> >>> >>>>>>>>>>      >     > > >> > >> single
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> byte
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > for each
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > header
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > id < 128
> >>> would
> >>> >>>>> also
> >>> >>>>>>>>> need only a
> >>> >>>>>>>>>>      >     > single
> >>> >>>>>>>>>>      >     > > >> byte?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > On Wed,
> Nov
> >>> 9,
> >>> >>>>> 2016
> >>> >>>>>>> at
> >>> >>>>>>>>> 6:43 AM,
> >>> >>>>>>>>>>      >     > radai <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >
> radai.rosenblatt@gmail.com
> >>> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> @magnus -
> >>> and
> >>> >>>>> very
> >>> >>>>>>>>> dangerous
> >>> >>>>>>>>>>      > (youre
> >>> >>>>>>>>>>      >     > > >> > >> essentially
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > downloading and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > executing
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> arbitrary
> >>> code
> >>> >>>>> off
> >>> >>>>>>>>> the
> >>> >>>>>>>>>>      > internet on
> >>> >>>>>>>>>>      >     > > your
> >>> >>>>>>>>>>      >     > > >> > >> servers
> >>> >>>>>>>>>>      >     > > >> > >> >> ...
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > bad
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > idea
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > without
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> sandbox,
> >>> even
> >>> >>>>> with)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > as for
> it
> >>> >>> being
> >>> >>>>> a
> >>> >>>>>>>>> purely
> >>> >>>>>>>>>>      >     > > administrative
> >>> >>>>>>>>>>      >     > > >> > task
> >>> >>>>>>>>>>      >     > > >> > >> - i
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> disagree.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > i wish
> it
> >>> >>> would,
> >>> >>>>>>>>> really,
> >>> >>>>>>>>>>      > because
> >>> >>>>>>>>>>      >     > > then my
> >>> >>>>>>>>>>      >     > > >> > >> earlier
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > point on
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > complexity
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > the
> >>> remapping
> >>> >>>>>>>>> process would
> >>> >>>>>>>>>>      > be
> >>> >>>>>>>>>>      >     > > invalid,
> >>> >>>>>>>>>>      >     > > >> > but
> >>> >>>>>>>>>>      >     > > >> > >> at
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > linkedin,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > example,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > we
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > (the
> team
> >>> im
> >>> >>> in)
> >>> >>>>>>> run
> >>> >>>>>>>>> kafka
> >>> >>>>>>>>>>      > as a
> >>> >>>>>>>>>>      >     > > service.
> >>> >>>>>>>>>>      >     > > >> > we
> >>> >>>>>>>>>>      >     > > >> > >> dont
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > really
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > know
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > what our
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > users
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> (developing
> >>> >>>>>>>>> applications
> >>> >>>>>>>>>>      > that use
> >>> >>>>>>>>>>      >     > > kafka)
> >>> >>>>>>>>>>      >     > > >> > are
> >>> >>>>>>>>>>      >     > > >> > >> up
> >>> >>>>>>>>>>      >     > > >> > >> >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> at
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > any
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > given
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  moment.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > it
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > is very
> >>> >>> possible
> >>> >>>>>>>>> (given the
> >>> >>>>>>>>>>      >     > > existance of
> >>> >>>>>>>>>>      >     > > >> > >> headers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> and a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > corresponding
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > plugin
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> ecosystem)
> >>> for
> >>> >>>>> some
> >>> >>>>>>>>>>      > application to
> >>> >>>>>>>>>>      >     > > >> "equip"
> >>> >>>>>>>>>>      >     > > >> > >> their
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> producers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > consumers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > with
> the
> >>> >>>>> required
> >>> >>>>>>>>> plugin
> >>> >>>>>>>>>>      > without us
> >>> >>>>>>>>>>      >     > > >> > knowing.
> >>> >>>>>>>>>>      >     > > >> > >> i
> >>> >>>>>>>>>>      >     > > >> > >> >> dont
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > mean
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > to imply
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  thats
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > bad, i
> just
> >>> >>>>> want to
> >>> >>>>>>>>> make the
> >>> >>>>>>>>>>      > point
> >>> >>>>>>>>>>      >     > > that
> >>> >>>>>>>>>>      >     > > >> > its
> >>> >>>>>>>>>>      >     > > >> > >> not
> >>> >>>>>>>>>>      >     > > >> > >> >> as
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > simple
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > keeping it
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  in
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > sync
> >>> across a
> >>> >>>>>>>>> large-enough
> >>> >>>>>>>>>>      >     > > organization.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > On Wed,
> >>> Nov 9,
> >>> >>>>> 2016
> >>> >>>>>>>>> at 6:17
> >>> >>>>>>>>>>      > AM,
> >>> >>>>>>>>>>      >     > > Magnus
> >>> >>>>>>>>>>      >     > > >> > >> Edenhill
> >>> >>>>>>>>>>      >     > > >> > >> >> <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> magnus@edenhill.se>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > I
> think
> >>> >>> there
> >>> >>>>> is
> >>> >>>>>>> a
> >>> >>>>>>>>> piece
> >>> >>>>>>>>>>      > missing
> >>> >>>>>>>>>>      >     > in
> >>> >>>>>>>>>>      >     > > >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> Strings
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > discussion,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > where
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>> pro-Stringers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> reason
> >>> that
> >>> >>> by
> >>> >>>>>>>>> providing
> >>> >>>>>>>>>>      > unique
> >>> >>>>>>>>>>      >     > > string
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> identifiers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > each
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > header
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> everything
> >>> >>>>> will
> >>> >>>>>>>>> just
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> magically
> >>> >>> work
> >>> >>>>>>> for
> >>> >>>>>>>>> all
> >>> >>>>>>>>>>      > parts of
> >>> >>>>>>>>>>      >     > the
> >>> >>>>>>>>>>      >     > > >> > stream
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> pipeline.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > But
> the
> >>> >>>>> strings
> >>> >>>>>>>>> dont mean
> >>> >>>>>>>>>>      >     > anything
> >>> >>>>>>>>>>      >     > > by
> >>> >>>>>>>>>>      >     > > >> > >> >> themselves,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > while we
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  could
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> probably
> >>> >>>>> envision
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > some
> auto
> >>> >>>>> plugin
> >>> >>>>>>>>> loader
> >>> >>>>>>>>>>      > that
> >>> >>>>>>>>>>      >     > > >> downloads,
> >>> >>>>>>>>>>      >     > > >> > >> >> compiles,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > links
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > runs
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > plugins
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> on-demand
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > as
> soon
> >>> as
> >>> >>>>>>> they're
> >>> >>>>>>>>> seen by
> >>> >>>>>>>>>>      > a
> >>> >>>>>>>>>>      >     > > >> consumer, I
> >>> >>>>>>>>>>      >     > > >> > >> dont
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> really
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> see
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > use-case
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> something
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > so
> >>> dynamic
> >>> >>>>> (and
> >>> >>>>>>>>> fragile) in
> >>> >>>>>>>>>>      >     > > practice.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > In
> the
> >>> real
> >>> >>>>> world
> >>> >>>>>>>>> an
> >>> >>>>>>>>>>      > application
> >>> >>>>>>>>>>      >     > > will
> >>> >>>>>>>>>>      >     > > >> be
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> configured
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> with
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > a set
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > plugins
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > to
> either
> >>> >>> add
> >>> >>>>>>>>> (producer)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > or
> read
> >>> >>>>>>> (consumer)
> >>> >>>>>>>>> headers.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > This
> is
> >>> an
> >>> >>>>>>>>> administrative
> >>> >>>>>>>>>>      > task
> >>> >>>>>>>>>>      >     > > based
> >>> >>>>>>>>>>      >     > > >> on
> >>> >>>>>>>>>>      >     > > >> > >> what
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > features a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > client
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>> needs/provides
> >>> >>>>>>> and
> >>> >>>>>>>>> results
> >>> >>>>>>>>>>      > in
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > some
> >>> sort of
> >>> >>>>>>>>> configuration
> >>> >>>>>>>>>>      > to
> >>> >>>>>>>>>>      >     > > enable
> >>> >>>>>>>>>>      >     > > >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> configure
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > desired
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > plugins.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > Since
> >>> this
> >>> >>>>> needs
> >>> >>>>>>>>> to be kept
> >>> >>>>>>>>>>      >     > > somewhat
> >>> >>>>>>>>>>      >     > > >> in
> >>> >>>>>>>>>>      >     > > >> > >> sync
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> across
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > an
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > organisation
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > (there
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > is no
> >>> point
> >>> >>> in
> >>> >>>>>>>>> having
> >>> >>>>>>>>>>      > producers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > add
> >>> headers
> >>> >>> no
> >>> >>>>>>>>> consumers
> >>> >>>>>>>>>>      > will
> >>> >>>>>>>>>>      >     > read,
> >>> >>>>>>>>>>      >     > > >> and
> >>> >>>>>>>>>>      >     > > >> > >> vice
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> versa),
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > added
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> complexity
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > of
> >>> assigning
> >>> >>>>> an
> >>> >>>>>>> id
> >>> >>>>>>>>>>      > namespace
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > for
> each
> >>> >>>>> plugin
> >>> >>>>>>> as
> >>> >>>>>>>>> it is
> >>> >>>>>>>>>>      > being
> >>> >>>>>>>>>>      >     > > >> > configured
> >>> >>>>>>>>>>      >     > > >> > >> >> should
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> be
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > tolerable.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> /Magnus
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> 2016-11-09
> >>> >>>>> 13:06
> >>> >>>>>>>>> GMT+01:00
> >>> >>>>>>>>>>      >     > Michael
> >>> >>>>>>>>>>      >     > > >> > Pearce <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     >
> >>> Michael.Pearce@ig.com>:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> Just
> >>> >>>>>>>>> following/catching
> >>> >>>>>>>>>>      > up on
> >>> >>>>>>>>>>      >     > > what
> >>> >>>>>>>>>>      >     > > >> > seems
> >>> >>>>>>>>>>      >     > > >> > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> be
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> an
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > active
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > night :)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> @Radai
> >>> >>>>> sorry if
> >>> >>>>>>>>> it may
> >>> >>>>>>>>>>      > seem
> >>> >>>>>>>>>>      >     > > obvious
> >>> >>>>>>>>>>      >     > > >> > but
> >>> >>>>>>>>>>      >     > > >> > >> what
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> does
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > MD
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > stand
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > for?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > My
> >>> take on
> >>> >>>>>>>>> String vs Int:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > I
> will
> >>> >>> state
> >>> >>>>>>>>> first I am
> >>> >>>>>>>>>>      > pro Int
> >>> >>>>>>>>>>      >     > > (16
> >>> >>>>>>>>>>      >     > > >> or
> >>> >>>>>>>>>>      >     > > >> > >> 32).
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > I
> do
> >>> >>> though
> >>> >>>>>>>>> playing
> >>> >>>>>>>>>>      > devils
> >>> >>>>>>>>>>      >     > > advocate
> >>> >>>>>>>>>>      >     > > >> > see a
> >>> >>>>>>>>>>      >     > > >> > >> >> big
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> plus
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > with the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > argument
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> String
> >>> >>> keys,
> >>> >>>>>>>>> this is
> >>> >>>>>>>>>>      > around
> >>> >>>>>>>>>>      >     > > >> > integrating
> >>> >>>>>>>>>>      >     > > >> > >> >> into an
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > existing
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > eco-system.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > As
> many
> >>> >>>>> other
> >>> >>>>>>>>> systems use
> >>> >>>>>>>>>>      >     > String
> >>> >>>>>>>>>>      >     > > >> based
> >>> >>>>>>>>>>      >     > > >> > >> >> headers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> (Flume,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > JMS)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > it
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > makes
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > it
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> much
> >>> >>> easier
> >>> >>>>> for
> >>> >>>>>>>>> these to
> >>> >>>>>>>>>>      > be
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > incorporated/integrated
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > into.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > How
> >>> with
> >>> >>> Int
> >>> >>>>>>>>> based
> >>> >>>>>>>>>>      > headers
> >>> >>>>>>>>>>      >     > could
> >>> >>>>>>>>>>      >     > > we
> >>> >>>>>>>>>>      >     > > >> > >> provide
> >>> >>>>>>>>>>      >     > > >> > >> >> a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > way/guidence to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  make
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > this
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>> integration
> >>> >>>>>>>>> simple /
> >>> >>>>>>>>>>      > easy with
> >>> >>>>>>>>>>      >     > > >> > transition
> >>> >>>>>>>>>>      >     > > >> > >> >> flows
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > over
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > kafka?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > *
> tough
> >>> >>> luck
> >>> >>>>>>>>> buddy
> >>> >>>>>>>>>>      > you're on
> >>> >>>>>>>>>>      >     > your
> >>> >>>>>>>>>>      >     > > >> own
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > *
> >>> simply
> >>> >>>>> hash
> >>> >>>>>>>>> the string
> >>> >>>>>>>>>>      > into
> >>> >>>>>>>>>>      >     > int
> >>> >>>>>>>>>>      >     > > >> code
> >>> >>>>>>>>>>      >     > > >> > >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> hope
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> no
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > collisions
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > (how
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> convert
> >>> >>> back
> >>> >>>>>>>>> though?)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > *
> http2
> >>> >>>>> style
> >>> >>>>>>> as
> >>> >>>>>>>>>>      > mentioned by
> >>> >>>>>>>>>>      >     > > nacho.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> cheers,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> Mike
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      > ______________________________
> >>> >>>>>>>>>>      >     > > >> > __________
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> From:
> >>> >>> radai
> >>> >>>>> <
> >>> >>>>>>>>>>      >     > > >> > radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> Sent:
> >>> >>>>>>> Wednesday,
> >>> >>>>>>>>>>      > November 9,
> >>> >>>>>>>>>>      >     > 2016
> >>> >>>>>>>>>>      >     > > >> > 8:12 AM
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > To:
> >>> >>>>>>>>> dev@kafka.apache.org
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> Subject:
> >>> >>> Re:
> >>> >>>>>>>>> [DISCUSS]
> >>> >>>>>>>>>>      > KIP-82 -
> >>> >>>>>>>>>>      >     > > Add
> >>> >>>>>>>>>>      >     > > >> > >> Record
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> Headers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> thinking
> >>> >>>>> about
> >>> >>>>>>>>> it some
> >>> >>>>>>>>>>      > more,
> >>> >>>>>>>>>>      >     > the
> >>> >>>>>>>>>>      >     > > >> best
> >>> >>>>>>>>>>      >     > > >> > >> way to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > transmit
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > header
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> remapping
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> data to
> >>> >>>>>>>>> consumers would
> >>> >>>>>>>>>>      > be to
> >>> >>>>>>>>>>      >     > > put it
> >>> >>>>>>>>>>      >     > > >> > in
> >>> >>>>>>>>>>      >     > > >> > >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> MD
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> response
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > payload,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  so
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > maybe
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > it
> >>> should
> >>> >>> be
> >>> >>>>>>>>> discussed
> >>> >>>>>>>>>>      > now.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > On
> Wed,
> >>> >>> Nov
> >>> >>>>> 9,
> >>> >>>>>>>>> 2016 at
> >>> >>>>>>>>>>      > 12:09
> >>> >>>>>>>>>>      >     > AM,
> >>> >>>>>>>>>>      >     > > >> > radai <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> >>>>>>> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> im
> >>> not
> >>> >>>>>>> opposed
> >>> >>>>>>>>> to the
> >>> >>>>>>>>>>      > idea of
> >>> >>>>>>>>>>      >     > > >> > namespace
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> mapping.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> all
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > im
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > saying
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  is
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > that
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > its
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> not
> >>> part
> >>> >>>>> of
> >>> >>>>>>>>> the "mvp"
> >>> >>>>>>>>>>      > and,
> >>> >>>>>>>>>>      >     > > since
> >>> >>>>>>>>>>      >     > > >> it
> >>> >>>>>>>>>>      >     > > >> > >> >> requires
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> no
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> wire
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > format
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > change,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > can
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> >>> always
> >>> >>> be
> >>> >>>>>>>>> added later.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> also,
> >>> >>> its
> >>> >>>>> not
> >>> >>>>>>>>> as
> >>> >>>>>>>>>>      > simple as
> >>> >>>>>>>>>>      >     > just
> >>> >>>>>>>>>>      >     > > >> > >> >> configuring
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> MM
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> do
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > transform:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > lets
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> say
> >>> i've
> >>> >>>>>>>>> implemented
> >>> >>>>>>>>>>      > large
> >>> >>>>>>>>>>      >     > > message
> >>> >>>>>>>>>>      >     > > >> > >> >> support as
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > {666,1} and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > on
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  some
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> mirror
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> >>> target
> >>> >>>>>>> cluster
> >>> >>>>>>>>> its been
> >>> >>>>>>>>>>      >     > > remapped
> >>> >>>>>>>>>>      >     > > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> {999,1}.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > consumer
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  plugin
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > code
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> would
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> also
> >>> >>> need
> >>> >>>>> to
> >>> >>>>>>>>> be told
> >>> >>>>>>>>>>      > to look
> >>> >>>>>>>>>>      >     > > for
> >>> >>>>>>>>>>      >     > > >> the
> >>> >>>>>>>>>>      >     > > >> > >> large
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > message
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > "part X
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  Y"
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > header
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> under
> >>> >>>>>>> {999,1}.
> >>> >>>>>>>>> doable,
> >>> >>>>>>>>>>      > but
> >>> >>>>>>>>>>      >     > > tricky.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> On
> >>> Tue,
> >>> >>>>> Nov
> >>> >>>>>>> 8,
> >>> >>>>>>>>> 2016 at
> >>> >>>>>>>>>>      > 10:29
> >>> >>>>>>>>>>      >     > > PM,
> >>> >>>>>>>>>>      >     > > >> > Gwen
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> Shapira <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>
> >>> gwen@confluent.io
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> While
> >>> >>> you
> >>> >>>>>>> can
> >>> >>>>>>>>> do
> >>> >>>>>>>>>>      > whatever
> >>> >>>>>>>>>>      >     > you
> >>> >>>>>>>>>>      >     > > >> want
> >>> >>>>>>>>>>      >     > > >> > >> with a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> namespace
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > your
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > code,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> what
> >>> >>> I'd
> >>> >>>>>>>>> expect is
> >>> >>>>>>>>>>      > for each
> >>> >>>>>>>>>>      >     > > app
> >>> >>>>>>>>>>      >     > > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> namespaces
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > configurable...
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> So
> >>> if I
> >>> >>>>>>>>> accidentally
> >>> >>>>>>>>>>      > used
> >>> >>>>>>>>>>      >     > 666
> >>> >>>>>>>>>>      >     > > for
> >>> >>>>>>>>>>      >     > > >> > my
> >>> >>>>>>>>>>      >     > > >> > >> HR
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> department,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > still
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > want
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> run
> >>> >>>>>>> RadaiApp,
> >>> >>>>>>>>> I can
> >>> >>>>>>>>>>      > config
> >>> >>>>>>>>>>      >     > > >> > >> "namespace=42"
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> for
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > RadaiApp and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> everything
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> will
> >>> >>> look
> >>> >>>>>>>>> normal.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> This
> >>> >>>>> means
> >>> >>>>>>>>> you only
> >>> >>>>>>>>>>      > need to
> >>> >>>>>>>>>>      >     > > sync
> >>> >>>>>>>>>>      >     > > >> > usage
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> inside
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > your
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > own
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> organization.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> Still
> >>> >>>>> hard,
> >>> >>>>>>>>> but
> >>> >>>>>>>>>>      > somewhat
> >>> >>>>>>>>>>      >     > > easier
> >>> >>>>>>>>>>      >     > > >> > than
> >>> >>>>>>>>>>      >     > > >> > >> >> syncing
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > with
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > entire
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > world.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> On
> >>> Tue,
> >>> >>>>> Nov
> >>> >>>>>>>>> 8, 2016
> >>> >>>>>>>>>>      > at 10:07
> >>> >>>>>>>>>>      >     > > PM,
> >>> >>>>>>>>>>      >     > > >> > >> radai <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >
> >>> and
> >>> >>> we
> >>> >>>>> can
> >>> >>>>>>>>> start
> >>> >>>>>>>>>>      > with
> >>> >>>>>>>>>>      >     > > >> {namespace,
> >>> >>>>>>>>>>      >     > > >> > >> id}
> >>> >>>>>>>>>>      >     > > >> > >> >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> no
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > re-mapping
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > support
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> always
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >
> >>> add
> >>> >>> it
> >>> >>>>>>>>> later on
> >>> >>>>>>>>>>      > if/when
> >>> >>>>>>>>>>      >     > > >> > collisions
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> actually
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > happen (i
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > dont
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > think
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> they'd
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> be
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> > a
> >>> >>>>>>> problem).
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >
> >>> every
> >>> >>>>>>>>> interested
> >>> >>>>>>>>>>      > party (so
> >>> >>>>>>>>>>      >     > > orgs
> >>> >>>>>>>>>>      >     > > >> > or
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > individuals)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > could
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > then
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> register
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >
> >>> >>> prefix
> >>> >>>>> (0
> >>> >>>>>>> =
> >>> >>>>>>>>>>      > reserved, 1 =
> >>> >>>>>>>>>>      >     > > >> > confluent
> >>> >>>>>>>>>>      >     > > >> > >> ...
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> 666
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > = me
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > :-) )
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  do
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> whatever
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> with
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >
> >>> the
> >>> >>>>> 2nd ID
> >>> >>>>>>>>> - so once
> >>> >>>>>>>>>>      >     > > linkedin
> >>> >>>>>>>>>>      >     > > >> > >> >> registers,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> say
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > 3,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > then
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  linkedin
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > devs
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > are
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> free
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> > to
> >>> >>> use
> >>> >>>>> {3,
> >>> >>>>>>>>> *} with a
> >>> >>>>>>>>>>      >     > > reasonable
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> expectation
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > collide
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  with
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> >>> anything
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >
> >>> else.
> >>> >>>>>>>>> further
> >>> >>>>>>>>>>      > partitioning
> >>> >>>>>>>>>>      >     > > of
> >>> >>>>>>>>>>      >     > > >> > that *
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> becomes
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > linkedin's
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > problem,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > but
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >
> >>> >>>>> "upstream
> >>> >>>>>>>>>>      > registration"
> >>> >>>>>>>>>>      >     > of a
> >>> >>>>>>>>>>      >     > > >> > >> namespace
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> only
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > has
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > happen
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > once.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> > On
> >>> >>> Tue,
> >>> >>>>>>> Nov
> >>> >>>>>>>>> 8, 2016
> >>> >>>>>>>>>>      > at
> >>> >>>>>>>>>>      >     > 9:03
> >>> >>>>>>>>>>      >     > > PM,
> >>> >>>>>>>>>>      >     > > >> > >> James
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> Cheng <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > >
> >>> >>>>> wushujames@gmail.com
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> > On
> >>> >>>>> Nov
> >>> >>>>>>>>> 8, 2016,
> >>> >>>>>>>>>>      > at 5:54
> >>> >>>>>>>>>>      >     > > PM,
> >>> >>>>>>>>>>      >     > > >> > Gwen
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> Shapira <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  >
> >>> gwen@confluent.io
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>>>> Thank
> >>> >>>>>>>>> you so
> >>> >>>>>>>>>>      > much for
> >>> >>>>>>>>>>      >     > > this
> >>> >>>>>>>>>>      >     > > >> > clear
> >>> >>>>>>>>>>      >     > > >> > >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> fair
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > summary of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> arguments.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>> I'm
> >>> >>>>> in
> >>> >>>>>>>>> favor of
> >>> >>>>>>>>>>      > ints.
> >>> >>>>>>>>>>      >     > > Not a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> deal-breaker,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > but
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > in
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > favor.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>> Even
> >>> >>>>>>>>> more in
> >>> >>>>>>>>>>      > favor of
> >>> >>>>>>>>>>      >     > > >> Magnus's
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > decentralized
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > suggestion
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > with
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> Roger's
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>>>> tweak:
> >>> >>>>>>>>> add a
> >>> >>>>>>>>>>      > namespace
> >>> >>>>>>>>>>      >     > > for
> >>> >>>>>>>>>>      >     > > >> > >> headers.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> This
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > will
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > allow
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > each
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > app
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> just
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>> use
> >>> >>>>>>>>> whatever IDs
> >>> >>>>>>>>>>      > it
> >>> >>>>>>>>>>      >     > wants
> >>> >>>>>>>>>>      >     > > >> > >> >> internally,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> then
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > let
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > admin
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> >>> deploying
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>> the
> >>> >>>>> app
> >>> >>>>>>>>> figure
> >>> >>>>>>>>>>      > out an
> >>> >>>>>>>>>>      >     > > >> > available
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> namespace
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > ID
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > for the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > app
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > live
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > in.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> > So
> >>> >>>>>>>>>>      >     > > >> > io.confluent.schema-registry
> >>> >>>>>>>>>>      >     > > >> > >> can
> >>> >>>>>>>>>>      >     > > >> > >> >> be
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > namespace
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > 0x01 on
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  my
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> >>>>> deployment
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>> and
> >>> >>>>>>> 0x57
> >>> >>>>>>>>> on
> >>> >>>>>>>>>>      > yours, and
> >>> >>>>>>>>>>      >     > > the
> >>> >>>>>>>>>>      >     > > >> > poor
> >>> >>>>>>>>>>      >     > > >> > >> guys
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > developing the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > app
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > don't
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > need
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > to
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>>>> worry
> >>> >>>>>>>>> about that.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>> Gwen,
> >>> >>>>> if
> >>> >>>>>>> I
> >>> >>>>>>>>>>      > understand
> >>> >>>>>>>>>>      >     > your
> >>> >>>>>>>>>>      >     > > >> > example
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> right, an
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > application
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > deployer
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> might
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>> decide to
> >>> >>>>>>>>> use 0x01
> >>> >>>>>>>>>>      > in one
> >>> >>>>>>>>>>      >     > > >> > >> deployment,
> >>> >>>>>>>>>>      >     > > >> > >> >> and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > that
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > means
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > that
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > once
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> message
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> is
> >>> >>>>>>> written
> >>> >>>>>>>>> into the
> >>> >>>>>>>>>>      >     > > broker, it
> >>> >>>>>>>>>>      >     > > >> > >> will be
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > saved on
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > broker
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > with
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > that
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>> specific
> >>> >>>>>>>>> namespace
> >>> >>>>>>>>>>      >     > (0x01).
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> If
> >>> >>> you
> >>> >>>>>>>>> were to
> >>> >>>>>>>>>>      > mirror
> >>> >>>>>>>>>>      >     > that
> >>> >>>>>>>>>>      >     > > >> > message
> >>> >>>>>>>>>>      >     > > >> > >> >> into
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > another
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > cluster,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > 0x01
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> would
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>> accompany
> >>> >>>>>>>>> the
> >>> >>>>>>>>>>      > message,
> >>> >>>>>>>>>>      >     > > right?
> >>> >>>>>>>>>>      >     > > >> > What
> >>> >>>>>>>>>>      >     > > >> > >> if
> >>> >>>>>>>>>>      >     > > >> > >> >> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > deployers of
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > same
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > app
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > in
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>> other
> >>> >>>>>>>>> cluster uses
> >>> >>>>>>>>>>      > 0x57?
> >>> >>>>>>>>>>      >     > > They
> >>> >>>>>>>>>>      >     > > >> > won't
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > understand
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > each
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > other?
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> I'm
> >>> >>>>> not
> >>> >>>>>>>>> sure
> >>> >>>>>>>>>>      > that's an
> >>> >>>>>>>>>>      >     > > >> avoidable
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> problem. I
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > think it
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > simply
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > means
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> that
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> in
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>> order
> >>> >>>>> to
> >>> >>>>>>>>> share
> >>> >>>>>>>>>>      > data, you
> >>> >>>>>>>>>>      >     > > have
> >>> >>>>>>>>>>      >     > > >> to
> >>> >>>>>>>>>>      >     > > >> > >> also
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> have a
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > shared
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > (agreed
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > upon)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>>>>>> understanding of
> >>> >>>>>>>>>>      > what the
> >>> >>>>>>>>>>      >     > > >> > >> namespaces
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> mean.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> Which
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > I
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > think
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > makes
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> sense,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>> because
> >>> >>>>>>> the
> >>> >>>>>>>>>>      > alternate
> >>> >>>>>>>>>>      >     > > (sharing
> >>> >>>>>>>>>>      >     > > >> > >> >> *nothing*
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> at
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> all)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > would
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > mean
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > that
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > >
> there
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>> would
> >>> >>>>> be
> >>> >>>>>>>>> no way to
> >>> >>>>>>>>>>      >     > > understand
> >>> >>>>>>>>>>      >     > > >> > each
> >>> >>>>>>>>>>      >     > > >> > >> >> other.
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>> -James
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>> Gwen
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> > On
> >>> >>>>> Tue,
> >>> >>>>>>>>> Nov 8,
> >>> >>>>>>>>>>      > 2016 at
> >>> >>>>>>>>>>      >     > > 4:23
> >>> >>>>>>>>>>      >     > > >> > PM,
> >>> >>>>>>>>>>      >     > > >> > >> >> radai <
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > >
> >>> >>>>>>>>> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>>      > >
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>> wrote:
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>
> >>> >>> +1
> >>> >>>>> for
> >>> >>>>>>>>> sean's
> >>> >>>>>>>>>>      >     > document.
> >>> >>>>>>>>>>      >     > > it
> >>> >>>>>>>>>>      >     > > >> > >> covers
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> pretty
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> much
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > all
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > the
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > >
> trade-offs
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > and
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>
> >>> >>>>>>> provides
> >>> >>>>>>>>>>      > concrete
> >>> >>>>>>>>>>      >     > > figures
> >>> >>>>>>>>>>      >     > > >> to
> >>> >>>>>>>>>>      >     > > >> > >> argue
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> about
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > :-)
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>
> >>> >>
> >>> >>>>>>>>> (nit-picking -
> >>> >>>>>>>>>>      > used
> >>> >>>>>>>>>>      >     > the
> >>> >>>>>>>>>>      >     > > >> same
> >>> >>>>>>>>>>      >     > > >> > >> xkcd
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> twice,
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> also
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> > trove
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > has
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > been
> >>> >>>>>>>>>>      >     > > >> > >> >> >>> > >> >     > >>  > > > > > >>
> >>> >>>>> superceded
> >>> >>>>>>>>>>      >     > > >> > >> >
> >>> >>>>>>>>>>      >     > > >> >
> >>> >>>>>>>>>>      >     > > >> >
> >>> >>>>>>>>>>      >     > > >> >
> >>> >>>>>>>>>>      >     > > >> > --
> >>> >>>>>>>>>>      >     > > >> > Gwen Shapira
> >>> >>>>>>>>>>      >     > > >> > Product Manager | Confluent
> >>> >>>>>>>>>>      >     > > >> > 650.450.2760 <(650)%20450-2760> |
> @gwenshap
> >>> >>>>>>>>>>      >     > > >> > Follow us: Twitter | blog
> >>> >>>>>>>>>>      >     > > >> >
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >> --
> >>> >>>>>>>>>>      >     > > >> *Todd Palino*
> >>> >>>>>>>>>>      >     > > >> Staff Site Reliability Engineer
> >>> >>>>>>>>>>      >     > > >> Data Infrastructure Streaming
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > > >> linkedin.com/in/toddpalino
> >>> >>>>>>>>>>      >     > > >>
> >>> >>>>>>>>>>      >     > >
> >>> >>>>>>>>>>      >     > >
> >>> >>>>>>>>>>      >     > >
> >>> >>>>>>>>>>      >     > > --
> >>> >>>>>>>>>>      >     > > Gwen Shapira
> >>> >>>>>>>>>>      >     > > Product Manager | Confluent
> >>> >>>>>>>>>>      >     > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >>> >>>>>>>>>>      >     > > Follow us: Twitter | blog
> >>> >>>>>>>>>>      >     > >
> >>> >>>>>>>>>>      >     >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>      > The information contained in this email is strictly
> >>> >>>>> confidential
> >>> >>>>>>>>> and for
> >>> >>>>>>>>>>      > the use of the addressee only, unless otherwise
> >>> indicated.
> >>> >>> If
> >>> >>>>> you
> >>> >>>>>>>>> are not
> >>> >>>>>>>>>>      > the intended recipient, please do not read, copy,
> use or
> >>> >>>>> disclose
> >>> >>>>>>>>> to others
> >>> >>>>>>>>>>      > this message or any attachment. Please also notify
> the
> >>> >>> sender
> >>> >>>>> by
> >>> >>>>>>>>> replying
> >>> >>>>>>>>>>      > to this email or by telephone (+44(020 7896 0011) and
> >>> then
> >>> >>>>> delete
> >>> >>>>>>>>> the email
> >>> >>>>>>>>>>      > and any copies of it. Opinions, conclusion (etc)
> that do
> >>> not
> >>> >>>>>>>>> relate to the
> >>> >>>>>>>>>>      > official business of this company shall be
> understood as
> >>> >>>>> neither
> >>> >>>>>>>>> given nor
> >>> >>>>>>>>>>      > endorsed by it. IG is a trading name of IG Markets
> >>> Limited
> >>> >>> (a
> >>> >>>>>>>>> company
> >>> >>>>>>>>>>      > registered in England and Wales, company number
> 04008957)
> >>> >>> and
> >>> >>>>> IG
> >>> >>>>>>>>> Index
> >>> >>>>>>>>>>      > Limited (a company registered in England and Wales,
> >>> company
> >>> >>>>>>> number
> >>> >>>>>>>>>>      > 01190902). Registered address at Cannon Bridge
> House, 25
> >>> >>>>> Dowgate
> >>> >>>>>>>>> Hill,
> >>> >>>>>>>>>>      > London EC4R 2YA. Both IG Markets Limited (register
> number
> >>> >>>>> 195355)
> >>> >>>>>>>>> and IG
> >>> >>>>>>>>>>      > Index Limited (register number 114059) are
> authorised and
> >>> >>>>>>>>> regulated by the
> >>> >>>>>>>>>>      > Financial Conduct Authority.
> >>> >>>>>>>>>>      >
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> The information contained in this email is strictly
> confidential
> >>> >>> and
> >>> >>>>>>> for
> >>> >>>>>>>>> the use of the addressee only, unless otherwise indicated.
> If you
> >>> >>> are
> >>> >>>>>>> not
> >>> >>>>>>>>> the intended recipient, please do not read, copy, use or
> disclose
> >>> >>> to
> >>> >>>>>>> others
> >>> >>>>>>>>> this message or any attachment. Please also notify the
> sender by
> >>> >>>>>>> replying
> >>> >>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then
> delete
> >>> >>> the
> >>> >>>>>>> email
> >>> >>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not
> >>> relate
> >>> >>>>> to
> >>> >>>>>>> the
> >>> >>>>>>>>> official business of this company shall be understood as
> neither
> >>> >>>>> given
> >>> >>>>>>> nor
> >>> >>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
> >>> >>> company
> >>> >>>>>>>>> registered in England and Wales, company number 04008957)
> and IG
> >>> >>>>> Index
> >>> >>>>>>>>> Limited (a company registered in England and Wales, company
> >>> number
> >>> >>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> >>> >>>>> Hill,
> >>> >>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> >>> >>>>> and IG
> >>> >>>>>>>>> Index Limited (register number 114059) are authorised and
> >>> regulated
> >>> >>>>> by
> >>> >>>>>>> the
> >>> >>>>>>>>> Financial Conduct Authority.
> >>> >>>>>>>>>
> >>> >>>>>>>
> >>> >>>>>
> >>> >>> The information contained in this email is strictly confidential
> and
> >>> for
> >>> >>> the use of the addressee only, unless otherwise indicated. If you
> are
> >>> not
> >>> >>> the intended recipient, please do not read, copy, use or disclose
> to
> >>> others
> >>> >>> this message or any attachment. Please also notify the sender by
> >>> replying
> >>> >>> to this email or by telephone (+44(020 7896 0011) and then delete
> the
> >>> email
> >>> >>> and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> >>> the
> >>> >>> official business of this company shall be understood as neither
> given
> >>> nor
> >>> >>> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >>> >>> registered in England and Wales, company number 04008957) and IG
> Index
> >>> >>> Limited (a company registered in England and Wales, company number
> >>> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> >>> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> >>> IG
> >>> >>> Index Limited (register number 114059) are authorised and
> regulated by
> >>> the
> >>> >>> Financial Conduct Authority.
> >>> >>>
> >>> >
> >>> >
> >>>
> >>> The information contained in this email is strictly confidential and
> for
> >>> the use of the addressee only, unless otherwise indicated. If you are
> not
> >>> the intended recipient, please do not read, copy, use or disclose to
> others
> >>> this message or any attachment. Please also notify the sender by
> replying
> >>> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >>> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >>> official business of this company shall be understood as neither given
> nor
> >>> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >>> registered in England and Wales, company number 04008957) and IG Index
> >>> Limited (a company registered in England and Wales, company number
> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >>> Index Limited (register number 114059) are authorised and regulated by
> the
> >>> Financial Conduct Authority.
> >>>
> >
> >
> >
> > --
> > Nacho - Ignacio Solis - isolis@igso.net
> >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message