kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean McCauliff <sean.mccaul...@gmail.com>
Subject Re: [DISCUSS] KIP-82 - Add Record Headers
Date Mon, 23 Jan 2017 22:33:36 GMT
Scroll down to "Parse Header, create collection, populate collection
search collection"

https://docs.google.com/document/d/1tfT-6SZdnKOLyWGDH82kS30PnUkmgb7nPLdw6p65pAI/edit#heading=h.66o3tyythvas

If you want string keys this means accepting a header deserialization
time of 1.5-2.5x slower than having int keys. It's not super clear
what performance target one wants to hit but if the user needs to
access the header keys then header processing would be the limiting
CPU performance factor when 1M or so messages needs to be processed
per second and there are header 5-20 keys that need to be accessed.
This would be much lower if the accessing code actually needs to do
something the associated value.

If performance is really a concern then int keys make the most sense.
Compression will probably not make CPU utilization better.  Really
only a few people in any organization should care about the value of
these keys and that number may actually be zero if there is a large
third party community that provides solutions that uses the headers.
The only issue is then key namespace which is much better if people
could manage their own namespaces with strings.  Otherwise some kind
of namespace scheme with integers (or longs) needs to be created as
others have pointed out.

Preregistering headers means that all producers and consumers need to
see the same mapping.  This seems very problematic to get right with a
large Kafka deployment.

The lazy map thing is a good idea but is really a micro optimization.
This can be added later.

Rather than having a new type, like Headers , which involves more
object allocation, just have a method like byte[] header(int
headerKey) in ConsumerRecord that completely hides the implementation
of headers.

Regarding the need for detailed use cases for headers:  the use case
for headers is not mixing control (or metadata) with data; that's it.
 If I have metadata in my data then it's difficult to reuse code that
wants this data.  If I have data in my metadata then it's difficult to
reuse the control code that consumes the metadata.  This is a basic
concept that is is implemented in many protocols, serialized data
formats, file systems and more.

Sean


On Wed, Jan 11, 2017 at 9:05 AM, Jay Kreps <jay@confluent.io> wrote:
> Yeah I think I share a couple of concerns around the details of the
> proposal beyond the use cases:
>
>    1. I think numeric keys are just a non-starter for a user-facing
>    interface that is meant to work across third party components (not just
>    within an organization). We've gone down this path of making cryptic
>    interfaces before based on what's easy to implement (simple consumer!). The
>    result is a mess we are still trying to walk away from. I think we are
>    solving a compression/normalization problem by having the user implement a
>    compression algorithm in their head and share it across their company and
>    between components and plugin. This is definitely easier for us, but not
>    the right interface! I think we should consider instead (1) just take the
>    perf hit and claim it's okay since it's optional (I don't necessarily buy
>    that since it makes a lot of common use cases pretty questionable), (2)
>    work on getting more efficient compression in place so we can recommend
>    that without reservation (e.g. if that facebook lib pans out) or (3) allow
>    the client to optionally pre-register headers to get a more compact
>    numerical mapping.
>    2. I think we should think about creating the Map lazily to avoid
>    parsing out all the headers into little objects. HashMaps themselves are
>    kind of expensive and the consumer is very perf sensitive so and making
>    gazillions of hashmaps that may or may not get used is probably a bad idea.
>    3. We should think about having our own type, Headers or something
>    rather than a Map. I kind of dislike this kind of unnecessary type in
>    internal code, but for this kind of public API the ability to get just the
>    methods that make sense and document the details of the semantic can be
>    worthwhile.
>
> -Jay
>
> On Tue, Jan 10, 2017 at 10:38 PM, Ewen Cheslack-Postava <ewen@confluent.io>
> wrote:
>
>> I'm horribly behind and outdated on this thread, but wanted to mention one
>> other potential issue.
>>
>> The KIP talks about using simple Maps for the headers. For a lot of this
>> discussion, I tend to either think of completely opaque headers (i.e just
>> another byte[] blob, which seems to have been rejected based on
>> interoperability grounds but I think has the nice property of making it
>> easy to blame the implementer of the serde for any problems with the header
>> implementation) or HTTP-style headers (seemingly the most ubiquitous
>> example of headers, though certainly there are many others, and one which
>> we/users may need/want to interoperate with).
>>
>> Strictly speaking, the keys on HTTP headers aren't unique. See
>> https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2
>> specifically:
>>
>> > Multiple message-header fields with the same field-name MAY be present in
>> a message if and only if the entire field-value for that header field is
>> defined as a comma-separated list [i.e., #(values)]. It MUST be possible to
>> combine the multiple header fields into one "field-name: field-value" pair,
>> without changing the semantics of the message, by appending each subsequent
>> field-value to the first, each separated by a comma. The order in which
>> header fields with the same field-name are received is therefore
>> significant to the interpretation of the combined field value, and thus a
>> proxy MUST NOT change the order of these field values when a message is
>> forwarded.
>>
>> The spec guarantees that you can collapse repeated headers into a single
>> instance. But in practice there's a ton of stuff that doesn't. And every
>> robust HTTP library I've encountered uses a multimap to represent headers.
>>
>> I'm not necessarily advocating for supporting this, just suggesting that we
>> think about it. As you add these features and get closer to mapping to
>> other systems, people will inevitably try to map them. Headers are an area
>> where, if we're going to add them, it's worth considering compatibility as
>> someone will inevitably come and complain that system X does Y with headers
>> and we should also support Y because any decent system that provides
>> headers will do so.
>>
>> -Ewen
>>
>> On Tue, Jan 10, 2017 at 5:27 PM, Jay Kreps <jay@confluent.io> wrote:
>>
>> > Hey Gwen, yes, your approach is much better than what I described and
>> > solves one of my objections. I didn't think of doing it in MM rather than
>> > in the original producer. I don't think it adds any overhead beyond the
>> > additional header bytes if I'm not mistaken, since MM already unpacks and
>> > repacks messages (unlike the broker which tries to avoid repacking and
>> > hence can't efficiently add a header). I think my other objection around
>> > size is likely solvable too by some kind of aliasing scheme? If you guys
>> > think this can be made to work then I think we have two pretty credible
>> use
>> > cases with this and the tracing id family of uses, plus a number of
>> things
>> > that are at least moderately improved. If so, I'm convinced.
>> >
>> > -Jay
>> >
>> >
>> > On Tue, Jan 10, 2017 at 8:48 AM, Michael Pearce <Michael.Pearce@ig.com>
>> > wrote:
>> >
>> > > Hi Jay et al.
>> > >
>> > > On the replication using cluster id.
>> > >
>> > > Two options depending on how a company wants or needs to implement
>> their
>> > > replication as there are options of doing this as a mesh, star, or
>> ring.
>> > >
>> > > Either as some companies enforce a provider/custom producer/consumer
>> > > wrapper, it can be enforced the interceptors needed to be added, as you
>> > can
>> > > enforce to get support of the platform you must use company x’s
>> internal
>> > > wrappers.
>> > > An alternative approach is to do this at the mirror maker, as Gwen
>> noted
>> > > we can actually make the mirror maker append this, like wise for a more
>> > > complex implementation, where you can have multiple hops/routes this
>> can
>> > > also be the message routing.
>> > >
>> > > Re the size of the clusterId, yes the auto generated one is “cough”
>> > > hideous in size.
>> > >
>> > > Most organisations though will only need a space to uniquely identify
>> 127
>> > > (byte) clusters, or at worst 2 bytes (short) they need to mirror and
>> > > replicate between, and as such can either as platform teams generally
>> > will
>> > > roll out the mirror makers / replicators with the cluster can either
>> > > manually adjust the clusterId on ZK, or use a mapping file all their
>> > mirror
>> > > makers have to map the larger id to a smaller one.
>> > >
>> > > Also as Gwen states this is something you should be able to opt into or
>> > > not, this is the benefit of headers, if you don’t use it, it doesn’t
>> use
>> > up
>> > > the space, its entirely optional.
>> > >
>> > > Regards
>> > > Mike
>> > >
>> > >
>> > > On 09/01/2017, 19:40, "Gwen Shapira" <gwen@confluent.io> wrote:
>> > >
>> > >     Mirroring + ClusterIDs:
>> > >
>> > >     Kafka clusters already have an automatically generated ID that is
>> > >     exposed to clients:
>> > >     https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 78%3A+Cluster+Id
>> > >
>> > >     It sounds trivial to get MirrorMaker's consumer to grab this ID
>> > (which
>> > >     always exists) from the source cluster and add it to the message
>> > >     headers.
>> > >
>> > >     Obviously adding stuff to the message makes it larger so this
>> should
>> > >     be a feature you should be able to turn on/off depending on the
>> > >     replication requirements. If you need the feature - you pay the
>> > price.
>> > >     I don't think there is an immediate need to put an effort into
>> > >     shrinking the Cluster ID, but others can comment on that.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >     On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <jay@confluent.io>
>> wrote:
>> > >     > Hey Michael,
>> > >     >
>> > >     > Sorry for lagging, I've been out for the holiday break. What
>> would
>> > > help me
>> > >     > would be to get to a consolidated list of use cases we've worked
>> > out
>> > > and
>> > >     > think add real value. I think getting down to the details really
>> > > matters
>> > >     > for these.
>> > >     >
>> > >     > The two lists I'm aware of are the one in the KIP and the one you
>> > > gave in
>> > >     > this email thread. Both seem to be more at the sketch level. Let
>> me
>> > > review
>> > >     > the use cases and see if I have it right. If so let's get it in
>> the
>> > > design
>> > >     > proposal.
>> > >     >
>> > >     > Your use cases:
>> > >     >
>> > >     >    1. Tracing IDs -- I get this use case and think I see how it
>> > > would work.
>> > >     >    I think because the tracing is from a third party system there
>> > is
>> > > some
>> > >     >    value to having a header versus a custom format to make this
>> > work
>> > > fully
>> > >     >    transparently.
>> > >     >    2. Mirroring -- I'm a bit unsure how this will work, I'll
>> > explain
>> > > my
>> > >     >    thinking below
>> > >     >    3. End-to-end encryption -- I think what you are saying is
>> that
>> > > if you
>> > >     >    want "routing' fields that are accessible in transit without
>> > > decrypting you
>> > >     >    need a scheme to leave this decrypted.I think the idea is to
>> > avoid
>> > >     >    decrypting to get a header, but I'm not sure if I get how this
>> > is
>> > > better
>> > >     >    than a custom message format that doesn't encrypt the things
>> you
>> > > don't want
>> > >     >    encrypted since you have to be aware of the scheme anyway.
>> > >     >    4. EOS -- I'm not convinced this is a good approach for EOS.
>> > >     >
>> > >     > Linked from KIP:
>> > >     >
>> > >     >    1. Schema-Registry-aware avro serialization -- not clear how
>> > this
>> > > is
>> > >     >    better than keeping the schema id in the value
>> > >     >    2. Polyglot kafka topics -- marginally better than a container
>> > > format,
>> > >     >    but of course you still have to all agree on the the header
>> and
>> > >     >    header=>serialization mapping so not sure if it's really
>> better
>> > > than a
>> > >     >    container format. Also not wild about polyglot topics :-)
>> > >     >    3. End-to-End routing trail including mirroring --- I'd put
>> this
>> > > under
>> > >     >    tracing ids
>> > >     >    4. Cross-service process tracing -- ditto
>> > >     >    5. Signing -- I like this use case but it seems to depend on a
>> > > feature
>> > >     >    we don't have and aren't proposing adding, right?
>> > >     >    6. Per-record custom compression -- why do you want this?
>> > > compressing
>> > >     >    per message won't work well compared to batch compression,
>> > right?
>> > >     >    7. Large Message Support -- I think we can agree that although
>> > > this
>> > >     >    might work it's clearly not the right way to support large
>> > > messages in Kafka
>> > >     >
>> > >     > Matthias
>> > >     >
>> > >     >    1. Control messages --- Matthias, not sure if I get the
>> > > motivation in
>> > >     >    streams for this. If we did want to do it, wouldn't it be
>> easier
>> > > to
>> > >     >    implement and by allowing users to send control messages (in
>> the
>> > > EOS sense)
>> > >     >    and adding a consumer config to allow either passing these on
>> or
>> > >     >    suppressing them?
>> > >     >
>> > >     > So I if I were to summarize I'd list the use cases as follows:
>> > >     >
>> > >     >    - Much better with headers: tracing IDs
>> > >     >    - Somewhat better: end-to-end encryption, polyglot topics
>> > >     >    - Don't buy: Mirroring (though perhaps could be made to work
>> > with
>> > >     >    changes to the proposal), EOS, large messages, compression
>> > >     >    - Not sure: Control messages
>> > >     >
>> > >     > Does this match where other people are at? From my point of view
>> > for
>> > >     > something like this where we are changing the core data model we
>> > > should
>> > >     > ideally get a handful of use cases to the point where they are
>> > > really well
>> > >     > thought out and we know they can use the feature. I think we have
>> > > one solid
>> > >     > one with tracing IDs, and I think maybe mirroring could be
>> another
>> > > with
>> > >     > some work. I'm less sure about the others.
>> > >     >
>> > >     > Bi-directional Mirroring
>> > >     > I think bidirectional mirroring could potentially become a
>> > > compelling use
>> > >     > case if we an find a way to make it work. If I understand it, the
>> > > proposal
>> > >     > is to add a header like "origin-cluster" with the cluster id of
>> the
>> > > Kafka
>> > >     > cluster. Then modify the mirroring tool to disallow mirroring a
>> > > message
>> > >     > back to it's own origin cluster. This basic idea is definitely
>> > > workable and
>> > >     > useful, but can we really get to a good implementation using
>> > headers?
>> > >     > I imagine the approach would be to add this header in the
>> producer
>> > > if some
>> > >     > config is set. There are two problems:
>> > >     >
>> > >     >    - You add a pretty big honking cluster ID to each message. For
>> > > small
>> > >     >    messages this maybe larger than the contents itself.
>> > >     >    - What happens if someone somewhere forgets to set that config
>> > on
>> > > some
>> > >     >    client somewhere?
>> > >     >
>> > >     > If someone forgets to set the config in some client somewhere I
>> > > guess you
>> > >     > have various options in mirror maker:
>> > >     >
>> > >     >    1. drop the data (pretty bad, now your data is out of sync)
>> > >     >    2. crash (pretty bad--how do you recover)
>> > >     >    3. insert some default (pretty bad since would also lead to
>> data
>> > > loss if
>> > >     >    that's not right)
>> > >     >
>> > >     > Basically although I think a cluster tag could be a good way to
>> do
>> > >     > bi-directional mirroring, I'm not sure this naive approach I
>> > > described
>> > >     > would get us to a usable implementation. This could be fixable. I
>> > > think if
>> > >     > there was some scheme to check required headers on the server
>> that
>> > > could
>> > >     > alleviate the missing header problem. Not really sure what to do
>> > > about the
>> > >     > message bloat problem, but maybe there could be some way to
>> shrink
>> > > the
>> > >     > cluster id?
>> > >     >
>> > >     > Cheers,
>> > >     >
>> > >     > -Jay
>> > >     >
>> > >     >
>> > >     > On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <
>> > > Michael.Pearce@ig.com>
>> > >     > wrote:
>> > >     >
>> > >     >> @Jay, Ismael and a few others.
>> > >     >>
>> > >     >> Based on the discussion on kip-87, and the additional use case
>> by
>> > > Matthias.
>> > >     >>
>> > >     >> Just for clarity do we now have consensus on the earlier points
>> > (1)
>> > >     >> headers are useful (2) headers should be native in Kafka?
>> > >     >>
>> > >     >> Cheers
>> > >     >> Mike
>> > >     >> ________________________________________
>> > >     >> From: radai <radai.rosenblatt@gmail.com>
>> > >     >> Sent: Monday, December 19, 2016 5:01 PM
>> > >     >> To: dev@kafka.apache.org
>> > >     >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>> > >     >>
>> > >     >> having said that im perfectly fine with something like a "no
>> > > payload" flag
>> > >     >> (as opposed to null payload), which would mean such a msg is not
>> > > returned
>> > >     >> from poll() unless the user code specifically "opts in" and
>> would
>> > > serve
>> > >     >> only to carry headers (making it a control message)
>> > >     >>
>> > >     >> On Mon, Dec 19, 2016 at 8:58 AM, radai <
>> > radai.rosenblatt@gmail.com>
>> > > wrote:
>> > >     >>
>> > >     >> > if all you want is to tag "end of stream" wouldnt a header on
>> > the
>> > > last
>> > >     >> msg
>> > >     >> > in the stream be enough?
>> > >     >> >
>> > >     >> > On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
>> > > matthias@confluent.io>
>> > >     >> > wrote:
>> > >     >> >
>> > >     >> >> The details about headers for control messages are still to
>> > > define. But
>> > >     >> >> yes, the idea is to have some common default behavior that
>> > > clients would
>> > >     >> >> need to implement.
>> > >     >> >>
>> > >     >> >> The point is, that "regular headers" add meta data to regular
>> > > messages.
>> > >     >> >> Thus, those messages will be returned to the user via
>> .poll().
>> > > And after
>> > >     >> >> the message is received the user can check if meta data is
>> > > present and
>> > >     >> >> read it.
>> > >     >> >>
>> > >     >> >> For control messages, we do not want those to pop up via
>> > .poll()
>> > > as
>> > >     >> >> those are no regular messages. A client would need to opt-in
>> to
>> > > see
>> > >     >> >> those messages (either via poll() or maybe a callback). Thus,
>> > we
>> > > need
>> > >     >> >> some special (standardized) header IDs that indicate control
>> > > messages
>> > >     >> >> that should not be returned to the user via poll() by
>> default.
>> > >     >> >>
>> > >     >> >>
>> > >     >> >> -Matthias
>> > >     >> >>
>> > >     >> >> On 12/17/16 9:37 PM, Roger Hoover wrote:
>> > >     >> >> > Matthias,
>> > >     >> >> >
>> > >     >> >> > Thanks for your input.  I'm +1 on control messages as they
>> > > seem to be
>> > >     >> >> the
>> > >     >> >> > simplest way to implement watermarks (
>> > >     >> >> > https://www.oreilly.com/ideas/the-world-beyond-batch-
>> > > streaming-102),
>> > >     >> a
>> > >     >> >> > feature that would add a lot of value to Kafka Streams
>> IMHO.
>> > >     >> >> >
>> > >     >> >> > Your argument that the control-message indicator belongs in
>> > the
>> > >     >> >> client-only
>> > >     >> >> > section of the record format make sense.  Just to make
>> sure I
>> > >     >> >> understand,
>> > >     >> >> > are you suggesting that control messages would be indicated
>> > by
>> > > a
>> > >     >> >> standard
>> > >     >> >> > reserved header?  By standard, I mean that ALL Kafka
>> > consumers
>> > > would
>> > >     >> >> know
>> > >     >> >> > to handle these messages differently (possibly just
>> ignoring
>> > > them).
>> > >     >> >> This
>> > >     >> >> > would need to be added to the specification of the consumer
>> > > protocol
>> > >     >> so
>> > >     >> >> > that all Kafka clients implement it, right?  I think it's a
>> > > good idea
>> > >     >> >> but
>> > >     >> >> > just checking.
>> > >     >> >> >
>> > >     >> >> > Cheers,
>> > >     >> >> >
>> > >     >> >> > Roger
>> > >     >> >> >
>> > >     >> >> >
>> > >     >> >> > On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
>> > >     >> matthias@confluent.io
>> > >     >> >> >
>> > >     >> >> > wrote:
>> > >     >> >> >
>> > >     >> >> >> Yes and no. I did overload the term "control message".
>> > >     >> >> >>
>> > >     >> >> >> EOS control messages are for client-broker communication
>> and
>> > > thus
>> > >     >> never
>> > >     >> >> >> exposed to any application. And I think this is a good
>> > design
>> > > because
>> > >     >> >> >> broker needs to understand those control messages. Thus,
>> > this
>> > > should
>> > >     >> be
>> > >     >> >> >> a protocol change.
>> > >     >> >> >>
>> > >     >> >> >> The type of control messages I have in mind are for
>> > > client-client
>> > >     >> >> >> (application-application) communication and the broker is
>> > > agnostic to
>> > >     >> >> >> them. Thus, it should not be a protocol change.
>> > >     >> >> >>
>> > >     >> >> >>
>> > >     >> >> >> -Matthias
>> > >     >> >> >>
>> > >     >> >> >>
>> > >     >> >> >>
>> > >     >> >> >> On 12/14/16 9:42 AM, radai wrote:
>> > >     >> >> >>> arent control messages getting pushed as their own top
>> > level
>> > >     >> protocol
>> > >     >> >> >>> change (and a fairly massive one) for the transactions
>> KIP
>> > ?
>> > >     >> >> >>>
>> > >     >> >> >>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
>> > >     >> >> matthias@confluent.io>
>> > >     >> >> >>> wrote:
>> > >     >> >> >>>
>> > >     >> >> >>>> Hi,
>> > >     >> >> >>>>
>> > >     >> >> >>>> I want to add a completely new angle to this discussion.
>> > > For this,
>> > >     >> I
>> > >     >> >> >>>> want to propose an extension for the headers feature
>> that
>> > > enables
>> > >     >> new
>> > >     >> >> >>>> uses cases -- and those new use cases might convince
>> > people
>> > > to
>> > >     >> >> support
>> > >     >> >> >>>> headers (of course including the larger scoped
>> proposal).
>> > >     >> >> >>>>
>> > >     >> >> >>>> Extended Proposal:
>> > >     >> >> >>>>
>> > >     >> >> >>>> Allow messages with a certain header key to be special
>> > > "control
>> > >     >> >> >>>> messages" (w/ o w/o payload) that are not exposed to an
>> > > application
>> > >     >> >> via
>> > >     >> >> >>>> .poll().
>> > >     >> >> >>>>
>> > >     >> >> >>>> Thus, a consumer client would automatically skip over
>> > those
>> > >     >> >> messages. If
>> > >     >> >> >>>> an application knows about embedded control messages, it
>> > > can "sing
>> > >     >> >> up"
>> > >     >> >> >>>> to those messages by the consumer client and either get
>> a
>> > > callback
>> > >     >> or
>> > >     >> >> >>>> the consumer auto-drop for this messages gets disabled
>> > > (allowing to
>> > >     >> >> >>>> consumer those messages via poll()).
>> > >     >> >> >>>>
>> > >     >> >> >>>> (The details need further considerations/discussion. I
>> > just
>> > > want to
>> > >     >> >> >>>> sketch the main idea.)
>> > >     >> >> >>>>
>> > >     >> >> >>>> Usage:
>> > >     >> >> >>>>
>> > >     >> >> >>>> There is a shared topic (ie, used by multiple
>> > applications)
>> > > and a
>> > >     >> >> >>>> producer application wants to embed a special message in
>> > > the topic
>> > >     >> >> for a
>> > >     >> >> >>>> dedicated consumer application. Because only one
>> > > application will
>> > >     >> >> >>>> understand this message, it cannot be a regular message
>> as
>> > > this
>> > >     >> would
>> > >     >> >> >>>> break all applications that do not understand this
>> > message.
>> > > The
>> > >     >> >> producer
>> > >     >> >> >>>> application would set a special metadata key and no
>> > consumer
>> > >     >> >> application
>> > >     >> >> >>>> would see this control message by default because they
>> did
>> > > not
>> > >     >> enable
>> > >     >> >> >>>> their consumer client to return this message in poll()
>> > (and
>> > > the
>> > >     >> >> client
>> > >     >> >> >>>> would just drop this message with special metadata key).
>> > > Only the
>> > >     >> >> single
>> > >     >> >> >>>> application that should receive this message, will
>> > > subscribe to
>> > >     >> this
>> > >     >> >> >>>> message on its consumer client and process it.
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>> Concrete Use Case: Kafka Streams
>> > >     >> >> >>>>
>> > >     >> >> >>>> In Kafka Streams, we would like to propagate "control
>> > > messages"
>> > >     >> from
>> > >     >> >> >>>> subtopology to subtopology. There are multiple scenarios
>> > > for which
>> > >     >> >> this
>> > >     >> >> >>>> would be useful. For example, currently we do not
>> > guarantee
>> > > a
>> > >     >> >> >>>> "consistent shutdown" of an application. By this, I mean
>> > > that input
>> > >     >> >> >>>> records might not be completely processed by the whole
>> > > topology
>> > >     >> >> because
>> > >     >> >> >>>> the application shutdown happens "in between" and an
>> > > intermediate
>> > >     >> >> result
>> > >     >> >> >>>> topic gets "stock" in an intermediate topic. Thus, a
>> user
>> > > would see
>> > >     >> >> an
>> > >     >> >> >>>> committed offset of the source topic of the application,
>> > > but no
>> > >     >> >> >>>> corresponding result record in the output topic.
>> > >     >> >> >>>>
>> > >     >> >> >>>> Having "shutdown markers" would allow us, to first stop
>> > the
>> > >     >> upstream
>> > >     >> >> >>>> subtopology and write this marker into the intermediate
>> > > topic and
>> > >     >> the
>> > >     >> >> >>>> downstream subtopology would only shut down itself after
>> > is
>> > > sees
>> > >     >> the
>> > >     >> >> >>>> "shutdown marker". Thus, we can guarantee on shutdown,
>> > that
>> > > no
>> > >     >> >> >>>> "in-flight" messages got stuck in intermediate topics.
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>> A similar usage would be for KIP-95 (Incremental Batch
>> > > Processing).
>> > >     >> >> >>>> There was a discussion about the proposed metadata
>> topic,
>> > > and we
>> > >     >> >> could
>> > >     >> >> >>>> avoid this metadata topic if we would have "control
>> > > messages".
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>> Right now, we cannot insert an "application control
>> > message"
>> > >     >> because
>> > >     >> >> >>>> Kafka Streams does not own all topics it read/writes and
>> > > thus might
>> > >     >> >> >>>> break other consumer application (as described above) if
>> > we
>> > > inject
>> > >     >> >> >>>> random messages that are not understood by other apps.
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>> Of course, one can work around "embedded control
>> messaged"
>> > > by using
>> > >     >> >> an
>> > >     >> >> >>>> additional topic to propagate control messaged between
>> > > application
>> > >     >> >> (as
>> > >     >> >> >>>> suggestion in KIP-95 via a metadata topic for Kafka
>> > > Streams). But
>> > >     >> >> there
>> > >     >> >> >>>> are major concerns about adding this metadata topic in
>> the
>> > > KIP and
>> > >     >> >> this
>> > >     >> >> >>>> shows that other application that need a similar pattern
>> > > might
>> > >     >> profit
>> > >     >> >> >>>> from topic embedded "control messages", too.
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>> One last important consideration: those "control
>> messages"
>> > > are used
>> > >     >> >> for
>> > >     >> >> >>>> client to client communication and are not understood by
>> > the
>> > >     >> broker.
>> > >     >> >> >>>> Thus, those messages should not be enabled within the
>> > > message
>> > >     >> format
>> > >     >> >> >>>> (c.f. tombstone flag -- KIP-87). However, "client land"
>> > > record
>> > >     >> >> headers
>> > >     >> >> >>>> would be a nice way to implement them. Because KIP-82
>> did
>> > > consider
>> > >     >> >> key
>> > >     >> >> >>>> namespaces for metatdata keys, this extension should not
>> > be
>> > > an own
>> > >     >> >> KIP
>> > >     >> >> >>>> but should be included in KIP-82 to reserve a namespace
>> > for
>> > >     >> "control
>> > >     >> >> >>>> message" in the first place.
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>> Sorry for the long email... Looking forward to your
>> > > feedback.
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>> -Matthias
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
>> > >     >> >> >>>>> Hi Jun
>> > >     >> >> >>>>>
>> > >     >> >> >>>>> 100) each time a transaction exits a jvm for a remote
>> > > system
>> > >     >> >> (HTTP/JMS/
>> > >     >> >> >>>> Hopefully one day kafka) the APM tools stich in a unique
>> > id
>> > >     >> (though I
>> > >     >> >> >>>> believe it contains the end2end uuid embedded in this
>> id),
>> > > on
>> > >     >> >> receiving
>> > >     >> >> >> the
>> > >     >> >> >>>> message at the receiving JVM the apm code takes this
>> out,
>> > > and
>> > >     >> >> continues
>> > >     >> >> >> its
>> > >     >> >> >>>> tracing on the that new thread. Both JVM’s (and other
>> > > languages the
>> > >     >> >> APM
>> > >     >> >> >>>> tool supports) send this data async back to the central
>> > > controllers
>> > >     >> >> >> where
>> > >     >> >> >>>> the stiching togeather occurs. For this they need some
>> > > header space
>> > >     >> >> for
>> > >     >> >> >>>> them to put this id.
>> > >     >> >> >>>>>
>> > >     >> >> >>>>> 101) Yes indeed we have a business transaction Id in
>> the
>> > > payload.
>> > >     >> >> >> Though
>> > >     >> >> >>>> this is a system level tracing, that we need to have
>> marry
>> > > up. Also
>> > >     >> >> as
>> > >     >> >> >> per
>> > >     >> >> >>>> note on end2end encryption we’d be unable to prove the
>> > flow
>> > > if the
>> > >     >> >> >> payload
>> > >     >> >> >>>> is encrypted as we’d not have access to this at certain
>> > > points of
>> > >     >> the
>> > >     >> >> >> flow
>> > >     >> >> >>>> through the infrastructure/platform.
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>
>> > >     >> >> >>>>> 103) As said we use this mechanism in IG very
>> > > successfully, as
>> > >     >> >> stated
>> > >     >> >> >>>> per key we guarantee the transaction producing app to
>> > > handle the
>> > >     >> >> >>>> transaction of a key at one DC unless at point of
>> critical
>> > > failure
>> > >     >> >> >> where we
>> > >     >> >> >>>> have to flip processing to another. We care about key
>> > > ordering.
>> > >     >> >> >>>>> I disagree on the offset comment for the partition
>> > > solution unless
>> > >     >> >> you
>> > >     >> >> >>>> do full ISR, or expensive full XA transactions even with
>> > > partitions
>> > >     >> >> you
>> > >     >> >> >>>> cannot fully guarantee offsets would match.
>> > >     >> >> >>>>>
>> > >     >> >> >>>>> 105) Very much so, I need to have access at the
>> platform
>> > > level to
>> > >     >> >> the
>> > >     >> >> >>>> other meta data all mentioned, without having to need to
>> > > have
>> > >     >> access
>> > >     >> >> to
>> > >     >> >> >> the
>> > >     >> >> >>>> encryption keys of the payload.
>> > >     >> >> >>>>>
>> > >     >> >> >>>>> 106)
>> > >     >> >> >>>>> Techincally yes for AZ/Region/Cluster, but then we’d
>> need
>> > > to have
>> > >     >> a
>> > >     >> >> >>>> global producerId register which would be very hard to
>> > >     >> >> enforce/ensure is
>> > >     >> >> >>>> current and correct, just to understand the message
>> > origins
>> > > of its
>> > >     >> >> >>>> region/az/cluster for routing.
>> > >     >> >> >>>>> The client wrapper version, producerId can be the same,
>> > as
>> > >     >> obviously
>> > >     >> >> >> the
>> > >     >> >> >>>> producer could upgrade its wrapper, as such we need to
>> > know
>> > > what
>> > >     >> >> wrapper
>> > >     >> >> >>>> version the message is created with.
>> > >     >> >> >>>>> Likewise the IP address, as stated we can have our
>> > > producer move,
>> > >     >> >> where
>> > >     >> >> >>>> its IP would change.
>> > >     >> >> >>>>>
>> > >     >> >> >>>>> 107)
>> > >     >> >> >>>>> UUID is set on the message by interceptors before
>> actual
>> > > producer
>> > >     >> >> >>>> transport send. This is for platform level message
>> dedupe
>> > >     >> guarantee,
>> > >     >> >> the
>> > >     >> >> >>>> business payload should be agnostic to this. Please see
>> > >     >> >> >>>> https://activemq.apache.org/
>> artemis/docs/1.5.0/duplicate-
>> > > det
>> > >     >> >> ection.html
>> > >     >> >> >>>> note this is not touching business payloads.
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>
>> > >     >> >> >>>>> On 06/12/2016, 18:22, "Jun Rao" <jun@confluent.io>
>> > wrote:
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     Hi, Michael,
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     Thanks for the reply. I find it very helpful.
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     Data lineage:
>> > >     >> >> >>>>>     100. I'd like to understand the APM use case a bit
>> > > more. It
>> > >     >> >> sounds
>> > >     >> >> >>>> like
>> > >     >> >> >>>>>     that those APM plugins can generate a transaction
>> id
>> > > that we
>> > >     >> >> could
>> > >     >> >> >>>>>     potentially put in the header of every message. How
>> > > would you
>> > >     >> >> >>>> typically
>> > >     >> >> >>>>>     make use of such transaction ids? Are there other
>> > > metadata
>> > >     >> >> >>>> associated with
>> > >     >> >> >>>>>     the transaction id and if so, how are they
>> propagated
>> > >     >> >> downstream?
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     101. For the finance use case, if the concept of
>> > > transaction
>> > >     >> is
>> > >     >> >> >>>> important,
>> > >     >> >> >>>>>     wouldn't it be typically included in the message
>> > > payload
>> > >     >> >> instead of
>> > >     >> >> >>>> as an
>> > >     >> >> >>>>>     optional header field?
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     102. The data lineage that Altas and Navigator
>> > support
>> > > seems
>> > >     >> to
>> > >     >> >> be
>> > >     >> >> >>>> at the
>> > >     >> >> >>>>>     dataset level, not per record level? So, not sure
>> if
>> > > per
>> > >     >> message
>> > >     >> >> >>>> headers
>> > >     >> >> >>>>>     are relevant there.
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     Mirroring:
>> > >     >> >> >>>>>     103. The benefit of using separate partitions is
>> that
>> > > it
>> > >     >> >> >> potentially
>> > >     >> >> >>>> makes
>> > >     >> >> >>>>>     it easy to preserve offsets during mirroring. This
>> > > will make
>> > >     >> it
>> > >     >> >> >>>> easier for
>> > >     >> >> >>>>>     consumer to switch clusters. Currently, the
>> consumers
>> > > can
>> > >     >> switch
>> > >     >> >> >>>> clusters
>> > >     >> >> >>>>>     by using the timestampToOffset() api, but it has to
>> > > deal with
>> > >     >> >> >>>> duplicates.
>> > >     >> >> >>>>>     Good point on the issue with log compact and I am
>> not
>> > > sure how
>> > >     >> >> to
>> > >     >> >> >>>> address
>> > >     >> >> >>>>>     this. However, even if we mirror into the existing
>> > > partitions,
>> > >     >> >> the
>> > >     >> >> >>>> ordering
>> > >     >> >> >>>>>     for messages generated from different clusters
>> seems
>> > >     >> >> >>>> non-deterministic
>> > >     >> >> >>>>>     anyway. So, it seems that the consumers already
>> have
>> > > to deal
>> > >     >> >> with
>> > >     >> >> >>>> that? If
>> > >     >> >> >>>>>     a topic is compacted, does that mean which messages
>> > are
>> > >     >> >> preserved
>> > >     >> >> >> is
>> > >     >> >> >>>> also
>> > >     >> >> >>>>>     non-deterministic across clusters?
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     104. Good point on partition key.
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     End-to-end encryption:
>> > >     >> >> >>>>>     105. So, it seems end-to-end encryption is useful.
>> > Are
>> > > headers
>> > >     >> >> >>>> useful there?
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     Auditing:
>> > >     >> >> >>>>>     106. It seems other than the UUID, all other
>> metadata
>> > > are per
>> > >     >> >> >>>> producer?
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     EOS:
>> > >     >> >> >>>>>     107. How are those UUIDs generated? I am not sure
>> if
>> > > they can
>> > >     >> be
>> > >     >> >> >>>> generated
>> > >     >> >> >>>>>     in the producer library. An application may send
>> > > messages
>> > >     >> >> through a
>> > >     >> >> >>>> load
>> > >     >> >> >>>>>     balancer and on retry, the same message could be
>> > > routed to a
>> > >     >> >> >>>> different
>> > >     >> >> >>>>>     producer instance. So, it seems that the
>> application
>> > > has to
>> > >     >> >> >> generate
>> > >     >> >> >>>> the
>> > >     >> >> >>>>>     UUIDs. In that case, shouldn't the application just
>> > > put the
>> > >     >> >> UUID in
>> > >     >> >> >>>> the
>> > >     >> >> >>>>>     payload?
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     Thanks,
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     Jun
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
>> > >     >> >> >>>> Michael.Pearce@ig.com>
>> > >     >> >> >>>>>     wrote:
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>     > Hi Jun.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Per Transaction Tracing / Data Lineage.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > As Stated in the KIP this has the first use case
>> of
>> > > how many
>> > >     >> >> APM
>> > >     >> >> >>>> tools now
>> > >     >> >> >>>>>     > work.
>> > >     >> >> >>>>>     > I would find it impossible for any one to argue
>> > this
>> > > is not
>> > >     >> >> >>>> important or a
>> > >     >> >> >>>>>     > niche market as it has its own gartner report for
>> > > this
>> > >     >> space.
>> > >     >> >> >> Such
>> > >     >> >> >>>>>     > companies as Appdynamics, NewRelic, Dynatrace,
>> > > Hawqular are
>> > >     >> >> but a
>> > >     >> >> >>>> few.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Likewise these APM tools can help very rapidly
>> > track
>> > > down
>> > >     >> >> issues
>> > >     >> >> >>>> and
>> > >     >> >> >>>>>     > automatically capture metrics, perform actions
>> > based
>> > > on
>> > >     >> >> >> unexpected
>> > >     >> >> >>>> behavior
>> > >     >> >> >>>>>     > to auto recover services.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Before mentioning looking at aggregated stats, in
>> > > these
>> > >     >> cases
>> > >     >> >> >> where
>> > >     >> >> >>>>>     > actually on critical flows we cannot afford to
>> have
>> > >     >> aggregated
>> > >     >> >> >>>> rolled up
>> > >     >> >> >>>>>     > stats only.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > With the APM tool we use its actually able to
>> > detect
>> > > a
>> > >     >> single
>> > >     >> >> >>>> transaction
>> > >     >> >> >>>>>     > failure and capture the thread traces in the JVM
>> > > where it
>> > >     >> >> failed
>> > >     >> >> >>>> and
>> > >     >> >> >>>>>     > everything for us, to the point it sends us
>> alerts
>> > > where we
>> > >     >> >> have
>> > >     >> >> >>>> this
>> > >     >> >> >>>>>     > giving the line number of the code that caused
>> it,
>> > > the
>> > >     >> >> >> transaction
>> > >     >> >> >>>> trace
>> > >     >> >> >>>>>     > through all the services and endpoints
>> (supported)
>> > > upto the
>> > >     >> >> point
>> > >     >> >> >>>> of
>> > >     >> >> >>>>>     > failure, it can also capture the data in and out
>> > (so
>> > > we can
>> > >     >> >> >>>> replay).
>> > >     >> >> >>>>>     > Because atm Kafka doesn’t support us being able
>> to
>> > > stich in
>> > >     >> >> these
>> > >     >> >> >>>> tracing
>> > >     >> >> >>>>>     > transaction ids natively, we cannot get these
>> > > benefits as
>> > >     >> >> such is
>> > >     >> >> >>>> limiting
>> > >     >> >> >>>>>     > our ability support apps and monitor them to the
>> > same
>> > >     >> >> standards
>> > >     >> >> >> we
>> > >     >> >> >>>> come to
>> > >     >> >> >>>>>     > expect when on a kafka flow.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > This actually ties in with Data Lineage, as the
>> > same
>> > > tracing
>> > >     >> >> can
>> > >     >> >> >>>> be used
>> > >     >> >> >>>>>     > to back stich this. Essentially many times due to
>> > > the sums
>> > >     >> of
>> > >     >> >> >> money
>> > >     >> >> >>>>>     > involved there are disputes, and typically as a
>> > > financial
>> > >     >> >> >>>> institute the
>> > >     >> >> >>>>>     > easiest and cleanest way to prove when disputes
>> > > arise is to
>> > >     >> >> >>>> present the
>> > >     >> >> >>>>>     > actual flow and processes involved in a
>> > transaction.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Likewise as Hadoop matures its evident this case
>> is
>> > >     >> >> important, as
>> > >     >> >> >>>> tools
>> > >     >> >> >>>>>     > such as Atlas (Hortonworks led) and Navigator
>> > > (cloudera led)
>> > >     >> >> are
>> > >     >> >> >>>> evident
>> > >     >> >> >>>>>     > also I believe the importance here is very much
>> NOT
>> > > just a
>> > >     >> >> >>>> financial issue.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > From a MDM point of view any company wanting to
>> > care
>> > > about
>> > >     >> >> Data
>> > >     >> >> >>>> Quality
>> > >     >> >> >>>>>     > and Data Governance - Data Lineage is a key piece
>> > in
>> > > this
>> > >     >> >> puzzle.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > RE Mirroring,
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > As per the KIP in-fact this is exactly what we do
>> > re
>> > > cluster
>> > >     >> >> id,
>> > >     >> >> >>>> to mirror
>> > >     >> >> >>>>>     > a network of clusters between AZ’s / Regions. We
>> > > know a
>> > >     >> >> >>>> transaction for a
>> > >     >> >> >>>>>     > key will be done within a  AZ/Region, as such we
>> > > know the
>> > >     >> >> write
>> > >     >> >> >> to
>> > >     >> >> >>>> kafka
>> > >     >> >> >>>>>     > would be ordered per key. But we need eventual
>> view
>> > > of that
>> > >     >> >> >> across
>> > >     >> >> >>>> in our
>> > >     >> >> >>>>>     > other regions/az’s. When we have complete AZ or
>> > > Region
>> > >     >> >> failure we
>> > >     >> >> >>>> know
>> > >     >> >> >>>>>     > there will be a brief interruption whilst those
>> > > transactions
>> > >     >> >> are
>> > >     >> >> >>>> moved to
>> > >     >> >> >>>>>     > another region but we expect after it to
>> continue.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > As mentioned having separate Partions to do this
>> > > starts to
>> > >     >> get
>> > >     >> >> >>>>>     > ugly/complicated for us:
>> > >     >> >> >>>>>     > how would I do compaction where a key is in two
>> > > partitions?
>> > >     >> >> >>>>>     > How do we balance consumers so where multiple
>> > > partitions
>> > >     >> with
>> > >     >> >> the
>> > >     >> >> >>>> same key
>> > >     >> >> >>>>>     > goto the same consumer
>> > >     >> >> >>>>>     > What do you do if cluster 1 has 5 partitions but
>> > > cluster 20
>> > >     >> >> has
>> > >     >> >> >> 10
>> > >     >> >> >>>> because
>> > >     >> >> >>>>>     > its larger kit in our more core DC’s, as such key
>> > to
>> > >     >> partition
>> > >     >> >> >>>> mappings for
>> > >     >> >> >>>>>     > consumers get even more complicated.
>> > >     >> >> >>>>>     > What do you do if we add or remove a complete
>> > region
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Where as simple mirror will work we just need to
>> > > ensure we
>> > >     >> >> don’t
>> > >     >> >> >>>> have a
>> > >     >> >> >>>>>     > cycle which we can do with clusterId.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > We even have started to look at shortest path
>> > mirror
>> > > routing
>> > >     >> >> >> based
>> > >     >> >> >>>> on
>> > >     >> >> >>>>>     > clusterId, if we also had the region and az info
>> on
>> > > the
>> > >     >> >> >> originating
>> > >     >> >> >>>>>     > message, this we have not implemented but some
>> > ideas
>> > > come
>> > >     >> from
>> > >     >> >> >>>> network
>> > >     >> >> >>>>>     > routing, and also the dispatcher router in apache
>> > > qpid.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Also we need to have data perimeters e.g. certain
>> > > data
>> > >     >> cannot
>> > >     >> >> >> leave
>> > >     >> >> >>>>>     > certain countries borders. We want this all
>> > > automated so
>> > >     >> that
>> > >     >> >> at
>> > >     >> >> >>>> the
>> > >     >> >> >>>>>     > platform level without having to touch or look at
>> > the
>> > >     >> business
>> > >     >> >> >>>> data inside
>> > >     >> >> >>>>>     > we can have headers we can put tags into so that
>> we
>> > > can
>> > >     >> ensure
>> > >     >> >> >>>> this doesn’t
>> > >     >> >> >>>>>     > occur when we mirror. (actually links in to data
>> > > lineage /
>> > >     >> >> >> tracing
>> > >     >> >> >>>> as again
>> > >     >> >> >>>>>     > we need to tag messages at a platform level)
>> > > Examples are we
>> > >     >> >> are
>> > >     >> >> >>>> not
>> > >     >> >> >>>>>     > allowed Private customer details to leave
>> > > Switzerland, yet
>> > >     >> we
>> > >     >> >> >> need
>> > >     >> >> >>>> those
>> > >     >> >> >>>>>     > systems integrated.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Lastly around mirroring we have a partionKey
>> field,
>> > > as the
>> > >     >> key
>> > >     >> >> >>>> used for
>> > >     >> >> >>>>>     > portioning logic != compaction key all the time
>> but
>> > > we want
>> > >     >> to
>> > >     >> >> >>>> preserve it
>> > >     >> >> >>>>>     > for when we mirror so that if source cluster
>> > > partition count
>> > >     >> >> !=
>> > >     >> >> >>>> destination
>> > >     >> >> >>>>>     > cluster partition count we can honour the same
>> > > partitioning
>> > >     >> >> >> logic.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > RE End 2 End encryption
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > As I believe mentioned just before, the solution
>> > you
>> > > mention
>> > >     >> >> just
>> > >     >> >> >>>> doesn’t
>> > >     >> >> >>>>>     > cut the mustard these days with many regulators.
>> An
>> > >     >> operations
>> > >     >> >> >>>> person with
>> > >     >> >> >>>>>     > access to the box should not be able to have
>> access
>> > > to the
>> > >     >> >> data.
>> > >     >> >> >>>> Many now
>> > >     >> >> >>>>>     > actually impose quite literally the
>> implementation
>> > > expected
>> > >     >> >> being
>> > >     >> >> >>>> end2end
>> > >     >> >> >>>>>     > encryption for certain data (Singapore for us is
>> > one
>> > > that I
>> > >     >> am
>> > >     >> >> >>>> most aware
>> > >     >> >> >>>>>     > of). In fact we’re even now needing encrypt the
>> > data
>> > > and
>> > >     >> store
>> > >     >> >> >> the
>> > >     >> >> >>>> keys in
>> > >     >> >> >>>>>     > HSM modules.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Likewise the performance penalty on encrypting
>> > > decrypting as
>> > >     >> >> you
>> > >     >> >> >>>> produce
>> > >     >> >> >>>>>     > over wire, then again encrypt decrypt as the data
>> > is
>> > > stored
>> > >     >> on
>> > >     >> >> >> the
>> > >     >> >> >>>> brokers
>> > >     >> >> >>>>>     > disks and back again, then again encrypted and
>> > > decrypted
>> > >     >> back
>> > >     >> >> >> over
>> > >     >> >> >>>> the wire
>> > >     >> >> >>>>>     > each time for each consumer all adds up, ignoring
>> > > this
>> > >     >> >> doubling
>> > >     >> >> >>>> with mirror
>> > >     >> >> >>>>>     > makers etc. simply encrypting the value once on
>> > > write by the
>> > >     >> >> >>>> client and
>> > >     >> >> >>>>>     > again decrypting on consume by the consumer is
>> far
>> > > more
>> > >     >> >> >>>> performant, but
>> > >     >> >> >>>>>     > then the routing and platform meta data needs to
>> be
>> > > separate
>> > >     >> >> >> (thus
>> > >     >> >> >>>> headers)
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > RE Auditing:
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Our Auditing needs are:
>> > >     >> >> >>>>>     > Producer Id,
>> > >     >> >> >>>>>     > Origin Cluster Id that message first produced
>> into
>> > >     >> >> >>>>>     > Origin AZ – agreed we can derive this if we have
>> > > cluster id,
>> > >     >> >> but
>> > >     >> >> >>>> it makes
>> > >     >> >> >>>>>     > resolving this for audit reporting a lot easier.
>> > >     >> >> >>>>>     > Origin Region – agreed we can derive this if we
>> > have
>> > > cluster
>> > >     >> >> id,
>> > >     >> >> >>>> but it
>> > >     >> >> >>>>>     > makes resolving this for audit reporting a lot
>> > > easier.
>> > >     >> >> >>>>>     > Unique Message Identification (this is not the
>> same
>> > > as
>> > >     >> >> >> transaction
>> > >     >> >> >>>>>     > tracing) – note offset and partition are not the
>> > > same, as
>> > >     >> >> when we
>> > >     >> >> >>>> mirror or
>> > >     >> >> >>>>>     > have for what ever system failure duplicate send,
>> > >     >> >> >>>>>     > Custom Client wrapper version (where
>> organizations
>> > > have to
>> > >     >> >> wrap
>> > >     >> >> >>>> the kafka
>> > >     >> >> >>>>>     > client for added features) so we know what
>> version
>> > > of the
>> > >     >> >> wrapper
>> > >     >> >> >>>> is used
>> > >     >> >> >>>>>     > Producer IP address (in case of clients being in
>> > our
>> > > vm/open
>> > >     >> >> >> stack
>> > >     >> >> >>>> infra
>> > >     >> >> >>>>>     > where they can move around, producer id will stay
>> > > the same
>> > >     >> but
>> > >     >> >> >>>> this would
>> > >     >> >> >>>>>     > change)
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > RE Once and only once delivery case
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Using the same Message UUID for auditing we can
>> > > achieve this
>> > >     >> >> >> quite
>> > >     >> >> >>>> simply.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > As per how some other brokers do this (cough
>> qpid,
>> > > artemis)
>> > >     >> >> >>>> message uuid
>> > >     >> >> >>>>>     > are used to dedupe where message is sent and
>> > > produced but
>> > >     >> the
>> > >     >> >> >>>> client didn’t
>> > >     >> >> >>>>>     > receive the ack, and there for replays the send,
>> by
>> > > having a
>> > >     >> >> >>>> unique message
>> > >     >> >> >>>>>     > id per message, this can be filtered out, on
>> > > consumers where
>> > >     >> >> >>>> message
>> > >     >> >> >>>>>     > delivery may occur twice for what ever reasons a
>> > > message
>> > >     >> uuid
>> > >     >> >> can
>> > >     >> >> >>>> be used
>> > >     >> >> >>>>>     > to remove duplicates being deliverd , like wise
>> we
>> > > can do
>> > >     >> >> this in
>> > >     >> >> >>>> the
>> > >     >> >> >>>>>     > mirrormakers so if we detect a dupe message we
>> can
>> > > avoid
>> > >     >> >> >>>> replicating it.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > Cheers
>> > >     >> >> >>>>>     > Mike
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > On 02/12/2016, 22:09, "Jun Rao" <
>> jun@confluent.io>
>> > > wrote:
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >     Since this KIP affects message format, wire
>> > > protocol,
>> > >     >> >> apis, I
>> > >     >> >> >>>> think
>> > >     >> >> >>>>>     > it's
>> > >     >> >> >>>>>     >     worth spending a bit more time to nail down
>> the
>> > > concrete
>> > >     >> >> use
>> > >     >> >> >>>> cases. It
>> > >     >> >> >>>>>     >     would be bad if we add this feature, but when
>> > > start
>> > >     >> >> >>>> implementing it
>> > >     >> >> >>>>>     > for say
>> > >     >> >> >>>>>     >     mirroring, we then realize that header is not
>> > > the best
>> > >     >> >> >>>> approach.
>> > >     >> >> >>>>>     > Initially,
>> > >     >> >> >>>>>     >     I thought I was convinced of the use cases of
>> > > headers
>> > >     >> and
>> > >     >> >> was
>> > >     >> >> >>>> trying to
>> > >     >> >> >>>>>     >     write down a few use cases to convince
>> others.
>> > > That's
>> > >     >> >> when I
>> > >     >> >> >>>> became
>> > >     >> >> >>>>>     > less
>> > >     >> >> >>>>>     >     certain. For me to be convinced, I just want
>> to
>> > > see two
>> > >     >> >> >> strong
>> > >     >> >> >>>> use
>> > >     >> >> >>>>>     > cases
>> > >     >> >> >>>>>     >     (instead of 10 maybe use cases) in the
>> > > third-party
>> > >     >> space.
>> > >     >> >> The
>> > >     >> >> >>>> reason is
>> > >     >> >> >>>>>     >     that when we discussed the use cases within a
>> > > company,
>> > >     >> >> often
>> > >     >> >> >>>> it ends
>> > >     >> >> >>>>>     > with
>> > >     >> >> >>>>>     >     "we can't force everyone to use this standard
>> > > since we
>> > >     >> may
>> > >     >> >> >>>> have to
>> > >     >> >> >>>>>     >     integrate with third-party tools".
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >     At present, I am not sure why headers are
>> > useful
>> > > for
>> > >     >> >> things
>> > >     >> >> >>>> like
>> > >     >> >> >>>>>     > schemaId
>> > >     >> >> >>>>>     >     or encryption. In order to do anything useful
>> > to
>> > > the
>> > >     >> >> value,
>> > >     >> >> >>>> one needs
>> > >     >> >> >>>>>     > to
>> > >     >> >> >>>>>     >     know the schemaId or how data is encrypted,
>> but
>> > > header
>> > >     >> is
>> > >     >> >> >>>> optional.
>> > >     >> >> >>>>>     > But, I
>> > >     >> >> >>>>>     >     can be convinced if someone (Radai, Sean,
>> > Todd?)
>> > >     >> provides
>> > >     >> >> >> more
>> > >     >> >> >>>> details
>> > >     >> >> >>>>>     > on
>> > >     >> >> >>>>>     >     the argument.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >     I am not very sure header is the best
>> approach
>> > > for
>> > >     >> >> mirroring
>> > >     >> >> >>>> either. If
>> > >     >> >> >>>>>     >     someone has thought about this more, I'd be
>> > > happy to
>> > >     >> hear.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >     I can see the data lineage use case. I am
>> just
>> > > not sure
>> > >     >> >> how
>> > >     >> >> >>>> widely
>> > >     >> >> >>>>>     >     applicable this is. If someone familiar with
>> > > this space
>> > >     >> >> can
>> > >     >> >> >>>> justify
>> > >     >> >> >>>>>     > this is
>> > >     >> >> >>>>>     >     a significant use case, say in the finance
>> > > industry,
>> > >     >> this
>> > >     >> >> >>>> would be a
>> > >     >> >> >>>>>     > strong
>> > >     >> >> >>>>>     >     use case.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >     I can see the auditing use case. I am just
>> not
>> > > sure if a
>> > >     >> >> >> native
>> > >     >> >> >>>>>     > producer id
>> > >     >> >> >>>>>     >     solves that problem. If there are additional
>> > > metadata
>> > >     >> >> that's
>> > >     >> >> >>>> worth
>> > >     >> >> >>>>>     >     collecting but not covered by the producer
>> id,
>> > > that
>> > >     >> would
>> > >     >> >> >> make
>> > >     >> >> >>>> this a
>> > >     >> >> >>>>>     >     strong use case.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >     Thanks,
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >     Jun
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >     On Fri, Dec 2, 2016 at 1:41 PM, radai <
>> > >     >> >> >>>> radai.rosenblatt@gmail.com>
>> > >     >> >> >>>>>     > wrote:
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >     > this KIP is about enabling headers, nothing
>> > > more
>> > >     >> nothing
>> > >     >> >> >>>> less - so
>> > >     >> >> >>>>>     > no,
>> > >     >> >> >>>>>     >     > broker-side use of headers is not in the
>> KIP
>> > > scope.
>> > >     >> >> >>>>>     >     >
>> > >     >> >> >>>>>     >     > obviously though, once you have headers
>> > > potential use
>> > >     >> >> cases
>> > >     >> >> >>>> could
>> > >     >> >> >>>>>     > include
>> > >     >> >> >>>>>     >     > broker-side header-aware interceptors
>> (which
>> > > would be
>> > >     >> >> the
>> > >     >> >> >>>> topic of
>> > >     >> >> >>>>>     > other
>> > >     >> >> >>>>>     >     > future KIPs).
>> > >     >> >> >>>>>     >     >
>> > >     >> >> >>>>>     >     > a trivially clear use case (to me) would be
>> > > using such
>> > >     >> >> >>>> broker-side
>> > >     >> >> >>>>>     >     > interceptors to enforce compliance with
>> > > organizational
>> > >     >> >> >>>> policies - it
>> > >     >> >> >>>>>     > would
>> > >     >> >> >>>>>     >     > make our SREs lives much easier if instead
>> of
>> > >     >> >> retroactively
>> > >     >> >> >>>>>     > discovering
>> > >     >> >> >>>>>     >     > "rogue" topics/users those messages would
>> > have
>> > > been
>> > >     >> >> >> rejected
>> > >     >> >> >>>>>     > up-front.
>> > >     >> >> >>>>>     >     >
>> > >     >> >> >>>>>     >     > the kafka broker code is lacking any such
>> > >     >> extensibility
>> > >     >> >> >>>> support
>> > >     >> >> >>>>>     > (beyond
>> > >     >> >> >>>>>     >     > maybe authorizer) which is why these use
>> > cases
>> > > were
>> > >     >> left
>> > >     >> >> >> out
>> > >     >> >> >>>> of the
>> > >     >> >> >>>>>     > "case
>> > >     >> >> >>>>>     >     > for headers" doc - broker extensibility is
>> a
>> > > separate
>> > >     >> >> >>>> discussion.
>> > >     >> >> >>>>>     >     >
>> > >     >> >> >>>>>     >     > On Fri, Dec 2, 2016 at 12:59 PM, Gwen
>> > Shapira <
>> > >     >> >> >>>> gwen@confluent.io>
>> > >     >> >> >>>>>     > wrote:
>> > >     >> >> >>>>>     >     >
>> > >     >> >> >>>>>     >     > > Woah, I wasn't aware this is something
>> > we'll
>> > > do. It
>> > >     >> >> >> wasn't
>> > >     >> >> >>>> in the
>> > >     >> >> >>>>>     > KIP,
>> > >     >> >> >>>>>     >     > > right?
>> > >     >> >> >>>>>     >     > >
>> > >     >> >> >>>>>     >     > > I guess we could do it the same way ACLs
>> > > currently
>> > >     >> >> work.
>> > >     >> >> >>>>>     >     > > I had in mind something that will allow
>> > > admins to
>> > >     >> >> apply
>> > >     >> >> >>>> rules to
>> > >     >> >> >>>>>     > the
>> > >     >> >> >>>>>     >     > > new create/delete/config topic APIs. So
>> > Todd
>> > > can
>> > >     >> >> decide
>> > >     >> >> >> to
>> > >     >> >> >>>> reject
>> > >     >> >> >>>>>     >     > > "create topic" requests that ask for more
>> > > than 40
>> > >     >> >> >>>> partitions, or
>> > >     >> >> >>>>>     >     > > require exactly 3 replicas, or no more
>> than
>> > > 50GB
>> > >     >> >> >> partition
>> > >     >> >> >>>> size,
>> > >     >> >> >>>>>     > etc.
>> > >     >> >> >>>>>     >     > >
>> > >     >> >> >>>>>     >     > > ACLs were added a bit ad-hoc, if we are
>> > > planning to
>> > >     >> >> apply
>> > >     >> >> >>>> more
>> > >     >> >> >>>>>     > rules
>> > >     >> >> >>>>>     >     > > to requests (and I think we should), we
>> may
>> > > want a
>> > >     >> bit
>> > >     >> >> >>>> more generic
>> > >     >> >> >>>>>     >     > > design around that.
>> > >     >> >> >>>>>     >     > >
>> > >     >> >> >>>>>     >     > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
>> > >     >> >> >>>> radai.rosenblatt@gmail.com>
>> > >     >> >> >>>>>     >     > wrote:
>> > >     >> >> >>>>>     >     > > > "wouldn't you be in the business of
>> > making
>> > > sure
>> > >     >> >> >> everyone
>> > >     >> >> >>>> uses
>> > >     >> >> >>>>>     > them
>> > >     >> >> >>>>>     >     > > > properly?"
>> > >     >> >> >>>>>     >     > > >
>> > >     >> >> >>>>>     >     > > > thats where a broker-side plugin would
>> > > come handy
>> > >     >> -
>> > >     >> >> any
>> > >     >> >> >>>> incoming
>> > >     >> >> >>>>>     >     > message
>> > >     >> >> >>>>>     >     > > > that does not conform to org policy
>> (read
>> > > - does
>> > >     >> not
>> > >     >> >> >>>> have the
>> > >     >> >> >>>>>     > proper
>> > >     >> >> >>>>>     >     > > > headers) gets thrown out (with an error
>> > > returned
>> > >     >> to
>> > >     >> >> >> user)
>> > >     >> >> >>>>>     >     > > >
>> > >     >> >> >>>>>     >     > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd
>> > > Palino <
>> > >     >> >> >>>> tpalino@gmail.com>
>> > >     >> >> >>>>>     > wrote:
>> > >     >> >> >>>>>     >     > > >
>> > >     >> >> >>>>>     >     > > >> Come on, I’ve done at least 2 talks on
>> > > this one
>> > >     >> :)
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >> Producing counts to a topic is part of
>> > > it, but
>> > >     >> >> that’s
>> > >     >> >> >>>> only
>> > >     >> >> >>>>>     > part. So
>> > >     >> >> >>>>>     >     > you
>> > >     >> >> >>>>>     >     > > >> count you have 100 messages in topic
>> A.
>> > > When you
>> > >     >> >> >> mirror
>> > >     >> >> >>>> topic A
>> > >     >> >> >>>>>     > to
>> > >     >> >> >>>>>     >     > > another
>> > >     >> >> >>>>>     >     > > >> cluster, you have 99 messages. Where
>> was
>> > > your
>> > >     >> >> problem?
>> > >     >> >> >>>> Or
>> > >     >> >> >>>>>     > worse, you
>> > >     >> >> >>>>>     >     > > have
>> > >     >> >> >>>>>     >     > > >> 100 messages, but one producer
>> > duplicated
>> > >     >> messages
>> > >     >> >> and
>> > >     >> >> >>>> another
>> > >     >> >> >>>>>     > one
>> > >     >> >> >>>>>     >     > lost
>> > >     >> >> >>>>>     >     > > >> messages. You need details about where
>> > the
>> > >     >> message
>> > >     >> >> >> came
>> > >     >> >> >>>> from in
>> > >     >> >> >>>>>     > order
>> > >     >> >> >>>>>     >     > to
>> > >     >> >> >>>>>     >     > > >> pinpoint problems when they happen.
>> > Source
>> > >     >> producer
>> > >     >> >> >>>> info, where
>> > >     >> >> >>>>>     > it was
>> > >     >> >> >>>>>     >     > > >> produced into your infrastructure, and
>> > > when it
>> > >     >> was
>> > >     >> >> >>>> produced.
>> > >     >> >> >>>>>     > This
>> > >     >> >> >>>>>     >     > > requires
>> > >     >> >> >>>>>     >     > > >> you to add the information to the
>> > message.
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >> And yes, you still need to maintain
>> your
>> > > clients.
>> > >     >> >> So
>> > >     >> >> >>>> maybe my
>> > >     >> >> >>>>>     > original
>> > >     >> >> >>>>>     >     > > >> example was not the best. My thoughts
>> on
>> > > not
>> > >     >> >> wanting
>> > >     >> >> >> to
>> > >     >> >> >>>> be
>> > >     >> >> >>>>>     > responsible
>> > >     >> >> >>>>>     >     > > for
>> > >     >> >> >>>>>     >     > > >> message formats stands, because that’s
>> > > very much
>> > >     >> >> >>>> separate from
>> > >     >> >> >>>>>     > the
>> > >     >> >> >>>>>     >     > > client.
>> > >     >> >> >>>>>     >     > > >> As you know, we have our own internal
>> > > client
>> > >     >> >> library
>> > >     >> >> >>>> that can
>> > >     >> >> >>>>>     > insert
>> > >     >> >> >>>>>     >     > the
>> > >     >> >> >>>>>     >     > > >> right headers, and right now inserts
>> the
>> > > right
>> > >     >> >> audit
>> > >     >> >> >>>>>     > information into
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> message fields. If they exist, and
>> > > assuming the
>> > >     >> >> >> message
>> > >     >> >> >>>> is Avro
>> > >     >> >> >>>>>     >     > encoded.
>> > >     >> >> >>>>>     >     > > >> What if someone wants to use JSON
>> > instead
>> > > for a
>> > >     >> >> good
>> > >     >> >> >>>> reason?
>> > >     >> >> >>>>>     > What if
>> > >     >> >> >>>>>     >     > > user X
>> > >     >> >> >>>>>     >     > > >> wants to encrypt messages, but user Y
>> > > does not?
>> > >     >> >> >>>> Maintaining the
>> > >     >> >> >>>>>     > client
>> > >     >> >> >>>>>     >     > > >> library is still much easier than
>> > > maintaining the
>> > >     >> >> >>>> message
>> > >     >> >> >>>>>     > formats.
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >> -Todd
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen
>> > > Shapira <
>> > >     >> >> >>>> gwen@confluent.io
>> > >     >> >> >>>>>     > >
>> > >     >> >> >>>>>     >     > wrote:
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >> > Based on your last sentence,
>> consider
>> > me
>> > >     >> >> convinced
>> > >     >> >> >> :)
>> > >     >> >> >>>>>     >     > > >> >
>> > >     >> >> >>>>>     >     > > >> > I get why headers are critical for
>> > > Mirroring
>> > >     >> (you
>> > >     >> >> >>>> need tags to
>> > >     >> >> >>>>>     >     > prevent
>> > >     >> >> >>>>>     >     > > >> > loops and sometimes to route
>> messages
>> > > to the
>> > >     >> >> correct
>> > >     >> >> >>>>>     > destination).
>> > >     >> >> >>>>>     >     > > >> > But why do you need headers to
>> audit?
>> > > We are
>> > >     >> >> >> auditing
>> > >     >> >> >>>> by
>> > >     >> >> >>>>>     > producing
>> > >     >> >> >>>>>     >     > > >> > counts to a side topic (and I was
>> > under
>> > > the
>> > >     >> >> >>>> impression you do
>> > >     >> >> >>>>>     > the
>> > >     >> >> >>>>>     >     > > >> > same), so we never need to modify
>> the
>> > > message.
>> > >     >> >> >>>>>     >     > > >> >
>> > >     >> >> >>>>>     >     > > >> > Another thing - after we added
>> > headers,
>> > >     >> wouldn't
>> > >     >> >> you
>> > >     >> >> >>>> be in the
>> > >     >> >> >>>>>     >     > > >> > business of making sure everyone
>> uses
>> > > them
>> > >     >> >> properly?
>> > >     >> >> >>>> Making
>> > >     >> >> >>>>>     > sure
>> > >     >> >> >>>>>     >     > > >> > everyone includes the right headers
>> > you
>> > > need,
>> > >     >> not
>> > >     >> >> >>>> using the
>> > >     >> >> >>>>>     > header
>> > >     >> >> >>>>>     >     > > >> > names you intend to use, etc. I
>> don't
>> > > think the
>> > >     >> >> >>>> "policing"
>> > >     >> >> >>>>>     > business
>> > >     >> >> >>>>>     >     > > >> > will ever go away.
>> > >     >> >> >>>>>     >     > > >> >
>> > >     >> >> >>>>>     >     > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd
>> > > Palino <
>> > >     >> >> >>>>>     > tpalino@gmail.com>
>> > >     >> >> >>>>>     >     > > wrote:
>> > >     >> >> >>>>>     >     > > >> > > Got it. As an ops guy, I'm not
>> very
>> > > happy
>> > >     >> with
>> > >     >> >> the
>> > >     >> >> >>>>>     > workaround.
>> > >     >> >> >>>>>     >     > Avro
>> > >     >> >> >>>>>     >     > > >> means
>> > >     >> >> >>>>>     >     > > >> > > that I have to be concerned with
>> the
>> > > format
>> > >     >> of
>> > >     >> >> the
>> > >     >> >> >>>> messages
>> > >     >> >> >>>>>     > in
>> > >     >> >> >>>>>     >     > > order to
>> > >     >> >> >>>>>     >     > > >> > run
>> > >     >> >> >>>>>     >     > > >> > > the infrastructure (audit,
>> > mirroring,
>> > > etc.).
>> > >     >> >> That
>> > >     >> >> >>>> means
>> > >     >> >> >>>>>     > that I
>> > >     >> >> >>>>>     >     > have
>> > >     >> >> >>>>>     >     > > to
>> > >     >> >> >>>>>     >     > > >> > > handle the schemas, and I have to
>> > > enforce
>> > >     >> rules
>> > >     >> >> >>>> about good
>> > >     >> >> >>>>>     >     > formats.
>> > >     >> >> >>>>>     >     > > >> This
>> > >     >> >> >>>>>     >     > > >> > is
>> > >     >> >> >>>>>     >     > > >> > > not something I want to be in the
>> > > business
>> > >     >> of,
>> > >     >> >> >>>> because I
>> > >     >> >> >>>>>     > should be
>> > >     >> >> >>>>>     >     > > able
>> > >     >> >> >>>>>     >     > > >> > to
>> > >     >> >> >>>>>     >     > > >> > > run a service infrastructure
>> without
>> > > needing
>> > >     >> >> to be
>> > >     >> >> >>>> in the
>> > >     >> >> >>>>>     > weeds of
>> > >     >> >> >>>>>     >     > > >> > dealing
>> > >     >> >> >>>>>     >     > > >> > > with customer data formats.
>> > >     >> >> >>>>>     >     > > >> > >
>> > >     >> >> >>>>>     >     > > >> > > Trust me, a sizable portion of my
>> > > support
>> > >     >> time
>> > >     >> >> is
>> > >     >> >> >>>> spent
>> > >     >> >> >>>>>     > dealing
>> > >     >> >> >>>>>     >     > with
>> > >     >> >> >>>>>     >     > > >> > schema
>> > >     >> >> >>>>>     >     > > >> > > issues. I really would like to get
>> > > away from
>> > >     >> >> that.
>> > >     >> >> >>>> Maybe
>> > >     >> >> >>>>>     > I'd have
>> > >     >> >> >>>>>     >     > > more
>> > >     >> >> >>>>>     >     > > >> > time
>> > >     >> >> >>>>>     >     > > >> > > for other hobbies. Like writing.
>> ;)
>> > >     >> >> >>>>>     >     > > >> > >
>> > >     >> >> >>>>>     >     > > >> > > -Todd
>> > >     >> >> >>>>>     >     > > >> > >
>> > >     >> >> >>>>>     >     > > >> > > On Thu, Dec 1, 2016 at 4:04 PM
>> Gwen
>> > > Shapira <
>> > >     >> >> >>>>>     > gwen@confluent.io>
>> > >     >> >> >>>>>     >     > > wrote:
>> > >     >> >> >>>>>     >     > > >> > >
>> > >     >> >> >>>>>     >     > > >> > >> I'm pretty satisfied with the
>> > current
>> > >     >> >> workarounds
>> > >     >> >> >>>> (Avro
>> > >     >> >> >>>>>     > container
>> > >     >> >> >>>>>     >     > > >> > >> format), so I'm not too excited
>> > > about the
>> > >     >> >> extra
>> > >     >> >> >>>> work
>> > >     >> >> >>>>>     > required to
>> > >     >> >> >>>>>     >     > do
>> > >     >> >> >>>>>     >     > > >> > >> headers in Kafka. I absolutely
>> > don't
>> > > mind it
>> > >     >> >> if
>> > >     >> >> >>>> you do
>> > >     >> >> >>>>>     > it...
>> > >     >> >> >>>>>     >     > > >> > >> I think the Apache convention for
>> > > "good
>> > >     >> idea,
>> > >     >> >> but
>> > >     >> >> >>>> not
>> > >     >> >> >>>>>     > willing to
>> > >     >> >> >>>>>     >     > > put
>> > >     >> >> >>>>>     >     > > >> > >> any work toward it" is +0.5?
>> > anyway,
>> > > that's
>> > >     >> >> what
>> > >     >> >> >> I
>> > >     >> >> >>>> was
>> > >     >> >> >>>>>     > trying to
>> > >     >> >> >>>>>     >     > > >> > >> convey :)
>> > >     >> >> >>>>>     >     > > >> > >>
>> > >     >> >> >>>>>     >     > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM,
>> > Todd
>> > > Palino
>> > >     >> <
>> > >     >> >> >>>>>     > tpalino@gmail.com>
>> > >     >> >> >>>>>     >     > > >> wrote:
>> > >     >> >> >>>>>     >     > > >> > >> > Well I guess my question for
>> you,
>> > > then, is
>> > >     >> >> what
>> > >     >> >> >>>> is
>> > >     >> >> >>>>>     > holding you
>> > >     >> >> >>>>>     >     > > back
>> > >     >> >> >>>>>     >     > > >> > from
>> > >     >> >> >>>>>     >     > > >> > >> > full support for headers?
>> What’s
>> > > the bit
>> > >     >> >> that
>> > >     >> >> >>>> you’re
>> > >     >> >> >>>>>     > missing
>> > >     >> >> >>>>>     >     > that
>> > >     >> >> >>>>>     >     > > >> has
>> > >     >> >> >>>>>     >     > > >> > you
>> > >     >> >> >>>>>     >     > > >> > >> > under a full +1?
>> > >     >> >> >>>>>     >     > > >> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> > -Todd
>> > >     >> >> >>>>>     >     > > >> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM,
>> > > Gwen
>> > >     >> >> Shapira <
>> > >     >> >> >>>>>     >     > gwen@confluent.io>
>> > >     >> >> >>>>>     >     > > >> > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> I know why people who support
>> > > headers
>> > >     >> >> support
>> > >     >> >> >>>> them, and
>> > >     >> >> >>>>>     > I've
>> > >     >> >> >>>>>     >     > > seen
>> > >     >> >> >>>>>     >     > > >> > what
>> > >     >> >> >>>>>     >     > > >> > >> >> the discussion is like.
>> > >     >> >> >>>>>     >     > > >> > >> >>
>> > >     >> >> >>>>>     >     > > >> > >> >> This is why I'm asking people
>> > who
>> > > are
>> > >     >> >> against
>> > >     >> >> >>>> headers
>> > >     >> >> >>>>>     >     > > (especially
>> > >     >> >> >>>>>     >     > > >> > >> >> committers) what will make
>> them
>> > > change
>> > >     >> >> their
>> > >     >> >> >>>> mind - so
>> > >     >> >> >>>>>     > we can
>> > >     >> >> >>>>>     >     > > get
>> > >     >> >> >>>>>     >     > > >> > this
>> > >     >> >> >>>>>     >     > > >> > >> >> part over one way or another.
>> > >     >> >> >>>>>     >     > > >> > >> >>
>> > >     >> >> >>>>>     >     > > >> > >> >> If I sound frustrated it is
>> not
>> > > at Radai,
>> > >     >> >> Jun
>> > >     >> >> >>>> or you
>> > >     >> >> >>>>>     > (Todd)...
>> > >     >> >> >>>>>     >     > > I am
>> > >     >> >> >>>>>     >     > > >> > >> >> just looking for something
>> > > concrete we
>> > >     >> can
>> > >     >> >> do
>> > >     >> >> >>>> to move
>> > >     >> >> >>>>>     > the
>> > >     >> >> >>>>>     >     > > >> discussion
>> > >     >> >> >>>>>     >     > > >> > >> >> along to the yummy design
>> > details
>> > > (which
>> > >     >> is
>> > >     >> >> >> the
>> > >     >> >> >>>>>     > argument I
>> > >     >> >> >>>>>     >     > > really
>> > >     >> >> >>>>>     >     > > >> am
>> > >     >> >> >>>>>     >     > > >> > >> >> looking forward to).
>> > >     >> >> >>>>>     >     > > >> > >> >>
>> > >     >> >> >>>>>     >     > > >> > >> >> On Thu, Dec 1, 2016 at 1:53
>> PM,
>> > > Todd
>> > >     >> >> Palino <
>> > >     >> >> >>>>>     >     > tpalino@gmail.com>
>> > >     >> >> >>>>>     >     > > >> > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> > So, Gwen, to your question
>> > > (even though
>> > >     >> >> I’m
>> > >     >> >> >>>> not a
>> > >     >> >> >>>>>     >     > > committer)...
>> > >     >> >> >>>>>     >     > > >> > >> >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> > I have always been a strong
>> > > supporter
>> > >     >> of
>> > >     >> >> >>>> introducing
>> > >     >> >> >>>>>     > the
>> > >     >> >> >>>>>     >     > > concept
>> > >     >> >> >>>>>     >     > > >> > of an
>> > >     >> >> >>>>>     >     > > >> > >> >> > envelope to messages, which
>> > > headers
>> > >     >> >> >>>> accomplishes. The
>> > >     >> >> >>>>>     >     > message
>> > >     >> >> >>>>>     >     > > key
>> > >     >> >> >>>>>     >     > > >> > is
>> > >     >> >> >>>>>     >     > > >> > >> >> > already an example of a
>> piece
>> > of
>> > >     >> envelope
>> > >     >> >> >>>>>     > information. By
>> > >     >> >> >>>>>     >     > > >> > providing a
>> > >     >> >> >>>>>     >     > > >> > >> >> means
>> > >     >> >> >>>>>     >     > > >> > >> >> > to do this within Kafka
>> > itself,
>> > > and not
>> > >     >> >> >>>> relying on
>> > >     >> >> >>>>>     > use-case
>> > >     >> >> >>>>>     >     > > >> > specific
>> > >     >> >> >>>>>     >     > > >> > >> >> > implementations, you make it
>> > > much
>> > >     >> easier
>> > >     >> >> for
>> > >     >> >> >>>>>     > components to
>> > >     >> >> >>>>>     >     > > >> > >> interoperate.
>> > >     >> >> >>>>>     >     > > >> > >> >> It
>> > >     >> >> >>>>>     >     > > >> > >> >> > simplifies development of
>> all
>> > > these
>> > >     >> >> things
>> > >     >> >> >>>> (message
>> > >     >> >> >>>>>     > routing,
>> > >     >> >> >>>>>     >     > > >> > auditing,
>> > >     >> >> >>>>>     >     > > >> > >> >> > encryption, etc.) because
>> each
>> > > one does
>> > >     >> >> not
>> > >     >> >> >>>> have to
>> > >     >> >> >>>>>     > reinvent
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > >> wheel.
>> > >     >> >> >>>>>     >     > > >> > >> >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> > It also makes it much easier
>> > > from a
>> > >     >> >> client
>> > >     >> >> >>>> point of
>> > >     >> >> >>>>>     > view if
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > >> headers
>> > >     >> >> >>>>>     >     > > >> > >> >> are
>> > >     >> >> >>>>>     >     > > >> > >> >> > defined as part of the
>> > protocol
>> > > and/or
>> > >     >> >> >>>> message format
>> > >     >> >> >>>>>     > in
>> > >     >> >> >>>>>     >     > > general
>> > >     >> >> >>>>>     >     > > >> > >> because
>> > >     >> >> >>>>>     >     > > >> > >> >> > you can easily produce and
>> > > consume
>> > >     >> >> messages
>> > >     >> >> >>>> without
>> > >     >> >> >>>>>     > having
>> > >     >> >> >>>>>     >     > to
>> > >     >> >> >>>>>     >     > > >> take
>> > >     >> >> >>>>>     >     > > >> > >> into
>> > >     >> >> >>>>>     >     > > >> > >> >> > account specific cases. For
>> > > example, I
>> > >     >> >> want
>> > >     >> >> >>>> to route
>> > >     >> >> >>>>>     >     > messages,
>> > >     >> >> >>>>>     >     > > >> but
>> > >     >> >> >>>>>     >     > > >> > >> >> client A
>> > >     >> >> >>>>>     >     > > >> > >> >> > doesn’t support the way
>> audit
>> > >     >> implemented
>> > >     >> >> >>>> headers, and
>> > >     >> >> >>>>>     >     > client
>> > >     >> >> >>>>>     >     > > B
>> > >     >> >> >>>>>     >     > > >> > >> doesn’t
>> > >     >> >> >>>>>     >     > > >> > >> >> > support the way encryption
>> or
>> > > routing
>> > >     >> >> >>>> implemented
>> > >     >> >> >>>>>     > headers,
>> > >     >> >> >>>>>     >     > so
>> > >     >> >> >>>>>     >     > > now
>> > >     >> >> >>>>>     >     > > >> > my
>> > >     >> >> >>>>>     >     > > >> > >> >> > application has to create
>> some
>> > > really
>> > >     >> >> >> fragile
>> > >     >> >> >>>> (my
>> > >     >> >> >>>>>     >     > autocorrect
>> > >     >> >> >>>>>     >     > > >> just
>> > >     >> >> >>>>>     >     > > >> > >> tried
>> > >     >> >> >>>>>     >     > > >> > >> >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> > make that “tragic”, which is
>> > > probably
>> > >     >> >> >>>> appropriate
>> > >     >> >> >>>>>     > too) code
>> > >     >> >> >>>>>     >     > to
>> > >     >> >> >>>>>     >     > > >> > strip
>> > >     >> >> >>>>>     >     > > >> > >> >> > everything off, rather than
>> > just
>> > >     >> >> consuming
>> > >     >> >> >> the
>> > >     >> >> >>>>>     > messages,
>> > >     >> >> >>>>>     >     > > picking
>> > >     >> >> >>>>>     >     > > >> > out
>> > >     >> >> >>>>>     >     > > >> > >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> 1
>> > >     >> >> >>>>>     >     > > >> > >> >> > or 2 headers it’s interested
>> > > in, and
>> > >     >> >> >>>> performing its
>> > >     >> >> >>>>>     >     > function.
>> > >     >> >> >>>>>     >     > > >> > >> >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> > Honestly, this discussion
>> has
>> > > been
>> > >     >> going
>> > >     >> >> on
>> > >     >> >> >>>> for a
>> > >     >> >> >>>>>     > long time,
>> > >     >> >> >>>>>     >     > > and
>> > >     >> >> >>>>>     >     > > >> > it’s
>> > >     >> >> >>>>>     >     > > >> > >> >> > always “Oh, you came up
>> with 2
>> > > use
>> > >     >> cases,
>> > >     >> >> >> and
>> > >     >> >> >>>> yeah,
>> > >     >> >> >>>>>     > those
>> > >     >> >> >>>>>     >     > use
>> > >     >> >> >>>>>     >     > > >> cases
>> > >     >> >> >>>>>     >     > > >> > >> are
>> > >     >> >> >>>>>     >     > > >> > >> >> > real things that someone
>> would
>> > > want to
>> > >     >> >> do.
>> > >     >> >> >>>> Here’s an
>> > >     >> >> >>>>>     >     > alternate
>> > >     >> >> >>>>>     >     > > >> way
>> > >     >> >> >>>>>     >     > > >> > to
>> > >     >> >> >>>>>     >     > > >> > >> >> > implement them so let’s not
>> do
>> > >     >> headers.”
>> > >     >> >> If
>> > >     >> >> >>>> we have a
>> > >     >> >> >>>>>     > few
>> > >     >> >> >>>>>     >     > use
>> > >     >> >> >>>>>     >     > > >> cases
>> > >     >> >> >>>>>     >     > > >> > >> that
>> > >     >> >> >>>>>     >     > > >> > >> >> we
>> > >     >> >> >>>>>     >     > > >> > >> >> > actually came up with, you
>> can
>> > > be sure
>> > >     >> >> that
>> > >     >> >> >>>> over the
>> > >     >> >> >>>>>     > next
>> > >     >> >> >>>>>     >     > year
>> > >     >> >> >>>>>     >     > > >> > >> there’s a
>> > >     >> >> >>>>>     >     > > >> > >> >> > dozen others that we didn’t
>> > > think of
>> > >     >> that
>> > >     >> >> >>>> someone
>> > >     >> >> >>>>>     > would like
>> > >     >> >> >>>>>     >     > > to
>> > >     >> >> >>>>>     >     > > >> > do. I
>> > >     >> >> >>>>>     >     > > >> > >> >> > really think it’s time to
>> stop
>> > >     >> rehashing
>> > >     >> >> >> this
>> > >     >> >> >>>>>     > discussion and
>> > >     >> >> >>>>>     >     > > >> > instead
>> > >     >> >> >>>>>     >     > > >> > >> >> focus
>> > >     >> >> >>>>>     >     > > >> > >> >> > on a workable standard that
>> we
>> > > can
>> > >     >> adopt.
>> > >     >> >> >>>>>     >     > > >> > >> >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> > -Todd
>> > >     >> >> >>>>>     >     > > >> > >> >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39
>> > PM,
>> > > Todd
>> > >     >> >> Palino
>> > >     >> >> >> <
>> > >     >> >> >>>>>     >     > > tpalino@gmail.com>
>> > >     >> >> >>>>>     >     > > >> > >> wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >> C. per message encryption
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> One drawback of this
>> > approach
>> > > is that
>> > >     >> >> this
>> > >     >> >> >>>>>     > significantly
>> > >     >> >> >>>>>     >     > > reduce
>> > >     >> >> >>>>>     >     > > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> effectiveness of
>> > compression,
>> > > which
>> > >     >> >> >> happens
>> > >     >> >> >>>> on a
>> > >     >> >> >>>>>     > set of
>> > >     >> >> >>>>>     >     > > >> > serialized
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> messages. An alternative
>> is
>> > > to enable
>> > >     >> >> SSL
>> > >     >> >> >>>> for wire
>> > >     >> >> >>>>>     >     > > encryption
>> > >     >> >> >>>>>     >     > > >> and
>> > >     >> >> >>>>>     >     > > >> > >> rely
>> > >     >> >> >>>>>     >     > > >> > >> >> on
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> the storage system (e.g.
>> > > LUKS) for at
>> > >     >> >> rest
>> > >     >> >> >>>>>     > encryption.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >> Jun, this is not
>> sufficient.
>> > > While
>> > >     >> this
>> > >     >> >> >> does
>> > >     >> >> >>>> cover
>> > >     >> >> >>>>>     > the case
>> > >     >> >> >>>>>     >     > > of
>> > >     >> >> >>>>>     >     > > >> > >> removing
>> > >     >> >> >>>>>     >     > > >> > >> >> a
>> > >     >> >> >>>>>     >     > > >> > >> >> >> drive from the system, it
>> > will
>> > > not
>> > >     >> >> satisfy
>> > >     >> >> >>>> most
>> > >     >> >> >>>>>     > compliance
>> > >     >> >> >>>>>     >     > > >> > >> requirements
>> > >     >> >> >>>>>     >     > > >> > >> >> for
>> > >     >> >> >>>>>     >     > > >> > >> >> >> encryption of data as
>> whoever
>> > > has
>> > >     >> >> access to
>> > >     >> >> >>>> the
>> > >     >> >> >>>>>     > broker
>> > >     >> >> >>>>>     >     > itself
>> > >     >> >> >>>>>     >     > > >> > still
>> > >     >> >> >>>>>     >     > > >> > >> has
>> > >     >> >> >>>>>     >     > > >> > >> >> >> access to the unencrypted
>> > > data. For
>> > >     >> >> >>>> end-to-end
>> > >     >> >> >>>>>     > encryption
>> > >     >> >> >>>>>     >     > you
>> > >     >> >> >>>>>     >     > > >> > need to
>> > >     >> >> >>>>>     >     > > >> > >> >> >> encrypt at the producer,
>> > > before it
>> > >     >> >> enters
>> > >     >> >> >> the
>> > >     >> >> >>>>>     > system, and
>> > >     >> >> >>>>>     >     > > >> decrypt
>> > >     >> >> >>>>>     >     > > >> > at
>> > >     >> >> >>>>>     >     > > >> > >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >> consumer, after it exits
>> the
>> > > system.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >> -Todd
>> > >     >> >> >>>>>     >     > > >> > >> >> >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03
>> > > PM, radai
>> > >     >> <
>> > >     >> >> >>>>>     >     > > >> radai.rosenblatt@gmail.com
>> > >     >> >> >>>>>     >     > > >> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> another big plus of
>> headers
>> > > in the
>> > >     >> >> >> protocol
>> > >     >> >> >>>> is that
>> > >     >> >> >>>>>     > it
>> > >     >> >> >>>>>     >     > would
>> > >     >> >> >>>>>     >     > > >> > enable
>> > >     >> >> >>>>>     >     > > >> > >> >> rapid
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> iteration on ideas outside
>> > of
>> > > core
>> > >     >> >> kafka
>> > >     >> >> >>>> and would
>> > >     >> >> >>>>>     > reduce
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > >> number of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> future wire format changes
>> > > required.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> a lot of what is
>> currently a
>> > > KIP
>> > >     >> >> >> represents
>> > >     >> >> >>>> use
>> > >     >> >> >>>>>     > cases that
>> > >     >> >> >>>>>     >     > > are
>> > >     >> >> >>>>>     >     > > >> > not
>> > >     >> >> >>>>>     >     > > >> > >> 100%
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> relevant to all users, and
>> > > some of
>> > >     >> them
>> > >     >> >> >>>> require
>> > >     >> >> >>>>>     > rather
>> > >     >> >> >>>>>     >     > > invasive
>> > >     >> >> >>>>>     >     > > >> > wire
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> protocol changes. a thing
>> a
>> > > good
>> > >     >> recent
>> > >     >> >> >>>> example of
>> > >     >> >> >>>>>     > this is
>> > >     >> >> >>>>>     >     > > >> > kip-98.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> tx-utilizing traffic is
>> > > expected to
>> > >     >> be
>> > >     >> >> a
>> > >     >> >> >>>> very small
>> > >     >> >> >>>>>     >     > > fraction of
>> > >     >> >> >>>>>     >     > > >> > >> total
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> traffic and yet the
>> changes
>> > > are
>> > >     >> >> invasive.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> every such wire format
>> > change
>> > >     >> >> translates
>> > >     >> >> >>>> into
>> > >     >> >> >>>>>     > painful and
>> > >     >> >> >>>>>     >     > > slow
>> > >     >> >> >>>>>     >     > > >> > >> >> adoption of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> new versions.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> i think a lot of
>> > functionality
>> > >     >> >> currently
>> > >     >> >> >> in
>> > >     >> >> >>>> KIPs
>> > >     >> >> >>>>>     > could be
>> > >     >> >> >>>>>     >     > > "spun
>> > >     >> >> >>>>>     >     > > >> > out"
>> > >     >> >> >>>>>     >     > > >> > >> >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> implemented as opt-in
>> > plugins
>> > >     >> >> transmitting
>> > >     >> >> >>>> data over
>> > >     >> >> >>>>>     >     > > headers.
>> > >     >> >> >>>>>     >     > > >> > this
>> > >     >> >> >>>>>     >     > > >> > >> >> would
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> keep the core wire format
>> > > stable(r),
>> > >     >> >> core
>> > >     >> >> >>>> codebase
>> > >     >> >> >>>>>     >     > smaller,
>> > >     >> >> >>>>>     >     > > and
>> > >     >> >> >>>>>     >     > > >> > >> avoid
>> > >     >> >> >>>>>     >     > > >> > >> >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> "burden of proof" thats
>> > > sometimes
>> > >     >> >> required
>> > >     >> >> >>>> to prove
>> > >     >> >> >>>>>     > a
>> > >     >> >> >>>>>     >     > > certain
>> > >     >> >> >>>>>     >     > > >> > >> feature
>> > >     >> >> >>>>>     >     > > >> > >> >> is
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> useful enough for a
>> > > wide-enough
>> > >     >> >> audience
>> > >     >> >> >> to
>> > >     >> >> >>>> warrant
>> > >     >> >> >>>>>     > a wire
>> > >     >> >> >>>>>     >     > > >> format
>> > >     >> >> >>>>>     >     > > >> > >> >> change
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> and code complexity
>> > additions.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> (to be clear - kip-98 goes
>> > > beyond
>> > >     >> >> "mere"
>> > >     >> >> >>>> wire format
>> > >     >> >> >>>>>     >     > changes
>> > >     >> >> >>>>>     >     > > >> and
>> > >     >> >> >>>>>     >     > > >> > im
>> > >     >> >> >>>>>     >     > > >> > >> not
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> saying it could have been
>> > > completely
>> > >     >> >> done
>> > >     >> >> >>>> with
>> > >     >> >> >>>>>     > headers,
>> > >     >> >> >>>>>     >     > but
>> > >     >> >> >>>>>     >     > > >> > >> >> exactly-once
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> delivery certainly could)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> On Thu, Dec 1, 2016 at
>> 11:20
>> > > AM, Gwen
>> > >     >> >> >>>> Shapira <
>> > >     >> >> >>>>>     >     > > >> gwen@confluent.io
>> > >     >> >> >>>>>     >     > > >> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > On Thu, Dec 1, 2016 at
>> > > 10:24 AM,
>> > >     >> >> radai <
>> > >     >> >> >>>>>     >     > > >> > >> radai.rosenblatt@gmail.com>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > "For use cases within
>> an
>> > >     >> >> organization,
>> > >     >> >> >>>> one could
>> > >     >> >> >>>>>     >     > always
>> > >     >> >> >>>>>     >     > > use
>> > >     >> >> >>>>>     >     > > >> > >> other
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > approaches such as
>> > > company-wise
>> > >     >> >> >>>> containers"
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > this is what linkedin
>> > has
>> > >     >> >> >> traditionally
>> > >     >> >> >>>> done
>> > >     >> >> >>>>>     > but there
>> > >     >> >> >>>>>     >     > > are
>> > >     >> >> >>>>>     >     > > >> > now
>> > >     >> >> >>>>>     >     > > >> > >> >> cases
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > (read
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > - topics) where this
>> is
>> > > not
>> > >     >> >> >> acceptable.
>> > >     >> >> >>>> this
>> > >     >> >> >>>>>     > makes
>> > >     >> >> >>>>>     >     > > headers
>> > >     >> >> >>>>>     >     > > >> > >> useful
>> > >     >> >> >>>>>     >     > > >> > >> >> even
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > within single orgs for
>> > > cases
>> > >     >> where
>> > >     >> >> >>>>>     >     > > one-container-fits-all
>> > >     >> >> >>>>>     >     > > >> > cannot
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> apply.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > as for the particular
>> > use
>> > > cases
>> > >     >> >> >> listed,
>> > >     >> >> >>>> i dont
>> > >     >> >> >>>>>     > want
>> > >     >> >> >>>>>     >     > > this to
>> > >     >> >> >>>>>     >     > > >> > >> devolve
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> to a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > discussion of
>> particular
>> > > use
>> > >     >> cases
>> > >     >> >> - i
>> > >     >> >> >>>> think its
>> > >     >> >> >>>>>     >     > enough
>> > >     >> >> >>>>>     >     > > >> that
>> > >     >> >> >>>>>     >     > > >> > >> some
>> > >     >> >> >>>>>     >     > > >> > >> >> of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> them
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > I think a main point of
>> > > contention
>> > >     >> is
>> > >     >> >> >>>> that: We
>> > >     >> >> >>>>>     >     > identified
>> > >     >> >> >>>>>     >     > > few
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > use-cases where headers
>> > are
>> > > useful,
>> > >     >> >> do
>> > >     >> >> >> we
>> > >     >> >> >>>> want
>> > >     >> >> >>>>>     > Kafka to
>> > >     >> >> >>>>>     >     > > be a
>> > >     >> >> >>>>>     >     > > >> > >> system
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > that supports those
>> > > use-cases?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > For example, Jun said:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > "Not sure how widely
>> > useful
>> > >     >> >> record-level
>> > >     >> >> >>>> lineage
>> > >     >> >> >>>>>     > is
>> > >     >> >> >>>>>     >     > though
>> > >     >> >> >>>>>     >     > > >> > since
>> > >     >> >> >>>>>     >     > > >> > >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > overhead could
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > be significant."
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > We know NiFi supports
>> > > record level
>> > >     >> >> >>>> lineage. I
>> > >     >> >> >>>>>     > don't
>> > >     >> >> >>>>>     >     > think
>> > >     >> >> >>>>>     >     > > it
>> > >     >> >> >>>>>     >     > > >> > was
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > developed for lols, I
>> > think
>> > > it is
>> > >     >> >> safe
>> > >     >> >> >> to
>> > >     >> >> >>>> assume
>> > >     >> >> >>>>>     > that
>> > >     >> >> >>>>>     >     > the
>> > >     >> >> >>>>>     >     > > NSA
>> > >     >> >> >>>>>     >     > > >> > >> needed
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > that functionality. We
>> > also
>> > > know
>> > >     >> that
>> > >     >> >> >>>> certain
>> > >     >> >> >>>>>     > financial
>> > >     >> >> >>>>>     >     > > >> > institutes
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > need to track tampering
>> > with
>> > >     >> records
>> > >     >> >> at
>> > >     >> >> >> a
>> > >     >> >> >>>> record
>> > >     >> >> >>>>>     > level
>> > >     >> >> >>>>>     >     > and
>> > >     >> >> >>>>>     >     > > >> > there
>> > >     >> >> >>>>>     >     > > >> > >> are
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > federal regulations that
>> > > absolutely
>> > >     >> >> >>>> require
>> > >     >> >> >>>>>     > this.  They
>> > >     >> >> >>>>>     >     > > also
>> > >     >> >> >>>>>     >     > > >> > need
>> > >     >> >> >>>>>     >     > > >> > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > prove that routing apps
>> > that
>> > >     >> >> "touches"
>> > >     >> >> >> the
>> > >     >> >> >>>>>     > messages and
>> > >     >> >> >>>>>     >     > > >> either
>> > >     >> >> >>>>>     >     > > >> > >> reads
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > or updates headers
>> > couldn't
>> > > have
>> > >     >> >> >> possibly
>> > >     >> >> >>>>>     > modified the
>> > >     >> >> >>>>>     >     > > >> payload
>> > >     >> >> >>>>>     >     > > >> > >> >> itself.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > They use record level
>> > > encryption to
>> > >     >> >> do
>> > >     >> >> >>>> that -
>> > >     >> >> >>>>>     > apps can
>> > >     >> >> >>>>>     >     > > read
>> > >     >> >> >>>>>     >     > > >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > (sometimes) modify
>> headers
>> > > but
>> > >     >> can't
>> > >     >> >> >>>> touch the
>> > >     >> >> >>>>>     > payload.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > We can totally say
>> "those
>> > > are
>> > >     >> corner
>> > >     >> >> >>>> cases and
>> > >     >> >> >>>>>     > not worth
>> > >     >> >> >>>>>     >     > > >> adding
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > headers to Kafka for",
>> > they
>> > > should
>> > >     >> >> use a
>> > >     >> >> >>>> different
>> > >     >> >> >>>>>     >     > pubsub
>> > >     >> >> >>>>>     >     > > >> > message
>> > >     >> >> >>>>>     >     > > >> > >> for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > that (Nifi or one of the
>> > > other 1000
>> > >     >> >> that
>> > >     >> >> >>>> cater
>> > >     >> >> >>>>>     >     > > specifically
>> > >     >> >> >>>>>     >     > > >> to
>> > >     >> >> >>>>>     >     > > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > financial industry).
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > But this gets us into a
>> > > catch 22:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > If we discuss a specific
>> > > use-case,
>> > >     >> >> >>>> someone can
>> > >     >> >> >>>>>     > always
>> > >     >> >> >>>>>     >     > say
>> > >     >> >> >>>>>     >     > > it
>> > >     >> >> >>>>>     >     > > >> > isn't
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > interesting enough for
>> > > Kafka. If we
>> > >     >> >> >>>> discuss more
>> > >     >> >> >>>>>     > general
>> > >     >> >> >>>>>     >     > > >> > trends,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > others can say "well, we
>> > > are not
>> > >     >> sure
>> > >     >> >> >> any
>> > >     >> >> >>>> of them
>> > >     >> >> >>>>>     > really
>> > >     >> >> >>>>>     >     > > >> needs
>> > >     >> >> >>>>>     >     > > >> > >> >> headers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > specifically. This is
>> just
>> > > hand
>> > >     >> >> waving
>> > >     >> >> >>>> and not
>> > >     >> >> >>>>>     >     > > interesting.".
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > I think discussing
>> > > use-cases in
>> > >     >> >> >> specifics
>> > >     >> >> >>>> is super
>> > >     >> >> >>>>>     >     > > important
>> > >     >> >> >>>>>     >     > > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> decide
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > implementation details
>> for
>> > > headers
>> > >     >> >> (my
>> > >     >> >> >>>> use-cases
>> > >     >> >> >>>>>     > lean
>> > >     >> >> >>>>>     >     > > toward
>> > >     >> >> >>>>>     >     > > >> > >> >> numerical
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > keys with namespaces and
>> > > object
>> > >     >> >> values,
>> > >     >> >> >>>> others
>> > >     >> >> >>>>>     > differ),
>> > >     >> >> >>>>>     >     > > but I
>> > >     >> >> >>>>>     >     > > >> > >> think
>> > >     >> >> >>>>>     >     > > >> > >> >> we
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > need to answer the
>> general
>> > > "Are we
>> > >     >> >> going
>> > >     >> >> >>>> to have
>> > >     >> >> >>>>>     >     > headers"
>> > >     >> >> >>>>>     >     > > >> > question
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > first.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > I'd love to hear from
>> the
>> > > other
>> > >     >> >> >>>> committers in the
>> > >     >> >> >>>>>     >     > > discussion:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > What would it take to
>> > > convince you
>> > >     >> >> that
>> > >     >> >> >>>> headers
>> > >     >> >> >>>>>     > in Kafka
>> > >     >> >> >>>>>     >     > > are
>> > >     >> >> >>>>>     >     > > >> a
>> > >     >> >> >>>>>     >     > > >> > >> good
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > idea in general, so we
>> can
>> > > move
>> > >     >> ahead
>> > >     >> >> >> and
>> > >     >> >> >>>> try to
>> > >     >> >> >>>>>     > agree
>> > >     >> >> >>>>>     >     > on
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > >> >> details?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > I feel like we keep
>> moving
>> > > the goal
>> > >     >> >> >> posts
>> > >     >> >> >>>> and
>> > >     >> >> >>>>>     > this is
>> > >     >> >> >>>>>     >     > > truly
>> > >     >> >> >>>>>     >     > > >> > >> >> exhausting.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > For the record, I mildly
>> > > support
>> > >     >> >> adding
>> > >     >> >> >>>> headers
>> > >     >> >> >>>>>     > to Kafka
>> > >     >> >> >>>>>     >     > > >> > (+0.5?).
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > The community can
>> continue
>> > > to find
>> > >     >> >> >>>> workarounds to
>> > >     >> >> >>>>>     > the
>> > >     >> >> >>>>>     >     > > issue
>> > >     >> >> >>>>>     >     > > >> and
>> > >     >> >> >>>>>     >     > > >> > >> there
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > are some benefits to
>> > > keeping the
>> > >     >> >> message
>> > >     >> >> >>>> format
>> > >     >> >> >>>>>     > and
>> > >     >> >> >>>>>     >     > > clients
>> > >     >> >> >>>>>     >     > > >> > >> simpler.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > But I see the usefulness
>> > of
>> > > headers
>> > >     >> >> to
>> > >     >> >> >>>> many
>> > >     >> >> >>>>>     > use-cases
>> > >     >> >> >>>>>     >     > and
>> > >     >> >> >>>>>     >     > > if
>> > >     >> >> >>>>>     >     > > >> we
>> > >     >> >> >>>>>     >     > > >> > >> can
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > find a good and
>> generally
>> > > useful
>> > >     >> way
>> > >     >> >> to
>> > >     >> >> >>>> add it to
>> > >     >> >> >>>>>     > Kafka,
>> > >     >> >> >>>>>     >     > > it
>> > >     >> >> >>>>>     >     > > >> > will
>> > >     >> >> >>>>>     >     > > >> > >> make
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > Kafka easier to use for
>> > > many -
>> > >     >> worthy
>> > >     >> >> >>>> goal in my
>> > >     >> >> >>>>>     > eyes.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > are
>> > interesting/feasible,
>> > > but:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > A+B. i think there are
>> > > use cases
>> > >     >> >> for
>> > >     >> >> >>>> polyglot
>> > >     >> >> >>>>>     > topics.
>> > >     >> >> >>>>>     >     > > >> > >> especially if
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> kafka
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > is being used to
>> "trunk"
>> > >     >> something
>> > >     >> >> >> else.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > D. multiple topics
>> would
>> > > make it
>> > >     >> >> >> harder
>> > >     >> >> >>>> to write
>> > >     >> >> >>>>>     >     > > portable
>> > >     >> >> >>>>>     >     > > >> > >> consumer
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> code.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > partition remapping
>> > would
>> > > mess
>> > >     >> with
>> > >     >> >> >>>> locality of
>> > >     >> >> >>>>>     >     > > consumption
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> guarantees.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > E+F. a use case I see
>> > for
>> > >     >> >> >>>> lineage/metadata is
>> > >     >> >> >>>>>     >     > > >> > >> billing/chargeback.
>> > >     >> >> >>>>>     >     > > >> > >> >> for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > that
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > use case it is not
>> > enough
>> > > to
>> > >     >> simply
>> > >     >> >> >>>> record the
>> > >     >> >> >>>>>     > point
>> > >     >> >> >>>>>     >     > of
>> > >     >> >> >>>>>     >     > > >> > origin,
>> > >     >> >> >>>>>     >     > > >> > >> but
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> every
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > replication stop
>> (think
>> > > mirror
>> > >     >> >> maker)
>> > >     >> >> >>>> must also
>> > >     >> >> >>>>>     > add a
>> > >     >> >> >>>>>     >     > > >> record
>> > >     >> >> >>>>>     >     > > >> > to
>> > >     >> >> >>>>>     >     > > >> > >> >> form a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > "transit log".
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > as for stream
>> processing
>> > > on top
>> > >     >> of
>> > >     >> >> >>>> kafka - i
>> > >     >> >> >>>>>     > know
>> > >     >> >> >>>>>     >     > samza
>> > >     >> >> >>>>>     >     > > >> has a
>> > >     >> >> >>>>>     >     > > >> > >> >> metadata
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > map
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > which they carry
>> around
>> > in
>> > >     >> >> addition to
>> > >     >> >> >>>> user
>> > >     >> >> >>>>>     > values.
>> > >     >> >> >>>>>     >     > > headers
>> > >     >> >> >>>>>     >     > > >> > are
>> > >     >> >> >>>>>     >     > > >> > >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > perfect
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > fit for these things.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > > On Wed, Nov 30, 2016
>> at
>> > > 6:50 PM,
>> > >     >> >> Jun
>> > >     >> >> >>>> Rao <
>> > >     >> >> >>>>>     >     > > jun@confluent.io
>> > >     >> >> >>>>>     >     > > >> >
>> > >     >> >> >>>>>     >     > > >> > >> wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> Hi, Michael,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> In order to answer
>> the
>> > > first two
>> > >     >> >> >>>> questions, it
>> > >     >> >> >>>>>     > would
>> > >     >> >> >>>>>     >     > be
>> > >     >> >> >>>>>     >     > > >> > helpful
>> > >     >> >> >>>>>     >     > > >> > >> >> if we
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > could
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> identify 1 or 2
>> strong
>> > > use cases
>> > >     >> >> for
>> > >     >> >> >>>> headers
>> > >     >> >> >>>>>     > in the
>> > >     >> >> >>>>>     >     > > space
>> > >     >> >> >>>>>     >     > > >> > for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > third-party
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> vendors. For use
>> cases
>> > > within an
>> > >     >> >> >>>> organization,
>> > >     >> >> >>>>>     > one
>> > >     >> >> >>>>>     >     > > could
>> > >     >> >> >>>>>     >     > > >> > always
>> > >     >> >> >>>>>     >     > > >> > >> >> use
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > other
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> approaches such as
>> > > company-wise
>> > >     >> >> >>>> containers to
>> > >     >> >> >>>>>     > get
>> > >     >> >> >>>>>     >     > > around
>> > >     >> >> >>>>>     >     > > >> w/o
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> headers. I
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> went through the use
>> > > cases in
>> > >     >> the
>> > >     >> >> KIP
>> > >     >> >> >>>> and in
>> > >     >> >> >>>>>     > Radai's
>> > >     >> >> >>>>>     >     > > wiki
>> > >     >> >> >>>>>     >     > > >> (
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > > https://cwiki.apache.org/confl
>> > >     >> >> >>>>>     > uence/display/KAFKA/A+
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > Case+for+Kafka+Headers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> ).
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> The following are the
>> > > ones that
>> > >     >> >> that
>> > >     >> >> >> I
>> > >     >> >> >>>>>     > understand and
>> > >     >> >> >>>>>     >     > > >> could
>> > >     >> >> >>>>>     >     > > >> > be
>> > >     >> >> >>>>>     >     > > >> > >> in
>> > >     >> >> >>>>>     >     > > >> > >> >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> third-party use case
>> > > category.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> A. content-type
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> It seems that in
>> > general,
>> > >     >> >> >> content-type
>> > >     >> >> >>>> should
>> > >     >> >> >>>>>     > be set
>> > >     >> >> >>>>>     >     > at
>> > >     >> >> >>>>>     >     > > >> the
>> > >     >> >> >>>>>     >     > > >> > >> topic
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> level.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> Not sure if mixing
>> > > messages with
>> > >     >> >> >>>> different
>> > >     >> >> >>>>>     > content
>> > >     >> >> >>>>>     >     > > types
>> > >     >> >> >>>>>     >     > > >> > >> should be
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> encouraged.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> B. schema id
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> Since the value is
>> > mostly
>> > >     >> useless
>> > >     >> >> >>>> without
>> > >     >> >> >>>>>     > schema id,
>> > >     >> >> >>>>>     >     > it
>> > >     >> >> >>>>>     >     > > >> > seems
>> > >     >> >> >>>>>     >     > > >> > >> that
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > storing
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> the schema id
>> together
>> > > with
>> > >     >> >> >> serialized
>> > >     >> >> >>>> bytes
>> > >     >> >> >>>>>     > in the
>> > >     >> >> >>>>>     >     > > value
>> > >     >> >> >>>>>     >     > > >> is
>> > >     >> >> >>>>>     >     > > >> > >> >> better?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> C. per message
>> > encryption
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> One drawback of this
>> > > approach is
>> > >     >> >> that
>> > >     >> >> >>>> this
>> > >     >> >> >>>>>     >     > > significantly
>> > >     >> >> >>>>>     >     > > >> > reduce
>> > >     >> >> >>>>>     >     > > >> > >> >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> effectiveness of
>> > > compression,
>> > >     >> >> which
>> > >     >> >> >>>> happens on
>> > >     >> >> >>>>>     > a set
>> > >     >> >> >>>>>     >     > of
>> > >     >> >> >>>>>     >     > > >> > >> serialized
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> messages. An
>> > alternative
>> > > is to
>> > >     >> >> enable
>> > >     >> >> >>>> SSL for
>> > >     >> >> >>>>>     > wire
>> > >     >> >> >>>>>     >     > > >> > encryption
>> > >     >> >> >>>>>     >     > > >> > >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> rely
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > on
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> the storage system
>> > (e.g.
>> > > LUKS)
>> > >     >> >> for at
>> > >     >> >> >>>> rest
>> > >     >> >> >>>>>     >     > encryption.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> D. cluster ID for
>> > > mirroring
>> > >     >> across
>> > >     >> >> >>>> Kafka
>> > >     >> >> >>>>>     > clusters
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> This is actually
>> > > interesting.
>> > >     >> >> Today,
>> > >     >> >> >>>> to avoid
>> > >     >> >> >>>>>     >     > > introducing
>> > >     >> >> >>>>>     >     > > >> > >> cycles
>> > >     >> >> >>>>>     >     > > >> > >> >> when
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > doing
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> mirroring across data
>> > > centers,
>> > >     >> one
>> > >     >> >> >>>> would
>> > >     >> >> >>>>>     > either have
>> > >     >> >> >>>>>     >     > to
>> > >     >> >> >>>>>     >     > > >> set
>> > >     >> >> >>>>>     >     > > >> > up
>> > >     >> >> >>>>>     >     > > >> > >> two
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> Kafka
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> clusters (a local and
>> > an
>> > >     >> >> aggregate)
>> > >     >> >> >>>> per data
>> > >     >> >> >>>>>     > center
>> > >     >> >> >>>>>     >     > or
>> > >     >> >> >>>>>     >     > > >> > rename
>> > >     >> >> >>>>>     >     > > >> > >> >> topics.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> Neither is ideal.
>> With
>> > > headers,
>> > >     >> >> the
>> > >     >> >> >>>> producer
>> > >     >> >> >>>>>     > could
>> > >     >> >> >>>>>     >     > tag
>> > >     >> >> >>>>>     >     > > >> each
>> > >     >> >> >>>>>     >     > > >> > >> >> message
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> with
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> the producing cluster
>> > ID
>> > > in the
>> > >     >> >> >> header.
>> > >     >> >> >>>>>     > MirrorMaker
>> > >     >> >> >>>>>     >     > > could
>> > >     >> >> >>>>>     >     > > >> > then
>> > >     >> >> >>>>>     >     > > >> > >> >> avoid
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> mirroring messages
>> to a
>> > > cluster
>> > >     >> if
>> > >     >> >> >>>> they are
>> > >     >> >> >>>>>     > tagged
>> > >     >> >> >>>>>     >     > with
>> > >     >> >> >>>>>     >     > > >> the
>> > >     >> >> >>>>>     >     > > >> > >> same
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> cluster
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> id.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> However, an
>> alternative
>> > > approach
>> > >     >> >> is
>> > >     >> >> >> to
>> > >     >> >> >>>>>     > introduce sth
>> > >     >> >> >>>>>     >     > > like
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> hierarchical
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> topic and store
>> > messages
>> > > from
>> > >     >> >> >> different
>> > >     >> >> >>>>>     > clusters in
>> > >     >> >> >>>>>     >     > > >> > different
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> partitions
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> under the same topic.
>> > > This
>> > >     >> >> approach
>> > >     >> >> >>>> avoids
>> > >     >> >> >>>>>     > filtering
>> > >     >> >> >>>>>     >     > > out
>> > >     >> >> >>>>>     >     > > >> > >> unneeded
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> data
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> makes offset
>> preserving
>> > > easier
>> > >     >> to
>> > >     >> >> >>>> support. It
>> > >     >> >> >>>>>     > may
>> > >     >> >> >>>>>     >     > make
>> > >     >> >> >>>>>     >     > > >> > >> compaction
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > trickier
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> though since the same
>> > > key may
>> > >     >> >> show up
>> > >     >> >> >>>> in
>> > >     >> >> >>>>>     > different
>> > >     >> >> >>>>>     >     > > >> > partitions.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> E. record-level
>> lineage
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> For example, a source
>> > > connector
>> > >     >> >> could
>> > >     >> >> >>>> store in
>> > >     >> >> >>>>>     > the
>> > >     >> >> >>>>>     >     > > message
>> > >     >> >> >>>>>     >     > > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> metadata
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> (e.g. UUID) of the
>> > source
>> > >     >> record.
>> > >     >> >> >>>> Similarly,
>> > >     >> >> >>>>>     > if a
>> > >     >> >> >>>>>     >     > > stream
>> > >     >> >> >>>>>     >     > > >> job
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> transforms
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> messages from topic A
>> > to
>> > > topic
>> > >     >> B,
>> > >     >> >> the
>> > >     >> >> >>>> library
>> > >     >> >> >>>>>     > could
>> > >     >> >> >>>>>     >     > > >> include
>> > >     >> >> >>>>>     >     > > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> source
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> message offset in
>> each
>> > > of the
>> > >     >> >> >>>> transformed
>> > >     >> >> >>>>>     > message in
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > >> header.
>> > >     >> >> >>>>>     >     > > >> > >> >> Not
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > sure
>> > >     >> >> >>>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> how widely useful
>> > > record-level
>> > >     >> >> >> lineage
>> > >     >> >> >>>> is
>> > >     >> >> >>>>>     > though
>> > >     >> >> >>>>>     >     > since
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > >> >> overhead
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > could
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> be significant.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> F. auditing metadata
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> We could put things
>> > like
>> > >     >> >> >>>> clientId/host/user in
>> > >     >> >> >>>>>     > the
>> > >     >> >> >>>>>     >     > > header
>> > >     >> >> >>>>>     >     > > >> in
>> > >     >> >> >>>>>     >     > > >> > >> each
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > message
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> for auditing. These
>> > > metadata are
>> > >     >> >> >>>> really at the
>> > >     >> >> >>>>>     >     > producer
>> > >     >> >> >>>>>     >     > > >> > level
>> > >     >> >> >>>>>     >     > > >> > >> >> though.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > So, a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> more efficient way is
>> > to
>> > > only
>> > >     >> >> >> include a
>> > >     >> >> >>>>>     > "producerId"
>> > >     >> >> >>>>>     >     > > per
>> > >     >> >> >>>>>     >     > > >> > >> message
>> > >     >> >> >>>>>     >     > > >> > >> >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > send
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> the producerId ->
>> > > metadata
>> > >     >> mapping
>> > >     >> >> >>>>>     > independently.
>> > >     >> >> >>>>>     >     > > KIP-98
>> > >     >> >> >>>>>     >     > > >> is
>> > >     >> >> >>>>>     >     > > >> > >> >> actually
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> proposing including
>> > such
>> > > a
>> > >     >> >> producerId
>> > >     >> >> >>>> natively
>> > >     >> >> >>>>>     > in the
>> > >     >> >> >>>>>     >     > > >> > message.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> So, overall, I not
>> sure
>> > > that I
>> > >     >> am
>> > >     >> >> >> fully
>> > >     >> >> >>>>>     > convinced of
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > strong
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > third-party
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> use cases of headers
>> > yet.
>> > >     >> Perhaps
>> > >     >> >> we
>> > >     >> >> >>>> could
>> > >     >> >> >>>>>     > discuss a
>> > >     >> >> >>>>>     >     > > bit
>> > >     >> >> >>>>>     >     > > >> > more
>> > >     >> >> >>>>>     >     > > >> > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> make
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > one
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> or two really
>> > convincing
>> > > use
>> > >     >> >> cases.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> Another orthogonal
>> > > question is
>> > >     >> >> >>>> whether header
>> > >     >> >> >>>>>     > should
>> > >     >> >> >>>>>     >     > > be
>> > >     >> >> >>>>>     >     > > >> > >> exposed
>> > >     >> >> >>>>>     >     > > >> > >> >> in
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > stream
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> processing systems
>> such
>> > > Kafka
>> > >     >> >> stream,
>> > >     >> >> >>>> Samza,
>> > >     >> >> >>>>>     > and
>> > >     >> >> >>>>>     >     > Spark
>> > >     >> >> >>>>>     >     > > >> > >> streaming.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> Currently, those
>> > systems
>> > > just
>> > >     >> deal
>> > >     >> >> >> with
>> > >     >> >> >>>>>     > key/value
>> > >     >> >> >>>>>     >     > > pairs.
>> > >     >> >> >>>>>     >     > > >> > >> Should we
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > expose a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> third thing header
>> > there
>> > > too or
>> > >     >> >> >>>> somehow map
>> > >     >> >> >>>>>     > header to
>> > >     >> >> >>>>>     >     > > key
>> > >     >> >> >>>>>     >     > > >> or
>> > >     >> >> >>>>>     >     > > >> > >> >> value?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> Thanks,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> Jun
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> On Tue, Nov 29, 2016
>> at
>> > > 3:35 AM,
>> > >     >> >> >>>> Michael
>> > >     >> >> >>>>>     > Pearce <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> Michael.Pearce@ig.com>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > I assume, that
>> after
>> > a
>> > > period
>> > >     >> >> of a
>> > >     >> >> >>>> week,
>> > >     >> >> >>>>>     > that there
>> > >     >> >> >>>>>     >     > > is
>> > >     >> >> >>>>>     >     > > >> no
>> > >     >> >> >>>>>     >     > > >> > >> >> concerns
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> now
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > with points 1, and
>> 2
>> > > and now
>> > >     >> we
>> > >     >> >> >> have
>> > >     >> >> >>>>>     > agreement that
>> > >     >> >> >>>>>     >     > > >> > headers
>> > >     >> >> >>>>>     >     > > >> > >> are
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> useful
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > needed in Kafka. As
>> > > such if
>> > >     >> put
>> > >     >> >> to
>> > >     >> >> >> a
>> > >     >> >> >>>> KIP
>> > >     >> >> >>>>>     > vote, this
>> > >     >> >> >>>>>     >     > > >> > wouldn’t
>> > >     >> >> >>>>>     >     > > >> > >> be
>> > >     >> >> >>>>>     >     > > >> > >> >> a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > reason
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > reject.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > @
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > Ignacio on point
>> 4).
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > I think for purpose
>> > of
>> > > getting
>> > >     >> >> this
>> > >     >> >> >>>> KIP
>> > >     >> >> >>>>>     > moving past
>> > >     >> >> >>>>>     >     > > >> this,
>> > >     >> >> >>>>>     >     > > >> > we
>> > >     >> >> >>>>>     >     > > >> > >> can
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> state
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > key will be a 4
>> bytes
>> > > space
>> > >     >> that
>> > >     >> >> >> can
>> > >     >> >> >>>> will be
>> > >     >> >> >>>>>     >     > > naturally
>> > >     >> >> >>>>>     >     > > >> > >> >> interpreted
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> as
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > an
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > Int32 (if
>> namespacing
>> > > is later
>> > >     >> >> >>>> wanted you can
>> > >     >> >> >>>>>     >     > easily
>> > >     >> >> >>>>>     >     > > >> split
>> > >     >> >> >>>>>     >     > > >> > >> this
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> into
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > two
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > int16 spaces), from
>> > > the wire
>> > >     >> >> >> protocol
>> > >     >> >> >>>>>     >     > implementation
>> > >     >> >> >>>>>     >     > > >> this
>> > >     >> >> >>>>>     >     > > >> > >> makes
>> > >     >> >> >>>>>     >     > > >> > >> >> no
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > difference I don’t
>> > > believe. Is
>> > >     >> >> this
>> > >     >> >> >>>>>     > reasonable to
>> > >     >> >> >>>>>     >     > > all?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > On 5) as per point
>> 4
>> > > therefor
>> > >     >> >> happy
>> > >     >> >> >>>> we keep
>> > >     >> >> >>>>>     > with 32
>> > >     >> >> >>>>>     >     > > >> bits.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > On 18/11/2016,
>> > 20:34, "
>> > >     >> >> >>>>>     > ignacio.solis@gmail.com on
>> > >     >> >> >>>>>     >     > > >> behalf
>> > >     >> >> >>>>>     >     > > >> > of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> Ignacio
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > Solis" <
>> > >     >> ignacio.solis@gmail.com
>> > >     >> >> on
>> > >     >> >> >>>> behalf of
>> > >     >> >> >>>>>     >     > > >> > isolis@igso.net
>> > >     >> >> >>>>>     >     > > >> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     Summary:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     3) Yes - Header
>> > > value as
>> > >     >> >> byte[]
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     4a) Int,Int -
>> No
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     4b) Int - Yes
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     4c) String -
>> > > Reluctant
>> > >     >> maybe
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     5) I believe
>> the
>> > > header
>> > >     >> >> system
>> > >     >> >> >>>> should
>> > >     >> >> >>>>>     > take a
>> > >     >> >> >>>>>     >     > > single
>> > >     >> >> >>>>>     >     > > >> > >> int.  I
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> think
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > 32bits is
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     a good size, if
>> > > you want
>> > >     >> to
>> > >     >> >> >>>> interpret
>> > >     >> >> >>>>>     > this as
>> > >     >> >> >>>>>     >     > to
>> > >     >> >> >>>>>     >     > > >> 16bit
>> > >     >> >> >>>>>     >     > > >> > >> >> numbers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> in
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > layer
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     above go right
>> > > ahead.  If
>> > >     >> >> >>>> somebody wants
>> > >     >> >> >>>>>     > to
>> > >     >> >> >>>>>     >     > argue
>> > >     >> >> >>>>>     >     > > >> for
>> > >     >> >> >>>>>     >     > > >> > 16
>> > >     >> >> >>>>>     >     > > >> > >> >> bits
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> or
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > 64
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > bits of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     header key
>> space
>> > I
>> > > would
>> > >     >> >> >> listen.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     Discussion:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     Dividing the
>> key
>> > > space
>> > >     >> into
>> > >     >> >> >>>> sub_key_1 and
>> > >     >> >> >>>>>     >     > > sub_key_2
>> > >     >> >> >>>>>     >     > > >> > >> makes no
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > sense to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > me at
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     this layer.
>> Are
>> > > we going
>> > >     >> to
>> > >     >> >> >>>> start
>> > >     >> >> >>>>>     > providing
>> > >     >> >> >>>>>     >     > > APIs to
>> > >     >> >> >>>>>     >     > > >> > get
>> > >     >> >> >>>>>     >     > > >> > >> all
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     sub_key_1s? or
>> > all
>> > > the
>> > >     >> >> >>>> sub_key_2s?  If
>> > >     >> >> >>>>>     > there is
>> > >     >> >> >>>>>     >     > > no
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> distinguishing
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > functions
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     that are
>> applied
>> > > to each
>> > >     >> one
>> > >     >> >> >>>> then they
>> > >     >> >> >>>>>     > should
>> > >     >> >> >>>>>     >     > be
>> > >     >> >> >>>>>     >     > > a
>> > >     >> >> >>>>>     >     > > >> > single
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> value.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > At
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > this
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     layer all we're
>> > > doing is
>> > >     >> >> >>>> equality.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     If the above
>> > layer
>> > > wants
>> > >     >> to
>> > >     >> >> >>>> interpret
>> > >     >> >> >>>>>     > this as
>> > >     >> >> >>>>>     >     > 2,
>> > >     >> >> >>>>>     >     > > 3
>> > >     >> >> >>>>>     >     > > >> or
>> > >     >> >> >>>>>     >     > > >> > >> more
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> values
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > that's a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     different
>> > > question.  I
>> > >     >> >> >>>> personally think
>> > >     >> >> >>>>>     > it's
>> > >     >> >> >>>>>     >     > all
>> > >     >> >> >>>>>     >     > > one
>> > >     >> >> >>>>>     >     > > >> > >> >> keyspace
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > that is
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     getting
>> assigned
>> > > using
>> > >     >> some
>> > >     >> >> >>>> structure,
>> > >     >> >> >>>>>     > but if
>> > >     >> >> >>>>>     >     > you
>> > >     >> >> >>>>>     >     > > >> > want to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > sub-assign
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > parts
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     of it then
>> that's
>> > > fine.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     The same
>> > discussion
>> > >     >> applies
>> > >     >> >> to
>> > >     >> >> >>>> strings.
>> > >     >> >> >>>>>     > If
>> > >     >> >> >>>>>     >     > > somebody
>> > >     >> >> >>>>>     >     > > >> > >> argued
>> > >     >> >> >>>>>     >     > > >> > >> >> for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > strings,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     would we be
>> > > arguing to
>> > >     >> >> divide
>> > >     >> >> >> the
>> > >     >> >> >>>>>     > strings with
>> > >     >> >> >>>>>     >     > > dots
>> > >     >> >> >>>>>     >     > > >> > ('.')
>> > >     >> >> >>>>>     >     > > >> > >> >> as a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > requirement?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     Would we want
>> > them
>> > > to give
>> > >     >> >> us
>> > >     >> >> >> the
>> > >     >> >> >>>>>     > different
>> > >     >> >> >>>>>     >     > name
>> > >     >> >> >>>>>     >     > > >> > segments
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > separately?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     Would we be
>> > > performing any
>> > >     >> >> >>>> actions on
>> > >     >> >> >>>>>     > this key
>> > >     >> >> >>>>>     >     > > other
>> > >     >> >> >>>>>     >     > > >> > than
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > matching?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     Nacho
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     On Fri, Nov 18,
>> > > 2016 at
>> > >     >> 9:30
>> > >     >> >> >> AM,
>> > >     >> >> >>>> Michael
>> > >     >> >> >>>>>     >     > Pearce <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >>
>> Michael.Pearce@ig.com
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > #jay #jun any
>> > > concerns
>> > >     >> on
>> > >     >> >> 1
>> > >     >> >> >>>> and 2
>> > >     >> >> >>>>>     > still?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > @all
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > To get this
>> > > moving
>> > >     >> along a
>> > >     >> >> >> bit
>> > >     >> >> >>>> more
>> > >     >> >> >>>>>     > I'd also
>> > >     >> >> >>>>>     >     > > like
>> > >     >> >> >>>>>     >     > > >> to
>> > >     >> >> >>>>>     >     > > >> > >> ask
>> > >     >> >> >>>>>     >     > > >> > >> >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> get
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > clarity on
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > the below
>> last
>> > > points:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > 3) I believe
>> > > we're all
>> > >     >> >> >> roughly
>> > >     >> >> >>>> happy
>> > >     >> >> >>>>>     > with the
>> > >     >> >> >>>>>     >     > > >> header
>> > >     >> >> >>>>>     >     > > >> > >> value
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > being a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > byte[]?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > 4) I believe
>> > > consensus
>> > >     >> has
>> > >     >> >> >>>> been for an
>> > >     >> >> >>>>>     >     > > namespace
>> > >     >> >> >>>>>     >     > > >> > based
>> > >     >> >> >>>>>     >     > > >> > >> int
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > approach
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > {int,int} for
>> > > the key.
>> > >     >> Any
>> > >     >> >> >>>> objections
>> > >     >> >> >>>>>     > if this
>> > >     >> >> >>>>>     >     > > is
>> > >     >> >> >>>>>     >     > > >> > what
>> > >     >> >> >>>>>     >     > > >> > >> we
>> > >     >> >> >>>>>     >     > > >> > >> >> go
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > with?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > 5) as we have
>> > if
>> > >     >> >> assumption
>> > >     >> >> >> in
>> > >     >> >> >>>> (4)  is
>> > >     >> >> >>>>>     >     > correct,
>> > >     >> >> >>>>>     >     > > >> > >> {int,int}
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> keys.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > Should both
>> > > int's be
>> > >     >> >> int16 or
>> > >     >> >> >>>> int32?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > I'm for them
>> > > being
>> > >     >> int16(2
>> > >     >> >> >>>> bytes) as
>> > >     >> >> >>>>>     > combined
>> > >     >> >> >>>>>     >     > > is
>> > >     >> >> >>>>>     >     > > >> > space
>> > >     >> >> >>>>>     >     > > >> > >> of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > 4bytes as
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > per
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > original and
>> > > gives
>> > >     >> plenty
>> > >     >> >> of
>> > >     >> >> >>>>>     > combinations for
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> foreseeable,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > keeps
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > the overhead
>> > > small.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > Do we see any
>> > > benefit in
>> > >     >> >> >>>> another kip
>> > >     >> >> >>>>>     > call to
>> > >     >> >> >>>>>     >     > > >> discuss
>> > >     >> >> >>>>>     >     > > >> > >> >> these at
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > all?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > Cheers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > Mike
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > >     >> >> >> ______________________________
>> > >     >> >> >>>>>     > __________
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > From: K
>> > Burstev <
>> > >     >> >> >>>> k.burstev@yandex.com>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > Sent: Friday,
>> > > November
>> > >     >> 18,
>> > >     >> >> >> 2016
>> > >     >> >> >>>>>     > 7:07:07 AM
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > To:
>> > >     >> dev@kafka.apache.org
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > Subject: Re:
>> > > [DISCUSS]
>> > >     >> >> KIP-82
>> > >     >> >> >>>> - Add
>> > >     >> >> >>>>>     > Record
>> > >     >> >> >>>>>     >     > > Headers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > For what it
>> is
>> > > worth
>> > >     >> also
>> > >     >> >> i
>> > >     >> >> >>>> agree. As
>> > >     >> >> >>>>>     > a user:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >  1) Yes -
>> > > Headers are
>> > >     >> >> >>>> worthwhile
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >  2) Yes -
>> > > Headers should
>> > >     >> >> be a
>> > >     >> >> >>>> top level
>> > >     >> >> >>>>>     >     > option
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > 14.11.2016,
>> > > 21:15,
>> > >     >> >> "Ignacio
>> > >     >> >> >>>> Solis" <
>> > >     >> >> >>>>>     >     > > >> isolis@igso.net
>> > >     >> >> >>>>>     >     > > >> > >:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > > 1) Yes -
>> > > Headers are
>> > >     >> >> >>>> worthwhile
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > > 2) Yes -
>> > > Headers
>> > >     >> should
>> > >     >> >> be
>> > >     >> >> >> a
>> > >     >> >> >>>> top
>> > >     >> >> >>>>>     > level
>> > >     >> >> >>>>>     >     > option
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > > On Mon, Nov
>> > > 14, 2016
>> > >     >> at
>> > >     >> >> >> 9:16
>> > >     >> >> >>>> AM,
>> > >     >> >> >>>>>     > Michael
>> > >     >> >> >>>>>     >     > > Pearce
>> > >     >> >> >>>>>     >     > > >> <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > Michael.Pearce@ig.com>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Hi Roger,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  The kip
>> > >     >> >> details/examples
>> > >     >> >> >>>> the
>> > >     >> >> >>>>>     > original
>> > >     >> >> >>>>>     >     > > proposal
>> > >     >> >> >>>>>     >     > > >> > for
>> > >     >> >> >>>>>     >     > > >> > >> key
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > spacing
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> ,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > not
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  new
>> > > mentioned as per
>> > >     >> >> >>>> discussion
>> > >     >> >> >>>>>     > namespace
>> > >     >> >> >>>>>     >     > > >> idea.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  We will
>> > need
>> > > to
>> > >     >> update
>> > >     >> >> >> the
>> > >     >> >> >>>> kip,
>> > >     >> >> >>>>>     > when we
>> > >     >> >> >>>>>     >     > get
>> > >     >> >> >>>>>     >     > > >> > >> agreement
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> this
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > is a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > better
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  approach
>> > > (which
>> > >     >> seems
>> > >     >> >> to
>> > >     >> >> >>>> be the
>> > >     >> >> >>>>>     > case if I
>> > >     >> >> >>>>>     >     > > have
>> > >     >> >> >>>>>     >     > > >> > >> >> understood
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > general
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  feeling
>> in
>> > > the
>> > >     >> >> >>>> conversation)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Re the
>> > > variable
>> > >     >> ints,
>> > >     >> >> at
>> > >     >> >> >>>> very
>> > >     >> >> >>>>>     > early stage
>> > >     >> >> >>>>>     >     > > we
>> > >     >> >> >>>>>     >     > > >> did
>> > >     >> >> >>>>>     >     > > >> > >> think
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> about
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > this. I
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > think
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  the added
>> > > complexity
>> > >     >> >> for
>> > >     >> >> >>>> the
>> > >     >> >> >>>>>     > saving isn't
>> > >     >> >> >>>>>     >     > > >> worth
>> > >     >> >> >>>>>     >     > > >> > it.
>> > >     >> >> >>>>>     >     > > >> > >> >> I'd
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > rather
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> go
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > with, if
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  we want
>> to
>> > > reduce
>> > >     >> >> >>>> overheads and
>> > >     >> >> >>>>>     > size
>> > >     >> >> >>>>>     >     > int16
>> > >     >> >> >>>>>     >     > > >> > (2bytes)
>> > >     >> >> >>>>>     >     > > >> > >> >> keys
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> as
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > it
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > keeps it
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  simple.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  On the
>> note
>> > > of no
>> > >     >> >> >> headers,
>> > >     >> >> >>>> there
>> > >     >> >> >>>>>     > is as
>> > >     >> >> >>>>>     >     > per
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > kip
>> > >     >> >> >>>>>     >     > > >> > >> as
>> > >     >> >> >>>>>     >     > > >> > >> >> we
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > use an
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > attribute
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  bit to
>> > > denote if
>> > >     >> >> headers
>> > >     >> >> >>>> are
>> > >     >> >> >>>>>     > present or
>> > >     >> >> >>>>>     >     > > not as
>> > >     >> >> >>>>>     >     > > >> > such
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > provides a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > zero
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  overhead
>> > > currently
>> > >     >> if
>> > >     >> >> >>>> headers are
>> > >     >> >> >>>>>     > not
>> > >     >> >> >>>>>     >     > used.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  I think
>> as
>> > > radai
>> > >     >> >> mentions
>> > >     >> >> >>>> would be
>> > >     >> >> >>>>>     > good
>> > >     >> >> >>>>>     >     > > first
>> > >     >> >> >>>>>     >     > > >> > if we
>> > >     >> >> >>>>>     >     > > >> > >> >> can
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> get
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > clarity if
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > do
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  we now
>> have
>> > > general
>> > >     >> >> >>>> consensus that
>> > >     >> >> >>>>>     > (1)
>> > >     >> >> >>>>>     >     > > headers
>> > >     >> >> >>>>>     >     > > >> > are
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > worthwhile
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > useful,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  and (2)
>> we
>> > > want it
>> > >     >> as
>> > >     >> >> a
>> > >     >> >> >>>> top level
>> > >     >> >> >>>>>     > entity.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Just to
>> > > state the
>> > >     >> >> >> obvious i
>> > >     >> >> >>>>>     > believe (1)
>> > >     >> >> >>>>>     >     > > >> headers
>> > >     >> >> >>>>>     >     > > >> > are
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > worthwhile
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > and (2)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  agree as
>> a
>> > > top level
>> > >     >> >> >>>> entity.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Cheers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Mike
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>> ______________________________
>> > >     >> >> >>>>>     > __________
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  From:
>> Roger
>> > > Hoover <
>> > >     >> >> >>>>>     >     > roger.hoover@gmail.com
>> > >     >> >> >>>>>     >     > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Sent:
>> > > Wednesday,
>> > >     >> >> November
>> > >     >> >> >>>> 9, 2016
>> > >     >> >> >>>>>     > 9:10:47
>> > >     >> >> >>>>>     >     > > PM
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  To:
>> > >     >> >> dev@kafka.apache.org
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Subject:
>> > Re:
>> > >     >> [DISCUSS]
>> > >     >> >> >>>> KIP-82 - Add
>> > >     >> >> >>>>>     >     > Record
>> > >     >> >> >>>>>     >     > > >> > Headers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Sorry for
>> > > going a
>> > >     >> >> little
>> > >     >> >> >>>> in the
>> > >     >> >> >>>>>     > weeds but
>> > >     >> >> >>>>>     >     > > >> thanks
>> > >     >> >> >>>>>     >     > > >> > >> for
>> > >     >> >> >>>>>     >     > > >> > >> >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> replies
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > regarding
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  varint.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Agreed
>> that
>> > > a prefix
>> > >     >> >> and
>> > >     >> >> >>>> {int,
>> > >     >> >> >>>>>     > int} can
>> > >     >> >> >>>>>     >     > be
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > >> same.
>> > >     >> >> >>>>>     >     > > >> > >> >> It
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > doesn't
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > look
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > like
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  that's
>> what
>> > > the KIP
>> > >     >> is
>> > >     >> >> >>>> saying the
>> > >     >> >> >>>>>     > "Open"
>> > >     >> >> >>>>>     >     > > >> > section.
>> > >     >> >> >>>>>     >     > > >> > >> The
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > example
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > shows
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  2100001
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  for New
>> > > Relic and
>> > >     >> >> 210002
>> > >     >> >> >>>> for App
>> > >     >> >> >>>>>     > Dynamics
>> > >     >> >> >>>>>     >     > > >> > implying
>> > >     >> >> >>>>>     >     > > >> > >> >> that
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > New
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > Relic
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > organization
>> > > will
>> > >     >> have
>> > >     >> >> >>>> only a
>> > >     >> >> >>>>>     > single
>> > >     >> >> >>>>>     >     > > header id
>> > >     >> >> >>>>>     >     > > >> > to
>> > >     >> >> >>>>>     >     > > >> > >> work
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > with. Or
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > is
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > 2100001
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  a prefix?
>> > > The main
>> > >     >> >> point
>> > >     >> >> >>>> of a
>> > >     >> >> >>>>>     > namespace
>> > >     >> >> >>>>>     >     > or
>> > >     >> >> >>>>>     >     > > >> > prefix
>> > >     >> >> >>>>>     >     > > >> > >> is
>> > >     >> >> >>>>>     >     > > >> > >> >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > reduce
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  overhead
>> of
>> > > config
>> > >     >> >> >> mapping
>> > >     >> >> >>>> or
>> > >     >> >> >>>>>     >     > registration
>> > >     >> >> >>>>>     >     > > >> > >> depending
>> > >     >> >> >>>>>     >     > > >> > >> >> on
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> how
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > > namespaces/prefixes
>> > >     >> >> are
>> > >     >> >> >>>> managed.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Would
>> love
>> > > to hear
>> > >     >> >> more
>> > >     >> >> >>>> feedback
>> > >     >> >> >>>>>     > on the
>> > >     >> >> >>>>>     >     > > >> > >> higher-level
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > questions
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > though...
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Cheers,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Roger
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  On Wed,
>> Nov
>> > > 9, 2016
>> > >     >> at
>> > >     >> >> >>>> 11:38 AM,
>> > >     >> >> >>>>>     > radai <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > > radai.rosenblatt@gmail.com>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > I think
>> > > this
>> > >     >> >> discussion
>> > >     >> >> >>>> is
>> > >     >> >> >>>>>     > getting a
>> > >     >> >> >>>>>     >     > bit
>> > >     >> >> >>>>>     >     > > >> into
>> > >     >> >> >>>>>     >     > > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> weeds on
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > technical
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > > implementation
>> > >     >> >> details.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > I'd
>> liek
>> > > to step
>> > >     >> >> back a
>> > >     >> >> >>>> minute
>> > >     >> >> >>>>>     > and try
>> > >     >> >> >>>>>     >     > > and
>> > >     >> >> >>>>>     >     > > >> > >> establish
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > where we
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > are in
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > larger
>> > > picture:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > (re-wording
>> > >     >> nacho's
>> > >     >> >> >> last
>> > >     >> >> >>>>>     > paragraph)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > 1. are
>> we
>> > > all in
>> > >     >> >> >>>> agreement that
>> > >     >> >> >>>>>     > headers
>> > >     >> >> >>>>>     >     > > are
>> > >     >> >> >>>>>     >     > > >> a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> worthwhile
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > useful
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> addition
>> > > to have?
>> > >     >> >> this
>> > >     >> >> >>>> was
>> > >     >> >> >>>>>     > contested
>> > >     >> >> >>>>>     >     > > early
>> > >     >> >> >>>>>     >     > > >> on
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > 2. are
>> we
>> > > all in
>> > >     >> >> >>>> agreement on
>> > >     >> >> >>>>>     > headers
>> > >     >> >> >>>>>     >     > as
>> > >     >> >> >>>>>     >     > > top
>> > >     >> >> >>>>>     >     > > >> > >> level
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> entity
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > vs
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > headers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > > squirreled-away in
>> > >     >> >> V?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > if
>> there
>> > > are still
>> > >     >> >> >>>> concerns
>> > >     >> >> >>>>>     > around
>> > >     >> >> >>>>>     >     > these
>> > >     >> >> >>>>>     >     > > #2
>> > >     >> >> >>>>>     >     > > >> > >> points
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> (#jay?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > #jun?)?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > (and
>> now
>> > > back to
>> > >     >> our
>> > >     >> >> >>>> normal
>> > >     >> >> >>>>>     > programming
>> > >     >> >> >>>>>     >     > > ...)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > varints
>> > > are nice.
>> > >     >> >> >> having
>> > >     >> >> >>>> said
>> > >     >> >> >>>>>     > that, its
>> > >     >> >> >>>>>     >     > > >> adding
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> complexity
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> (see
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > >     >> >> >>>> https://github.com/addthis/
>> > >     >> >> >>>>>     >     > > >> > >> >> stream-lib/blob/master/src/
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > >     >> >> >>>> main/java/com/clearspring/
>> > >     >> >> >>>>>     >     > > >> > >> >> analytics/util/Varint.java
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > as 1st
>> > > google
>> > >     >> >> result)
>> > >     >> >> >>>> and would
>> > >     >> >> >>>>>     > require
>> > >     >> >> >>>>>     >     > > >> anyone
>> > >     >> >> >>>>>     >     > > >> > >> >> writing
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > other
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > clients
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > (C?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > Python?
>> > > Go? Bash?
>> > >     >> >> ;-) )
>> > >     >> >> >>>> to
>> > >     >> >> >>>>>     >     > get/implement
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > >> same,
>> > >     >> >> >>>>>     >     > > >> > >> >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > relatively
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > little
>> > > gain (int
>> > >     >> vs
>> > >     >> >> >>>> string is
>> > >     >> >> >>>>>     > order of
>> > >     >> >> >>>>>     >     > > >> > magnitude,
>> > >     >> >> >>>>>     >     > > >> > >> >> this
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > isnt).
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > int
>> > > namespacing vs
>> > >     >> >> >> {int,
>> > >     >> >> >>>> int}
>> > >     >> >> >>>>>     >     > namespacing
>> > >     >> >> >>>>>     >     > > >> are
>> > >     >> >> >>>>>     >     > > >> > >> >> basically
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > same
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > thing -
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > youre
>> > just
>> > >     >> >> namespacing
>> > >     >> >> >>>> an int64
>> > >     >> >> >>>>>     > and
>> > >     >> >> >>>>>     >     > > giving
>> > >     >> >> >>>>>     >     > > >> > people
>> > >     >> >> >>>>>     >     > > >> > >> >> while
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > 2^32
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > ranges
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > at a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > time.
>> the
>> > > part i
>> > >     >> >> like
>> > >     >> >> >>>> about this
>> > >     >> >> >>>>>     > is
>> > >     >> >> >>>>>     >     > > letting
>> > >     >> >> >>>>>     >     > > >> > >> people
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> have a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> large
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > swath of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > numbers
>> > > with one
>> > >     >> >> >>>> registration so
>> > >     >> >> >>>>>     > they
>> > >     >> >> >>>>>     >     > > dont
>> > >     >> >> >>>>>     >     > > >> > have
>> > >     >> >> >>>>>     >     > > >> > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> come
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > back
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > every
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > single
>> > >     >> plugin/header
>> > >     >> >> >>>> they want to
>> > >     >> >> >>>>>     >     > > "reserve".
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > On Wed,
>> > > Nov 9,
>> > >     >> 2016
>> > >     >> >> at
>> > >     >> >> >>>> 11:01 AM,
>> > >     >> >> >>>>>     > Roger
>> > >     >> >> >>>>>     >     > > >> Hoover
>> > >     >> >> >>>>>     >     > > >> > <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > > roger.hoover@gmail.com>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > Since
>> > > some of
>> > >     >> the
>> > >     >> >> >>>> debate has
>> > >     >> >> >>>>>     > been
>> > >     >> >> >>>>>     >     > about
>> > >     >> >> >>>>>     >     > > >> > >> overhead +
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > performance, I'm
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > > wondering if we
>> > >     >> >> have
>> > >     >> >> >>>>>     > considered a
>> > >     >> >> >>>>>     >     > > varint
>> > >     >> >> >>>>>     >     > > >> > >> encoding
>> > >     >> >> >>>>>     >     > > >> > >> >> (
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > >     >> >> >>>> https://developers.google.com/
>> > >     >> >> >>>>>     >     > > >> > >> >> protocol-buffers/docs/
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > encoding#varints)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > the
>> > > header
>> > >     >> length
>> > >     >> >> >>>> field (int32
>> > >     >> >> >>>>>     > in the
>> > >     >> >> >>>>>     >     > > >> > proposal)
>> > >     >> >> >>>>>     >     > > >> > >> >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > header
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > ids? If
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > you
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > don't
>> > use
>> > >     >> headers,
>> > >     >> >> >> the
>> > >     >> >> >>>>>     > overhead would
>> > >     >> >> >>>>>     >     > > be a
>> > >     >> >> >>>>>     >     > > >> > >> single
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> byte
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > for each
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > header
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > id <
>> > 128
>> > > would
>> > >     >> >> also
>> > >     >> >> >>>> need only a
>> > >     >> >> >>>>>     >     > single
>> > >     >> >> >>>>>     >     > > >> byte?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > On
>> Wed,
>> > > Nov 9,
>> > >     >> >> 2016
>> > >     >> >> >> at
>> > >     >> >> >>>> 6:43 AM,
>> > >     >> >> >>>>>     >     > radai <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >
>> > > radai.rosenblatt@gmail.com>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > > @magnus - and
>> > >     >> >> very
>> > >     >> >> >>>> dangerous
>> > >     >> >> >>>>>     > (youre
>> > >     >> >> >>>>>     >     > > >> > >> essentially
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > downloading and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> executing
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > > arbitrary code
>> > >     >> >> off
>> > >     >> >> >>>> the
>> > >     >> >> >>>>>     > internet on
>> > >     >> >> >>>>>     >     > > your
>> > >     >> >> >>>>>     >     > > >> > >> servers
>> > >     >> >> >>>>>     >     > > >> > >> >> ...
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > bad
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > idea
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > without
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > > sandbox, even
>> > >     >> >> with)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > as
>> > for
>> > > it
>> > >     >> being
>> > >     >> >> a
>> > >     >> >> >>>> purely
>> > >     >> >> >>>>>     >     > > administrative
>> > >     >> >> >>>>>     >     > > >> > task
>> > >     >> >> >>>>>     >     > > >> > >> - i
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> disagree.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > i
>> > wish
>> > > it
>> > >     >> would,
>> > >     >> >> >>>> really,
>> > >     >> >> >>>>>     > because
>> > >     >> >> >>>>>     >     > > then my
>> > >     >> >> >>>>>     >     > > >> > >> earlier
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > point on
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > complexity
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > the
>> > > remapping
>> > >     >> >> >>>> process would
>> > >     >> >> >>>>>     > be
>> > >     >> >> >>>>>     >     > > invalid,
>> > >     >> >> >>>>>     >     > > >> > but
>> > >     >> >> >>>>>     >     > > >> > >> at
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > linkedin,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > example,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > we
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> (the
>> > > team im
>> > >     >> in)
>> > >     >> >> >> run
>> > >     >> >> >>>> kafka
>> > >     >> >> >>>>>     > as a
>> > >     >> >> >>>>>     >     > > service.
>> > >     >> >> >>>>>     >     > > >> > we
>> > >     >> >> >>>>>     >     > > >> > >> dont
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > really
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > know
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > what our
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > users
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > > (developing
>> > >     >> >> >>>> applications
>> > >     >> >> >>>>>     > that use
>> > >     >> >> >>>>>     >     > > kafka)
>> > >     >> >> >>>>>     >     > > >> > are
>> > >     >> >> >>>>>     >     > > >> > >> up
>> > >     >> >> >>>>>     >     > > >> > >> >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> at
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > any
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > given
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  moment.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > it
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > is
>> > very
>> > >     >> possible
>> > >     >> >> >>>> (given the
>> > >     >> >> >>>>>     >     > > existance of
>> > >     >> >> >>>>>     >     > > >> > >> headers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> and a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > corresponding
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> plugin
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > > ecosystem) for
>> > >     >> >> some
>> > >     >> >> >>>>>     > application to
>> > >     >> >> >>>>>     >     > > >> "equip"
>> > >     >> >> >>>>>     >     > > >> > >> their
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> producers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> consumers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> with
>> > > the
>> > >     >> >> required
>> > >     >> >> >>>> plugin
>> > >     >> >> >>>>>     > without us
>> > >     >> >> >>>>>     >     > > >> > knowing.
>> > >     >> >> >>>>>     >     > > >> > >> i
>> > >     >> >> >>>>>     >     > > >> > >> >> dont
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > mean
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > to imply
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  thats
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> bad,
>> > i
>> > > just
>> > >     >> >> want to
>> > >     >> >> >>>> make the
>> > >     >> >> >>>>>     > point
>> > >     >> >> >>>>>     >     > > that
>> > >     >> >> >>>>>     >     > > >> > its
>> > >     >> >> >>>>>     >     > > >> > >> not
>> > >     >> >> >>>>>     >     > > >> > >> >> as
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > simple
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > keeping it
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  in
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> sync
>> > > across a
>> > >     >> >> >>>> large-enough
>> > >     >> >> >>>>>     >     > > organization.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > On
>> > > Wed, Nov 9,
>> > >     >> >> 2016
>> > >     >> >> >>>> at 6:17
>> > >     >> >> >>>>>     > AM,
>> > >     >> >> >>>>>     >     > > Magnus
>> > >     >> >> >>>>>     >     > > >> > >> Edenhill
>> > >     >> >> >>>>>     >     > > >> > >> >> <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > > magnus@edenhill.se>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > I
>> > > think
>> > >     >> there
>> > >     >> >> is
>> > >     >> >> >> a
>> > >     >> >> >>>> piece
>> > >     >> >> >>>>>     > missing
>> > >     >> >> >>>>>     >     > in
>> > >     >> >> >>>>>     >     > > >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> Strings
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > discussion,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > where
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> pro-Stringers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > reason that
>> > >     >> by
>> > >     >> >> >>>> providing
>> > >     >> >> >>>>>     > unique
>> > >     >> >> >>>>>     >     > > string
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> identifiers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > each
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > header
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > everything
>> > >     >> >> will
>> > >     >> >> >>>> just
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > magically
>> > >     >> work
>> > >     >> >> >> for
>> > >     >> >> >>>> all
>> > >     >> >> >>>>>     > parts of
>> > >     >> >> >>>>>     >     > the
>> > >     >> >> >>>>>     >     > > >> > stream
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> pipeline.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> But
>> > > the
>> > >     >> >> strings
>> > >     >> >> >>>> dont mean
>> > >     >> >> >>>>>     >     > anything
>> > >     >> >> >>>>>     >     > > by
>> > >     >> >> >>>>>     >     > > >> > >> >> themselves,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > while we
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  could
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > probably
>> > >     >> >> envision
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > some
>> > > auto
>> > >     >> >> plugin
>> > >     >> >> >>>> loader
>> > >     >> >> >>>>>     > that
>> > >     >> >> >>>>>     >     > > >> downloads,
>> > >     >> >> >>>>>     >     > > >> > >> >> compiles,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > links
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > runs
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> plugins
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > on-demand
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> as
>> > > soon as
>> > >     >> >> >> they're
>> > >     >> >> >>>> seen by
>> > >     >> >> >>>>>     > a
>> > >     >> >> >>>>>     >     > > >> consumer, I
>> > >     >> >> >>>>>     >     > > >> > >> dont
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> really
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> see
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > use-case
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > something
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> so
>> > > dynamic
>> > >     >> >> (and
>> > >     >> >> >>>> fragile) in
>> > >     >> >> >>>>>     >     > > practice.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> In
>> > > the real
>> > >     >> >> world
>> > >     >> >> >>>> an
>> > >     >> >> >>>>>     > application
>> > >     >> >> >>>>>     >     > > will
>> > >     >> >> >>>>>     >     > > >> be
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> configured
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> with
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > a set
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> plugins
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> to
>> > > either
>> > >     >> add
>> > >     >> >> >>>> (producer)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> or
>> > > read
>> > >     >> >> >> (consumer)
>> > >     >> >> >>>> headers.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > This
>> > > is an
>> > >     >> >> >>>> administrative
>> > >     >> >> >>>>>     > task
>> > >     >> >> >>>>>     >     > > based
>> > >     >> >> >>>>>     >     > > >> on
>> > >     >> >> >>>>>     >     > > >> > >> what
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > features a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > client
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> needs/provides
>> > >     >> >> >> and
>> > >     >> >> >>>> results
>> > >     >> >> >>>>>     > in
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > some
>> > > sort of
>> > >     >> >> >>>> configuration
>> > >     >> >> >>>>>     > to
>> > >     >> >> >>>>>     >     > > enable
>> > >     >> >> >>>>>     >     > > >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> configure
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > desired
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> plugins.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > Since this
>> > >     >> >> needs
>> > >     >> >> >>>> to be kept
>> > >     >> >> >>>>>     >     > > somewhat
>> > >     >> >> >>>>>     >     > > >> in
>> > >     >> >> >>>>>     >     > > >> > >> sync
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> across
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > an
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > organisation
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > (there
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> is
>> > > no point
>> > >     >> in
>> > >     >> >> >>>> having
>> > >     >> >> >>>>>     > producers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> add
>> > > headers
>> > >     >> no
>> > >     >> >> >>>> consumers
>> > >     >> >> >>>>>     > will
>> > >     >> >> >>>>>     >     > read,
>> > >     >> >> >>>>>     >     > > >> and
>> > >     >> >> >>>>>     >     > > >> > >> vice
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> versa),
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > added
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > > complexity
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> of
>> > > assigning
>> > >     >> >> an
>> > >     >> >> >> id
>> > >     >> >> >>>>>     > namespace
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> for
>> > > each
>> > >     >> >> plugin
>> > >     >> >> >> as
>> > >     >> >> >>>> it is
>> > >     >> >> >>>>>     > being
>> > >     >> >> >>>>>     >     > > >> > configured
>> > >     >> >> >>>>>     >     > > >> > >> >> should
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> be
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > tolerable.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > /Magnus
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > 2016-11-09
>> > >     >> >> 13:06
>> > >     >> >> >>>> GMT+01:00
>> > >     >> >> >>>>>     >     > Michael
>> > >     >> >> >>>>>     >     > > >> > Pearce <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> > > Michael.Pearce@ig.com>:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > Just
>> > >     >> >> >>>> following/catching
>> > >     >> >> >>>>>     > up on
>> > >     >> >> >>>>>     >     > > what
>> > >     >> >> >>>>>     >     > > >> > seems
>> > >     >> >> >>>>>     >     > > >> > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> be
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> an
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > active
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > night :)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > @Radai
>> > >     >> >> sorry if
>> > >     >> >> >>>> it may
>> > >     >> >> >>>>>     > seem
>> > >     >> >> >>>>>     >     > > obvious
>> > >     >> >> >>>>>     >     > > >> > but
>> > >     >> >> >>>>>     >     > > >> > >> what
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> does
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > MD
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > stand
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > for?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > My
>> > > take on
>> > >     >> >> >>>> String vs Int:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > I
>> > > will
>> > >     >> state
>> > >     >> >> >>>> first I am
>> > >     >> >> >>>>>     > pro Int
>> > >     >> >> >>>>>     >     > > (16
>> > >     >> >> >>>>>     >     > > >> or
>> > >     >> >> >>>>>     >     > > >> > >> 32).
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > I
>> > > do
>> > >     >> though
>> > >     >> >> >>>> playing
>> > >     >> >> >>>>>     > devils
>> > >     >> >> >>>>>     >     > > advocate
>> > >     >> >> >>>>>     >     > > >> > see a
>> > >     >> >> >>>>>     >     > > >> > >> >> big
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> plus
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > with the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> argument
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > String
>> > >     >> keys,
>> > >     >> >> >>>> this is
>> > >     >> >> >>>>>     > around
>> > >     >> >> >>>>>     >     > > >> > integrating
>> > >     >> >> >>>>>     >     > > >> > >> >> into an
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > existing
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > eco-system.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > As
>> > > many
>> > >     >> >> other
>> > >     >> >> >>>> systems use
>> > >     >> >> >>>>>     >     > String
>> > >     >> >> >>>>>     >     > > >> based
>> > >     >> >> >>>>>     >     > > >> > >> >> headers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> (Flume,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > JMS)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > it
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > makes
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > it
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > much
>> > >     >> easier
>> > >     >> >> for
>> > >     >> >> >>>> these to
>> > >     >> >> >>>>>     > be
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > incorporated/integrated
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > into.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > How with
>> > >     >> Int
>> > >     >> >> >>>> based
>> > >     >> >> >>>>>     > headers
>> > >     >> >> >>>>>     >     > could
>> > >     >> >> >>>>>     >     > > we
>> > >     >> >> >>>>>     >     > > >> > >> provide
>> > >     >> >> >>>>>     >     > > >> > >> >> a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > way/guidence to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  make
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > this
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> integration
>> > >     >> >> >>>> simple /
>> > >     >> >> >>>>>     > easy with
>> > >     >> >> >>>>>     >     > > >> > transition
>> > >     >> >> >>>>>     >     > > >> > >> >> flows
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > over
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > kafka?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > *
>> > > tough
>> > >     >> luck
>> > >     >> >> >>>> buddy
>> > >     >> >> >>>>>     > you're on
>> > >     >> >> >>>>>     >     > your
>> > >     >> >> >>>>>     >     > > >> own
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > *
>> > > simply
>> > >     >> >> hash
>> > >     >> >> >>>> the string
>> > >     >> >> >>>>>     > into
>> > >     >> >> >>>>>     >     > int
>> > >     >> >> >>>>>     >     > > >> code
>> > >     >> >> >>>>>     >     > > >> > >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> hope
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> no
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > collisions
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > (how
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > convert
>> > >     >> back
>> > >     >> >> >>>> though?)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > *
>> > > http2
>> > >     >> >> style
>> > >     >> >> >> as
>> > >     >> >> >>>>>     > mentioned by
>> > >     >> >> >>>>>     >     > > nacho.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > cheers,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > Mike
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     > ______________________________
>> > >     >> >> >>>>>     >     > > >> > __________
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > From:
>> > >     >> radai
>> > >     >> >> <
>> > >     >> >> >>>>>     >     > > >> > radai.rosenblatt@gmail.com>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > Sent:
>> > >     >> >> >> Wednesday,
>> > >     >> >> >>>>>     > November 9,
>> > >     >> >> >>>>>     >     > 2016
>> > >     >> >> >>>>>     >     > > >> > 8:12 AM
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > To:
>> > >     >> >> >>>> dev@kafka.apache.org
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > Subject:
>> > >     >> Re:
>> > >     >> >> >>>> [DISCUSS]
>> > >     >> >> >>>>>     > KIP-82 -
>> > >     >> >> >>>>>     >     > > Add
>> > >     >> >> >>>>>     >     > > >> > >> Record
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> Headers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > thinking
>> > >     >> >> about
>> > >     >> >> >>>> it some
>> > >     >> >> >>>>>     > more,
>> > >     >> >> >>>>>     >     > the
>> > >     >> >> >>>>>     >     > > >> best
>> > >     >> >> >>>>>     >     > > >> > >> way to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > transmit
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > header
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > remapping
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > data to
>> > >     >> >> >>>> consumers would
>> > >     >> >> >>>>>     > be to
>> > >     >> >> >>>>>     >     > > put it
>> > >     >> >> >>>>>     >     > > >> > in
>> > >     >> >> >>>>>     >     > > >> > >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> MD
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> response
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > payload,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  so
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> maybe
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > it
>> > > should
>> > >     >> be
>> > >     >> >> >>>> discussed
>> > >     >> >> >>>>>     > now.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > On
>> > > Wed,
>> > >     >> Nov
>> > >     >> >> 9,
>> > >     >> >> >>>> 2016 at
>> > >     >> >> >>>>>     > 12:09
>> > >     >> >> >>>>>     >     > AM,
>> > >     >> >> >>>>>     >     > > >> > radai <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > >     >> >> >> radai.rosenblatt@gmail.com
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > > im not
>> > >     >> >> >> opposed
>> > >     >> >> >>>> to the
>> > >     >> >> >>>>>     > idea of
>> > >     >> >> >>>>>     >     > > >> > namespace
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> mapping.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> all
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > im
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > saying
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  is
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> that
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > its
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > > not part
>> > >     >> >> of
>> > >     >> >> >>>> the "mvp"
>> > >     >> >> >>>>>     > and,
>> > >     >> >> >>>>>     >     > > since
>> > >     >> >> >>>>>     >     > > >> it
>> > >     >> >> >>>>>     >     > > >> > >> >> requires
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> no
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> wire
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > format
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > change,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > can
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > > always
>> > >     >> be
>> > >     >> >> >>>> added later.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > > also,
>> > >     >> its
>> > >     >> >> not
>> > >     >> >> >>>> as
>> > >     >> >> >>>>>     > simple as
>> > >     >> >> >>>>>     >     > just
>> > >     >> >> >>>>>     >     > > >> > >> >> configuring
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> MM
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> do
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > transform:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > lets
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > > say i've
>> > >     >> >> >>>> implemented
>> > >     >> >> >>>>>     > large
>> > >     >> >> >>>>>     >     > > message
>> > >     >> >> >>>>>     >     > > >> > >> >> support as
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > {666,1} and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > on
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  some
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > mirror
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > > target
>> > >     >> >> >> cluster
>> > >     >> >> >>>> its been
>> > >     >> >> >>>>>     >     > > remapped
>> > >     >> >> >>>>>     >     > > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> {999,1}.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > consumer
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  plugin
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > code
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > would
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > > also
>> > >     >> need
>> > >     >> >> to
>> > >     >> >> >>>> be told
>> > >     >> >> >>>>>     > to look
>> > >     >> >> >>>>>     >     > > for
>> > >     >> >> >>>>>     >     > > >> the
>> > >     >> >> >>>>>     >     > > >> > >> large
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > message
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > "part X
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  Y"
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > header
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > > under
>> > >     >> >> >> {999,1}.
>> > >     >> >> >>>> doable,
>> > >     >> >> >>>>>     > but
>> > >     >> >> >>>>>     >     > > tricky.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > > On Tue,
>> > >     >> >> Nov
>> > >     >> >> >> 8,
>> > >     >> >> >>>> 2016 at
>> > >     >> >> >>>>>     > 10:29
>> > >     >> >> >>>>>     >     > > PM,
>> > >     >> >> >>>>>     >     > > >> > Gwen
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> Shapira <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>
>> > > gwen@confluent.io
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > While
>> > >     >> you
>> > >     >> >> >> can
>> > >     >> >> >>>> do
>> > >     >> >> >>>>>     > whatever
>> > >     >> >> >>>>>     >     > you
>> > >     >> >> >>>>>     >     > > >> want
>> > >     >> >> >>>>>     >     > > >> > >> with a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> namespace
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > your
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > code,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > what
>> > >     >> I'd
>> > >     >> >> >>>> expect is
>> > >     >> >> >>>>>     > for each
>> > >     >> >> >>>>>     >     > > app
>> > >     >> >> >>>>>     >     > > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> namespaces
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     >
>> configurable...
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > So if I
>> > >     >> >> >>>> accidentally
>> > >     >> >> >>>>>     > used
>> > >     >> >> >>>>>     >     > 666
>> > >     >> >> >>>>>     >     > > for
>> > >     >> >> >>>>>     >     > > >> > my
>> > >     >> >> >>>>>     >     > > >> > >> HR
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> department,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > still
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > want
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > run
>> > >     >> >> >> RadaiApp,
>> > >     >> >> >>>> I can
>> > >     >> >> >>>>>     > config
>> > >     >> >> >>>>>     >     > > >> > >> "namespace=42"
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> for
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > RadaiApp and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > > everything
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > will
>> > >     >> look
>> > >     >> >> >>>> normal.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > This
>> > >     >> >> means
>> > >     >> >> >>>> you only
>> > >     >> >> >>>>>     > need to
>> > >     >> >> >>>>>     >     > > sync
>> > >     >> >> >>>>>     >     > > >> > usage
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> inside
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > your
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > own
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > > organization.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > Still
>> > >     >> >> hard,
>> > >     >> >> >>>> but
>> > >     >> >> >>>>>     > somewhat
>> > >     >> >> >>>>>     >     > > easier
>> > >     >> >> >>>>>     >     > > >> > than
>> > >     >> >> >>>>>     >     > > >> > >> >> syncing
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > with
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > entire
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> world.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > On Tue,
>> > >     >> >> Nov
>> > >     >> >> >>>> 8, 2016
>> > >     >> >> >>>>>     > at 10:07
>> > >     >> >> >>>>>     >     > > PM,
>> > >     >> >> >>>>>     >     > > >> > >> radai <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > >     >> >> >>>> radai.rosenblatt@gmail.com>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > > and
>> > >     >> we
>> > >     >> >> can
>> > >     >> >> >>>> start
>> > >     >> >> >>>>>     > with
>> > >     >> >> >>>>>     >     > > >> {namespace,
>> > >     >> >> >>>>>     >     > > >> > >> id}
>> > >     >> >> >>>>>     >     > > >> > >> >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> no
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > re-mapping
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > support
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > always
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > > add
>> > >     >> it
>> > >     >> >> >>>> later on
>> > >     >> >> >>>>>     > if/when
>> > >     >> >> >>>>>     >     > > >> > collisions
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> actually
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > happen (i
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > dont
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > think
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > they'd
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > be
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > > a
>> > >     >> >> >> problem).
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > > every
>> > >     >> >> >>>> interested
>> > >     >> >> >>>>>     > party (so
>> > >     >> >> >>>>>     >     > > orgs
>> > >     >> >> >>>>>     >     > > >> > or
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > individuals)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > could
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > then
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > > register
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >
>> > >     >> prefix
>> > >     >> >> (0
>> > >     >> >> >> =
>> > >     >> >> >>>>>     > reserved, 1 =
>> > >     >> >> >>>>>     >     > > >> > confluent
>> > >     >> >> >>>>>     >     > > >> > >> ...
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> 666
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > = me
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > :-) )
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  do
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > whatever
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > with
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > > the
>> > >     >> >> 2nd ID
>> > >     >> >> >>>> - so once
>> > >     >> >> >>>>>     >     > > linkedin
>> > >     >> >> >>>>>     >     > > >> > >> >> registers,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> say
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > 3,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > then
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  linkedin
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > devs
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> are
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > free
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > > to
>> > >     >> use
>> > >     >> >> {3,
>> > >     >> >> >>>> *} with a
>> > >     >> >> >>>>>     >     > > reasonable
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> expectation
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > collide
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  with
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > anything
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > > else.
>> > >     >> >> >>>> further
>> > >     >> >> >>>>>     > partitioning
>> > >     >> >> >>>>>     >     > > of
>> > >     >> >> >>>>>     >     > > >> > that *
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> becomes
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > linkedin's
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> problem,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > but
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >
>> > >     >> >> "upstream
>> > >     >> >> >>>>>     > registration"
>> > >     >> >> >>>>>     >     > of a
>> > >     >> >> >>>>>     >     > > >> > >> namespace
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> only
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > has
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > happen
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > once.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > > On
>> > >     >> Tue,
>> > >     >> >> >> Nov
>> > >     >> >> >>>> 8, 2016
>> > >     >> >> >>>>>     > at
>> > >     >> >> >>>>>     >     > 9:03
>> > >     >> >> >>>>>     >     > > PM,
>> > >     >> >> >>>>>     >     > > >> > >> James
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> Cheng <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > >     >> >> wushujames@gmail.com
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> > On
>> > >     >> >> Nov
>> > >     >> >> >>>> 8, 2016,
>> > >     >> >> >>>>>     > at 5:54
>> > >     >> >> >>>>>     >     > > PM,
>> > >     >> >> >>>>>     >     > > >> > Gwen
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> Shapira <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  >
>> > > gwen@confluent.io
>> > >     >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> >> Thank
>> > >     >> >> >>>> you so
>> > >     >> >> >>>>>     > much for
>> > >     >> >> >>>>>     >     > > this
>> > >     >> >> >>>>>     >     > > >> > clear
>> > >     >> >> >>>>>     >     > > >> > >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> fair
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > summary of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > arguments.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> I'm
>> > >     >> >> in
>> > >     >> >> >>>> favor of
>> > >     >> >> >>>>>     > ints.
>> > >     >> >> >>>>>     >     > > Not a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> deal-breaker,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > but
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > in
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > favor.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> Even
>> > >     >> >> >>>> more in
>> > >     >> >> >>>>>     > favor of
>> > >     >> >> >>>>>     >     > > >> Magnus's
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > decentralized
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > suggestion
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > with
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > Roger's
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> >> tweak:
>> > >     >> >> >>>> add a
>> > >     >> >> >>>>>     > namespace
>> > >     >> >> >>>>>     >     > > for
>> > >     >> >> >>>>>     >     > > >> > >> headers.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> This
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > will
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > allow
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > each
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > app
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > just
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> use
>> > >     >> >> >>>> whatever IDs
>> > >     >> >> >>>>>     > it
>> > >     >> >> >>>>>     >     > wants
>> > >     >> >> >>>>>     >     > > >> > >> >> internally,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> then
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > let
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > admin
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > >     >> deploying
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> the
>> > >     >> >> app
>> > >     >> >> >>>> figure
>> > >     >> >> >>>>>     > out an
>> > >     >> >> >>>>>     >     > > >> > available
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> namespace
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > ID
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > for the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > app
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> live
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > in.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> > So
>> > >     >> >> >>>>>     >     > > >> > io.confluent.schema-registry
>> > >     >> >> >>>>>     >     > > >> > >> can
>> > >     >> >> >>>>>     >     > > >> > >> >> be
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > namespace
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > 0x01 on
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  my
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > >     >> >> deployment
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> and
>> > >     >> >> >> 0x57
>> > >     >> >> >>>> on
>> > >     >> >> >>>>>     > yours, and
>> > >     >> >> >>>>>     >     > > the
>> > >     >> >> >>>>>     >     > > >> > poor
>> > >     >> >> >>>>>     >     > > >> > >> guys
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > developing the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > app
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > don't
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > need
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > to
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> >> worry
>> > >     >> >> >>>> about that.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> Gwen,
>> > >     >> >> if
>> > >     >> >> >> I
>> > >     >> >> >>>>>     > understand
>> > >     >> >> >>>>>     >     > your
>> > >     >> >> >>>>>     >     > > >> > example
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> right, an
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > application
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > >
>> > deployer
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > might
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> decide to
>> > >     >> >> >>>> use 0x01
>> > >     >> >> >>>>>     > in one
>> > >     >> >> >>>>>     >     > > >> > >> deployment,
>> > >     >> >> >>>>>     >     > > >> > >> >> and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > that
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > means
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > that
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > once
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > message
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> is
>> > >     >> >> >> written
>> > >     >> >> >>>> into the
>> > >     >> >> >>>>>     >     > > broker, it
>> > >     >> >> >>>>>     >     > > >> > >> will be
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > saved on
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > broker
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > with
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > that
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> specific
>> > >     >> >> >>>> namespace
>> > >     >> >> >>>>>     >     > (0x01).
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> If
>> > >     >> you
>> > >     >> >> >>>> were to
>> > >     >> >> >>>>>     > mirror
>> > >     >> >> >>>>>     >     > that
>> > >     >> >> >>>>>     >     > > >> > message
>> > >     >> >> >>>>>     >     > > >> > >> >> into
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > another
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > cluster,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > 0x01
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > would
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> accompany
>> > >     >> >> >>>> the
>> > >     >> >> >>>>>     > message,
>> > >     >> >> >>>>>     >     > > right?
>> > >     >> >> >>>>>     >     > > >> > What
>> > >     >> >> >>>>>     >     > > >> > >> if
>> > >     >> >> >>>>>     >     > > >> > >> >> the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > deployers of
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > same
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > app
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > in
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> other
>> > >     >> >> >>>> cluster uses
>> > >     >> >> >>>>>     > 0x57?
>> > >     >> >> >>>>>     >     > > They
>> > >     >> >> >>>>>     >     > > >> > won't
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > understand
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > each
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > other?
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> I'm
>> > >     >> >> not
>> > >     >> >> >>>> sure
>> > >     >> >> >>>>>     > that's an
>> > >     >> >> >>>>>     >     > > >> avoidable
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> problem. I
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > think it
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > simply
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > means
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > that
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > in
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> order
>> > >     >> >> to
>> > >     >> >> >>>> share
>> > >     >> >> >>>>>     > data, you
>> > >     >> >> >>>>>     >     > > have
>> > >     >> >> >>>>>     >     > > >> to
>> > >     >> >> >>>>>     >     > > >> > >> also
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> have a
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > shared
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > (agreed
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > upon)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> >>>> understanding of
>> > >     >> >> >>>>>     > what the
>> > >     >> >> >>>>>     >     > > >> > >> namespaces
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> mean.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> Which
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > I
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > think
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > makes
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > > sense,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> because
>> > >     >> >> >> the
>> > >     >> >> >>>>>     > alternate
>> > >     >> >> >>>>>     >     > > (sharing
>> > >     >> >> >>>>>     >     > > >> > >> >> *nothing*
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> at
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> all)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > would
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > mean
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > that
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > > there
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> would
>> > >     >> >> be
>> > >     >> >> >>>> no way to
>> > >     >> >> >>>>>     >     > > understand
>> > >     >> >> >>>>>     >     > > >> > each
>> > >     >> >> >>>>>     >     > > >> > >> >> other.
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> -James
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> Gwen
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> > On
>> > >     >> >> Tue,
>> > >     >> >> >>>> Nov 8,
>> > >     >> >> >>>>>     > 2016 at
>> > >     >> >> >>>>>     >     > > 4:23
>> > >     >> >> >>>>>     >     > > >> > PM,
>> > >     >> >> >>>>>     >     > > >> > >> >> radai <
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > >
>> > >     >> >> >>>> radai.rosenblatt@gmail.com
>> > >     >> >> >>>>>     > >
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >>
>> > >     >> wrote:
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >>
>> > >     >> +1
>> > >     >> >> for
>> > >     >> >> >>>> sean's
>> > >     >> >> >>>>>     >     > document.
>> > >     >> >> >>>>>     >     > > it
>> > >     >> >> >>>>>     >     > > >> > >> covers
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> pretty
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> much
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > all
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > the
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > >
>> > > trade-offs
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > and
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >>
>> > >     >> >> >> provides
>> > >     >> >> >>>>>     > concrete
>> > >     >> >> >>>>>     >     > > figures
>> > >     >> >> >>>>>     >     > > >> to
>> > >     >> >> >>>>>     >     > > >> > >> argue
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> about
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > :-)
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > > >> >>
>> > >     >> >> >>>> (nit-picking -
>> > >     >> >> >>>>>     > used
>> > >     >> >> >>>>>     >     > the
>> > >     >> >> >>>>>     >     > > >> same
>> > >     >> >> >>>>>     >     > > >> > >> xkcd
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> twice,
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> also
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> > trove
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > has
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > been
>> > >     >> >> >>>>>     >     > > >> > >> >> >>> > >> >     > >>  > > > > >
>> > >>
>> > >     >> >> superceded
>> > >     >> >> >>>>>     >     > > >> > >> >
>> > >     >> >> >>>>>     >     > > >> >
>> > >     >> >> >>>>>     >     > > >> >
>> > >     >> >> >>>>>     >     > > >> >
>> > >     >> >> >>>>>     >     > > >> > --
>> > >     >> >> >>>>>     >     > > >> > Gwen Shapira
>> > >     >> >> >>>>>     >     > > >> > Product Manager | Confluent
>> > >     >> >> >>>>>     >     > > >> > 650.450.2760 <(650)%20450-2760> |
>> > > @gwenshap
>> > >     >> >> >>>>>     >     > > >> > Follow us: Twitter | blog
>> > >     >> >> >>>>>     >     > > >> >
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >> --
>> > >     >> >> >>>>>     >     > > >> *Todd Palino*
>> > >     >> >> >>>>>     >     > > >> Staff Site Reliability Engineer
>> > >     >> >> >>>>>     >     > > >> Data Infrastructure Streaming
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > > >> linkedin.com/in/toddpalino
>> > >     >> >> >>>>>     >     > > >>
>> > >     >> >> >>>>>     >     > >
>> > >     >> >> >>>>>     >     > >
>> > >     >> >> >>>>>     >     > >
>> > >     >> >> >>>>>     >     > > --
>> > >     >> >> >>>>>     >     > > Gwen Shapira
>> > >     >> >> >>>>>     >     > > Product Manager | Confluent
>> > >     >> >> >>>>>     >     > > 650.450.2760 <(650)%20450-2760> |
>> @gwenshap
>> > >     >> >> >>>>>     >     > > Follow us: Twitter | blog
>> > >     >> >> >>>>>     >     > >
>> > >     >> >> >>>>>     >     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>     > The information contained in this email is
>> strictly
>> > >     >> >> confidential
>> > >     >> >> >>>> and for
>> > >     >> >> >>>>>     > the use of the addressee only, unless otherwise
>> > > indicated.
>> > >     >> If
>> > >     >> >> you
>> > >     >> >> >>>> are not
>> > >     >> >> >>>>>     > the intended recipient, please do not read, copy,
>> > > use or
>> > >     >> >> disclose
>> > >     >> >> >>>> to others
>> > >     >> >> >>>>>     > this message or any attachment. Please also
>> notify
>> > > the
>> > >     >> sender
>> > >     >> >> by
>> > >     >> >> >>>> replying
>> > >     >> >> >>>>>     > to this email or by telephone (+44(020 7896 0011)
>> > > and then
>> > >     >> >> delete
>> > >     >> >> >>>> the email
>> > >     >> >> >>>>>     > and any copies of it. Opinions, conclusion (etc)
>> > > that do not
>> > >     >> >> >>>> relate to the
>> > >     >> >> >>>>>     > official business of this company shall be
>> > > understood as
>> > >     >> >> neither
>> > >     >> >> >>>> given nor
>> > >     >> >> >>>>>     > endorsed by it. IG is a trading name of IG
>> Markets
>> > > Limited
>> > >     >> (a
>> > >     >> >> >>>> company
>> > >     >> >> >>>>>     > registered in England and Wales, company number
>> > > 04008957)
>> > >     >> and
>> > >     >> >> IG
>> > >     >> >> >>>> Index
>> > >     >> >> >>>>>     > Limited (a company registered in England and
>> Wales,
>> > > company
>> > >     >> >> >> number
>> > >     >> >> >>>>>     > 01190902). Registered address at Cannon Bridge
>> > > House, 25
>> > >     >> >> Dowgate
>> > >     >> >> >>>> Hill,
>> > >     >> >> >>>>>     > London EC4R 2YA. Both IG Markets Limited
>> (register
>> > > number
>> > >     >> >> 195355)
>> > >     >> >> >>>> and IG
>> > >     >> >> >>>>>     > Index Limited (register number 114059) are
>> > > authorised and
>> > >     >> >> >>>> regulated by the
>> > >     >> >> >>>>>     > Financial Conduct Authority.
>> > >     >> >> >>>>>     >
>> > >     >> >> >>>>>
>> > >     >> >> >>>>>
>> > >     >> >> >>>>> The information contained in this email is strictly
>> > > confidential
>> > >     >> and
>> > >     >> >> >> for
>> > >     >> >> >>>> the use of the addressee only, unless otherwise
>> indicated.
>> > > If you
>> > >     >> are
>> > >     >> >> >> not
>> > >     >> >> >>>> the intended recipient, please do not read, copy, use or
>> > > disclose
>> > >     >> to
>> > >     >> >> >> others
>> > >     >> >> >>>> this message or any attachment. Please also notify the
>> > > sender by
>> > >     >> >> >> replying
>> > >     >> >> >>>> to this email or by telephone (+44(020 7896 0011) and
>> then
>> > > delete
>> > >     >> the
>> > >     >> >> >> email
>> > >     >> >> >>>> and any copies of it. Opinions, conclusion (etc) that do
>> > > not relate
>> > >     >> >> to
>> > >     >> >> >> the
>> > >     >> >> >>>> official business of this company shall be understood as
>> > > neither
>> > >     >> >> given
>> > >     >> >> >> nor
>> > >     >> >> >>>> endorsed by it. IG is a trading name of IG Markets
>> Limited
>> > > (a
>> > >     >> company
>> > >     >> >> >>>> registered in England and Wales, company number
>> 04008957)
>> > > and IG
>> > >     >> >> Index
>> > >     >> >> >>>> Limited (a company registered in England and Wales,
>> > company
>> > > number
>> > >     >> >> >>>> 01190902). Registered address at Cannon Bridge House, 25
>> > > Dowgate
>> > >     >> >> Hill,
>> > >     >> >> >>>> London EC4R 2YA. Both IG Markets Limited (register
>> number
>> > > 195355)
>> > >     >> >> and IG
>> > >     >> >> >>>> Index Limited (register number 114059) are authorised
>> and
>> > > regulated
>> > >     >> >> by
>> > >     >> >> >> the
>> > >     >> >> >>>> Financial Conduct Authority.
>> > >     >> >> >>>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>>
>> > >     >> >> >>>
>> > >     >> >> >>
>> > >     >> >> >>
>> > >     >> >> >
>> > >     >> >>
>> > >     >> >>
>> > >     >> >
>> > >     >> The information contained in this email is strictly confidential
>> > > and for
>> > >     >> the use of the addressee only, unless otherwise indicated. If
>> you
>> > > are not
>> > >     >> the intended recipient, please do not read, copy, use or
>> disclose
>> > > to others
>> > >     >> this message or any attachment. Please also notify the sender by
>> > > replying
>> > >     >> to this email or by telephone (+44(020 7896 0011) and then
>> delete
>> > > the email
>> > >     >> and any copies of it. Opinions, conclusion (etc) that do not
>> > relate
>> > > to the
>> > >     >> official business of this company shall be understood as neither
>> > > given nor
>> > >     >> endorsed by it. IG is a trading name of IG Markets Limited (a
>> > > company
>> > >     >> registered in England and Wales, company number 04008957) and IG
>> > > Index
>> > >     >> Limited (a company registered in England and Wales, company
>> number
>> > >     >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> > > Hill,
>> > >     >> London EC4R 2YA. Both IG Markets Limited (register number
>> 195355)
>> > > and IG
>> > >     >> Index Limited (register number 114059) are authorised and
>> > regulated
>> > > by the
>> > >     >> Financial Conduct Authority.
>> > >     >>
>> > >
>> > >
>> > >
>> > >     --
>> > >     Gwen Shapira
>> > >     Product Manager | Confluent
>> > >     650.450.2760 | @gwenshap
>> > >     Follow us: Twitter | blog
>> > >
>> > >
>> > > The information contained in this email is strictly confidential and
>> for
>> > > the use of the addressee only, unless otherwise indicated. If you are
>> not
>> > > the intended recipient, please do not read, copy, use or disclose to
>> > others
>> > > this message or any attachment. Please also notify the sender by
>> replying
>> > > to this email or by telephone (+44(020 7896 0011) and then delete the
>> > email
>> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
>> > the
>> > > official business of this company shall be understood as neither given
>> > nor
>> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> > > registered in England and Wales, company number 04008957) and IG Index
>> > > Limited (a company registered in England and Wales, company number
>> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
>> IG
>> > > Index Limited (register number 114059) are authorised and regulated by
>> > the
>> > > Financial Conduct Authority.
>> > >
>> >
>>


Mime
View raw message