kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anna Povzner <a...@confluent.io>
Subject Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors
Date Mon, 01 Feb 2016 18:32:39 GMT
Re: CRC

Thanks everyone for your comments!

To summarize the discussion: We agreed that CRC is useful and should be
part of metadata (rather than interceptor API). One concern regarding using
CRC over the whole record is that CRC may be over-written by the broker.
Therefore, another option is to expose CRC over user bytes only. However,
disadvantages of this option is that we will have two types of CRC, which
is confusing, and CRC over the whole record makes more sense for durability
because it supposed to verify the whole record.

To address the concerns I propose that we expose CRC over the whole record,
but address the possibility of CRC over-written by the broker in the code
documentation. AFAIK there are two cases when that happens (compression
that I mentioned earlier only changes outer message CRC, and we are
concerned about inner message CRCs): 1) upgrade after a message format
change; and 2) topic is configured with timestamp type = LogAppendTime
(KIP-32). Also, I think the concern is due to the fact that we start with
only producer and consumer interceptors, and they may not “see” the change
in CRC. This can potentially be addressed by broker interceptors in the
future.

I will update the wiki with CRC discussion and start the voting thread.

Thanks,
Anna


On Sat, Jan 30, 2016 at 4:56 PM, Becket Qin <becket.qin@gmail.com> wrote:

> Hi Neha,
>
> CRC definitely qualifies as metadata of a record.
>
> My concern is only for CRC including system bytes. I just want to make sure
> including system bytes does not hurt the usefulness of CRC.
> If it does not then all my concerns have been addressed.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Sat, Jan 30, 2016 at 3:29 PM, Neha Narkhede <neha@confluent.io> wrote:
>
> > Hey Becket,
> >
> > Whether or not a specific use case of CRCs breaks or not is totally
> > dependent on how you are making use of the CRC for enabling monitoring. I
> > don't think it is productive to brainstorm all possible algorithms that
> > might or might not make use of CRC.
> >
> > Again, the more productive discussion is whether or not the CRC qualifies
> > as the record's metadata or not.
> >
> > Thanks,
> > Neha
> >
> > On Sat, Jan 30, 2016 at 2:40 PM, Becket Qin <becket.qin@gmail.com>
> wrote:
> >
> > > Hi Neha and Jay,
> > >
> > > Yeah... Having multiple CRCs is ugly. I might be over concerning. The
> > > situation I was worrying is something as following.
> > >
> > > 1. Assuming there is a pipeline having two Kafka clusters with Mirror
> > Maker
> > > between them. Producers are producing to the MM source cluster and
> > > consumers are consuming from the MM target cluster.
> > >
> > > 2. We have some end to end auditing solution based on CRC reported by
> > > producer and consumer.
> > >
> > > 3. Initially both Kafka cluster are using the same message format so
> the
> > > CRC of the messages in the pipeline do not change. Producers and
> > consumers
> > > always see the same CRC for the same message.
> > >
> > > 4. Now we have a message format change and we want to upgrade the two
> > Kafka
> > > clusters and the clients. Because it is very difficult to ensure all
> the
> > > clients and brokers are upgraded at the same time, message format
> > > conversion might be unavoidable. Hence during the protocol upgrade,
> > > producers and consumers may see different CRC. This will break auditing
> > > which is based on the assumption that CRC does not change.
> > >
> > > I know this is a very specific use case of CRC and maybe in other cases
> > CRC
> > > mismatch between producer and consumer is not an issue. But since Anna
> > > mentioned end to end auditing, I just take this as an example.
> > >
> > > Am I missing something?
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > >
> > > On Sat, Jan 30, 2016 at 8:43 AM, Neha Narkhede <neha@confluent.io>
> > wrote:
> > >
> > > > Nice finding on the CRC class. It will be great to switch to that at
> > some
> > > > point.
> > > >
> > > > On exposing the CRC - I think we are overthinking the problem. CRC
> over
> > > the
> > > > entire record makes sense for the durability check and having
> multiple
> > > CRCs
> > > > is a bad idea.
> > > >
> > > > On Fri, Jan 29, 2016 at 4:47 PM, Jay Kreps <jay@confluent.io> wrote:
> > > >
> > > > > The rationale for the CRC covering the whole record was to check
> > > > corruption
> > > > > in the full record contents as corruption there will equally
> prevent
> > > > > someone trying to consume the data. I think you could argue either
> > way
> > > > but
> > > > > let's definitely not end up with two different CRC calculations,
> that
> > > > would
> > > > > just be indecisive.
> > > > >
> > > > > That is a super interesting finding about the CRC class. I think
> you
> > > are
> > > > > right that it became an intrinsic in java 8 with hw support if
> > > available.
> > > > > We should really switch to that. The CRC calculation is still one
> of
> > > the
> > > > > biggest bottlenecks in the producer and consumer so that should
> > > > > significantly speed things up.
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Fri, Jan 29, 2016 at 4:30 PM, Becket Qin <becket.qin@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi Anna,
> > > > > >
> > > > > > I think there is value if CRC for only user bytes can be used.
> This
> > > > will
> > > > > > help when we have future protocol updates. Otherwise any protocol
> > > > > migration
> > > > > > might break auditing if it largely relies on CRC including system
> > > > bytes.
> > > > > >
> > > > > > I did some test to understand the performance overhead of having
> a
> > > > > separate
> > > > > > user bytes CRC,
> > > > > >
> > > > > > On my desktop, it took ~4.7 seconds to compute CRC for 10G bytes.
> > So
> > > > for
> > > > > > each megabyte it takes 0.47 ms.
> > > > > > To compare with compression cost, I used the compressor used by
> > > > producer.
> > > > > > The test uses GZIP and compress 1MB of data for each batch.
> > > > > > It takes:
> > > > > > ~90 seconds to compress 10GB bytes that contains all same byte.
> > > > > > ~470 seconds to compress 10GB bytes containing random bytes.
> > > > > > >1000 seconds to compress 10GB bytes with 4, 8, 16, 32 and 64
> > > patterns.
> > > > > (I
> > > > > > guess it takes time to update the pattern mapping)
> > > > > >
> > > > > > So the overhead of a separate CRC computing is <1% considering
> the
> > > > > producer
> > > > > > do a lot of things other than compression.
> > > > > >
> > > > > > I am not sure if this overhead is worth taking or not, because we
> > do
> > > > > have a
> > > > > > huge number of messages to send, so any overhead should be
> avoided
> > if
> > > > > > possible.
> > > > > >
> > > > > > BTW. Another interesting finding is that the Crc32 class used in
> > > > clients
> > > > > > which used to be faster than java CRC32 in Java 1.6. It seems no
> > > longer
> > > > > the
> > > > > > case. What I see is that Java CRC32 class is 2x faster than the
> > Crc32
> > > > > class
> > > > > > we are using now.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 29, 2016 at 1:46 PM, Anna Povzner <anna@confluent.io
> >
> > > > wrote:
> > > > > >
> > > > > > > Joel, thanks for your feedback. I updated the wiki based on
> your
> > > > > comments
> > > > > > > about the wiki writeup.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 29, 2016 at 11:50 AM, Anna Povzner <
> > anna@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Becket,
> > > > > > > >
> > > > > > > > In your scenario with one message from producer A and one
> > message
> > > > > from
> > > > > > > > producer B, those are two different messages, and they should
> > be
> > > > > > tracked
> > > > > > > as
> > > > > > > > two different messages. So I would argue for using record CRC
> > --
> > > > CRC
> > > > > > that
> > > > > > > > is actually used by the system + it will not require
> computing
> > a
> > > > > > > different
> > > > > > > > CRC again which will add performance overhead.
> > > > > > > >
> > > > > > > > If the broker ever changes the CRC, the scenarios when that
> > > happens
> > > > > > > should
> > > > > > > > be very well defined. As far as I know, the scenarios when
> CRC
> > is
> > > > > > > > overwritten by the broker (including KIP-31/32 changes):
> > > > > > > > -- topic config is LogAppendTime for timestamp type
> > > > > > > > -- upgrade/downgrade
> > > > > > > > -- compression codec change (which could be inferred from
> > > config).
> > > > > > > >
> > > > > > > > Monitoring/audit just needs to know when CRCs are safe to
> use,
> > > > which
> > > > > is
> > > > > > > > most often is known from config. In the future, this can be
> > > further
> > > > > > > > addressed by broker interceptors.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Anna
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jan 29, 2016 at 11:30 AM, Becket Qin <
> > > becket.qin@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Neha,
> > > > > > > >>
> > > > > > > >> CRC is definitely an important type of metadata of a
> record. I
> > > am
> > > > > not
> > > > > > > >> arguing about that. But I think we should distinguish
> between
> > > two
> > > > > > types
> > > > > > > of
> > > > > > > >> checksum here, 1) the checksum of user data. and 2) the
> > checksum
> > > > > > > including
> > > > > > > >> system appended bytes.
> > > > > > > >>
> > > > > > > >> I completely agree that (1) is good to add. But I am not
> sure
> > if
> > > > we
> > > > > > > should
> > > > > > > >> expose (2) to user, because this means any underlying
> protocol
> > > > > change
> > > > > > > will
> > > > > > > >> give a different CRC for exact same message. For example,
> > let's
> > > > say
> > > > > > > >> producer A is sending message with timestamp. Producer B is
> > > > sending
> > > > > > > >> message
> > > > > > > >> without timestamp. Even they are given the exact same
> message,
> > > the
> > > > > CRC
> > > > > > > >> returned would be different.
> > > > > > > >>
> > > > > > > >> Also, Kafka broker will modify the system appended bytes in
> > > > > different
> > > > > > > >> scenarios, such as compression codec change, message format
> > > > > > > >> conversion(After KIP-31 and KIP-32).
> > > > > > > >>
> > > > > > > >> So my concern is that we are exposing CRC which including
> > system
> > > > > > > appended
> > > > > > > >> bytes to user.
> > > > > > > >>
> > > > > > > >> Other than this I think everything looks good. Nice work,
> > Anna.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >>
> > > > > > > >> Jiangjie (Becket) Qin
> > > > > > > >>
> > > > > > > >> On Fri, Jan 29, 2016 at 8:11 AM, Joel Koshy <
> > > jjkoshy.w@gmail.com>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Responding to some of the earlier comments in the thread:
> > > > > > > >> >
> > > > > > > >> > @Jay/@Neha,
> > > > > > > >> >
> > > > > > > >> > I think any one of onCommit/onAppend/onArrival would work
> > for
> > > > the
> > > > > > > >> concrete
> > > > > > > >> > use-case that I had outlined. I think onArrival is
> > > additionally
> > > > > > useful
> > > > > > > >> for
> > > > > > > >> > custom validation - i.e., reject the message and do not
> > append
> > > > if
> > > > > it
> > > > > > > >> > violates some cluster-specific rule (for e.g., if some
> > header
> > > > > > > timestamp
> > > > > > > >> is
> > > > > > > >> > older than xyz). However, the thing with user-supplied
> > > > validation
> > > > > is
> > > > > > > we
> > > > > > > >> > would have to do with a (new) generic error code in the
> > > producer
> > > > > > > >> response.
> > > > > > > >> > While there is a risk of a broker interceptor having high
> > > > latency
> > > > > I
> > > > > > > >> think
> > > > > > > >> > that is acceptable since it is the user's responsibility
> to
> > > > ensure
> > > > > > low
> > > > > > > >> > latency - the producer call-back and onAcknowledgment
> > > > interceptor
> > > > > > are
> > > > > > > >> > similar in this regard although those are less risky. Even
> > > so, I
> > > > > > think
> > > > > > > >> > there are clear use-cases for broker interceptors so I
> feel
> > > the
> > > > > risk
> > > > > > > >> part
> > > > > > > >> > is something that just needs to be documented. @Jay that
> is
> > a
> > > > good
> > > > > > > point
> > > > > > > >> > about moving from Message/MessageSet to Records although
> > that
> > > > may
> > > > > be
> > > > > > > >> less
> > > > > > > >> > difficult to absorb since it is a broker-side interceptor
> > and
> > > so
> > > > > > > people
> > > > > > > >> > don't need to get a ton of applications in their company
> to
> > > > switch
> > > > > > to
> > > > > > > >> use
> > > > > > > >> > it.
> > > > > > > >> >
> > > > > > > >> > Re: onEnqueued: monitoring serialization latency can be
> done
> > > via
> > > > > > > metrics
> > > > > > > >> > but this is more useful for recording whether
> serialization
> > > > > > succeeded
> > > > > > > or
> > > > > > > >> > not. onAcknowledgment subsumes this but it also subsumes
> > other
> > > > > > > possible
> > > > > > > >> > errors (such as produce errors). It is more fine-grained
> > than
> > > > most
> > > > > > > >> people
> > > > > > > >> > need though (i.e., I don't think we will use it even if it
> > is
> > > > > > > present.)
> > > > > > > >> >
> > > > > > > >> > Re: checksums: I think it is a good addition to metadata;
> > and
> > > > for
> > > > > > > >> > low-volume or super-critical topics can be used for very
> > > strict
> > > > > > > >> auditing.
> > > > > > > >> >
> > > > > > > >> > There are a couple of typos/edits for the wiki itself:
> > > > > > > >> >
> > > > > > > >> >    - Under Kafka Producer changes:
> > > > > > > >> >    - you have references to KafkaConsumer constructor and
> > > > > > > >> >       ConsumerConfig.originals.
> > > > > > > >> >       - sendRecord -> sentRecord (may be clearer)
> > > > > > > >> >    - Under ProducerInterceptor interface: there is a
> mention
> > > of
> > > > > > > >> onEnqueued
> > > > > > > >> >    which was rejected
> > > > > > > >> >    - Comment for ConsumerRecord.record should probably be:
> > //
> > > > NEW:
> > > > > > > >> record
> > > > > > > >> >    size in bytes (*after decompression*)
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > BTW - Anna, nice work on the KIP!
> > > > > > > >> >
> > > > > > > >> > Joel
> > > > > > > >> >
> > > > > > > >> > On Fri, Jan 29, 2016 at 6:57 AM, Neha Narkhede <
> > > > neha@confluent.io
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> > > Becket,
> > > > > > > >> > >
> > > > > > > >> > > Is your concern the presence of CRC in the
> RecordMetadata
> > or
> > > > do
> > > > > > you
> > > > > > > >> want
> > > > > > > >> > to
> > > > > > > >> > > brainstorm how CRC can be used for auditing? I think we
> > > > > shouldn't
> > > > > > > try
> > > > > > > >> to
> > > > > > > >> > > think about the various ways that people can do
> monitoring
> > > > using
> > > > > > > >> > > interceptors and the metadata we provide. The entire
> point
> > > of
> > > > > > having
> > > > > > > >> > > pluggable interceptors is so that people can employ
> their
> > > own
> > > > > > > creative
> > > > > > > >> > > mechanisms to make use of interceptors.
> > > > > > > >> > >
> > > > > > > >> > > I do think that it is worth discussing whether or not
> CRC
> > > > makes
> > > > > > > sense
> > > > > > > >> as
> > > > > > > >> > > record metadata to the user. My take is that the CRC is
> > the
> > > > best
> > > > > > > >> > size-bound
> > > > > > > >> > > summary of serialized record content available to us
> which
> > > is
> > > > > > > >> expensive
> > > > > > > >> > to
> > > > > > > >> > > recompute if the user were to redo it. I'd argue this
> > > summary
> > > > > of a
> > > > > > > >> record
> > > > > > > >> > > qualifies as its metadata. After all, we use the record
> > CRC
> > > > for
> > > > > a
> > > > > > > very
> > > > > > > >> > > important test of the system durability as it travels
> > > through
> > > > > the
> > > > > > > >> system.
> > > > > > > >> > >
> > > > > > > >> > > 1. Isn't the TopicPartition + Offset already uniquely
> > > > > identified a
> > > > > > > >> > message?
> > > > > > > >> > > > It seems better than CRC no matter from summary point
> of
> > > > view
> > > > > or
> > > > > > > >> > auditing
> > > > > > > >> > > > point of view.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > The offset is a system-assigned value of uniqueness to
> the
> > > > > > message.
> > > > > > > If
> > > > > > > >> > you
> > > > > > > >> > > trusted the system that much, you are not looking to
> > monitor
> > > > it
> > > > > > > >> > out-of-band
> > > > > > > >> > > :-)
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > > 2. Currently CRC only has 4 bytes. So it will have
> > > collision
> > > > > > when
> > > > > > > >> there
> > > > > > > >> > > are
> > > > > > > >> > > > more than ~4 billion messages. Take LinkedIn as an
> > > example,
> > > > we
> > > > > > > have
> > > > > > > >> 1.3
> > > > > > > >> > > > trillion messages per day. So there will be at least a
> > > > couple
> > > > > of
> > > > > > > >> > hundreds
> > > > > > > >> > > > collision for each CRC code every day, whereas
> > > > > > > TopicPartition+Offset
> > > > > > > >> > will
> > > > > > > >> > > > not have any collision.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > The CRC isn't sent over the wire and doesn't add any
> extra
> > > > > > overhead
> > > > > > > in
> > > > > > > >> > > processing, so what is your concern? If you aren't
> > convinced
> > > > > about
> > > > > > > its
> > > > > > > >> > > usefulness, you can always use the default do-nothing
> > > > > interceptor
> > > > > > at
> > > > > > > >> > > LinkedIn and ignore the CRC.
> > > > > > > >> > >
> > > > > > > >> > > Without having
> > > > > > > >> > > > the entire message bytes, they may not be able to
> verify
> > > its
> > > > > > > >> > correctness,
> > > > > > > >> > > > and the CRC could even be invalid if the broker ever
> > > > > overwritten
> > > > > > > any
> > > > > > > >> > > field
> > > > > > > >> > > > or did format conversion.
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > > This doesn't make sense to me. The CRC is used for the
> > most
> > > > > > > important
> > > > > > > >> > > durability check by Kafka - to verify that the message
> was
> > > not
> > > > > > > garbled
> > > > > > > >> > over
> > > > > > > >> > > the wire. The system can't change it; it has to match on
> > the
> > > > > > > consumer
> > > > > > > >> > side
> > > > > > > >> > > or we will not return it to the user.
> > > > > > > >> > >
> > > > > > > >> > > On Fri, Jan 29, 2016 at 3:23 AM, Becket Qin <
> > > > > becket.qin@gmail.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Anna,
> > > > > > > >> > > >
> > > > > > > >> > > > It is still not clear to me why we should expose CRC
> to
> > > end
> > > > > > user.
> > > > > > > >> > > > Followings are my confusions.
> > > > > > > >> > > >
> > > > > > > >> > > > 1. Isn't the TopicPartition + Offset already uniquely
> > > > > > identified a
> > > > > > > >> > > message?
> > > > > > > >> > > > It seems better than CRC no matter from summary point
> of
> > > > view
> > > > > or
> > > > > > > >> > auditing
> > > > > > > >> > > > point of view.
> > > > > > > >> > > >
> > > > > > > >> > > > 2. Currently CRC only has 4 bytes. So it will have
> > > collision
> > > > > > when
> > > > > > > >> there
> > > > > > > >> > > are
> > > > > > > >> > > > more than ~4 billion messages. Take LinkedIn as an
> > > example,
> > > > we
> > > > > > > have
> > > > > > > >> 1.3
> > > > > > > >> > > > trillion messages per day. So there will be at least a
> > > > couple
> > > > > of
> > > > > > > >> > hundreds
> > > > > > > >> > > > collision for each CRC code every day, whereas
> > > > > > > TopicPartition+Offset
> > > > > > > >> > will
> > > > > > > >> > > > not have any collision.
> > > > > > > >> > > >
> > > > > > > >> > > > 3. CRC is calculated after all the fields have been
> > filled
> > > > in
> > > > > by
> > > > > > > >> > > producer,
> > > > > > > >> > > > including timestamp, attributes, etc. It might also
> get
> > > > > > recomputed
> > > > > > > >> on
> > > > > > > >> > > > broker side. So if users only get CRC from record
> > > metadata.
> > > > > > > Without
> > > > > > > >> > > having
> > > > > > > >> > > > the entire message bytes, they may not be able to
> verify
> > > its
> > > > > > > >> > correctness,
> > > > > > > >> > > > and the CRC could even be invalid if the broker ever
> > > > > overwritten
> > > > > > > any
> > > > > > > >> > > field
> > > > > > > >> > > > or did format conversion.
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks,
> > > > > > > >> > > >
> > > > > > > >> > > > Jiangjie (Becket) Qin
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > On Thu, Jan 28, 2016 at 5:58 PM, Anna Povzner <
> > > > > > anna@confluent.io>
> > > > > > > >> > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > On a second thought, yes, I think we should expose
> > > record
> > > > > size
> > > > > > > >> that
> > > > > > > >> > > > > represents application bytes. This is Becket's
> option
> > > #1.
> > > > > > > >> > > > >
> > > > > > > >> > > > > I updated the KIP wiki with new fields in
> > RecordMetadata
> > > > and
> > > > > > > >> > > > > ConsumerRecord.
> > > > > > > >> > > > >
> > > > > > > >> > > > > I would like to start a voting thread tomorrow if
> > there
> > > > are
> > > > > no
> > > > > > > >> > > objections
> > > > > > > >> > > > > or more things to discuss.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > Anna
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Thu, Jan 28, 2016 at 2:25 PM, Anna Povzner <
> > > > > > > anna@confluent.io>
> > > > > > > >> > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Regarding record size as bytes sent over the wire,
> > my
> > > > > > concern
> > > > > > > is
> > > > > > > >> > that
> > > > > > > >> > > > it
> > > > > > > >> > > > > > is almost impossible to calculate per-record. We
> > could
> > > > do
> > > > > > as:
> > > > > > > 1)
> > > > > > > >> > > > > compressed
> > > > > > > >> > > > > > bytes / number of records in a compressed message,
> > as
> > > > Todd
> > > > > > > >> > mentioned;
> > > > > > > >> > > > or
> > > > > > > >> > > > > 2)
> > > > > > > >> > > > > > or same as #1 but take it proportional to
> > uncompressed
> > > > > > record
> > > > > > > >> size
> > > > > > > >> > > vs.
> > > > > > > >> > > > > > total uncompressed size of records. All of these
> > > > > > calculations
> > > > > > > >> give
> > > > > > > >> > us
> > > > > > > >> > > > an
> > > > > > > >> > > > > > estimate. So maybe record size as bytes sent over
> > the
> > > > wire
> > > > > > is
> > > > > > > >> not a
> > > > > > > >> > > > > > per-record metadata, but rather per
> topic/partition
> > > > > measure
> > > > > > > >> that is
> > > > > > > >> > > > > better
> > > > > > > >> > > > > > to be exposed through metrics?
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Thu, Jan 28, 2016 at 2:09 PM, Todd Palino <
> > > > > > > tpalino@gmail.com
> > > > > > > >> >
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> It may be difficult (or nearly impossible) to get
> > > > actual
> > > > > > > >> > compressed
> > > > > > > >> > > > > bytes
> > > > > > > >> > > > > >> for a message from a compressed batch, but I do
> > think
> > > > > it’s
> > > > > > > >> useful
> > > > > > > >> > > > > >> information to have available for the very reason
> > > > noted,
> > > > > > > >> bandwidth
> > > > > > > >> > > > > >> consumed. Does it make sense to have an
> interceptor
> > > at
> > > > > the
> > > > > > > >> batch
> > > > > > > >> > > level
> > > > > > > >> > > > > >> that
> > > > > > > >> > > > > >> can provide this? The other option is to estimate
> > it
> > > > > (such
> > > > > > as
> > > > > > > >> > making
> > > > > > > >> > > > an
> > > > > > > >> > > > > >> assumption that the messages in a batch are equal
> > in
> > > > > size,
> > > > > > > >> which
> > > > > > > >> > is
> > > > > > > >> > > > not
> > > > > > > >> > > > > >> necessarily true), which is probably not the
> right
> > > > > answer.
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> -Todd
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> On Thu, Jan 28, 2016 at 1:48 PM, Anna Povzner <
> > > > > > > >> anna@confluent.io>
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> > Hi Becket,
> > > > > > > >> > > > > >> >
> > > > > > > >> > > > > >> > It will be up to the interceptor to implement
> > their
> > > > > audit
> > > > > > > or
> > > > > > > >> > > > > monitoring
> > > > > > > >> > > > > >> > strategy. I would also imagine there is more
> than
> > > one
> > > > > > good
> > > > > > > >> way
> > > > > > > >> > to
> > > > > > > >> > > do
> > > > > > > >> > > > > >> audit.
> > > > > > > >> > > > > >> > So, I agree that some of the interceptors may
> not
> > > use
> > > > > > CRC,
> > > > > > > >> and
> > > > > > > >> > we
> > > > > > > >> > > > will
> > > > > > > >> > > > > >> not
> > > > > > > >> > > > > >> > require it. The question is now whether
> > > intercepting
> > > > > CRCs
> > > > > > > is
> > > > > > > >> > > > needed. I
> > > > > > > >> > > > > >> > think they are very useful for monitoring and
> > > audit,
> > > > > > > because
> > > > > > > >> CRC
> > > > > > > >> > > > > >> provides
> > > > > > > >> > > > > >> > an a easy way to get a summary of a message,
> > rather
> > > > > than
> > > > > > > >> using
> > > > > > > >> > > > message
> > > > > > > >> > > > > >> > bytes or key/value objects.
> > > > > > > >> > > > > >> >
> > > > > > > >> > > > > >> > Regarding record size, I agree that bandwidth
> > > example
> > > > > was
> > > > > > > >> not a
> > > > > > > >> > > good
> > > > > > > >> > > > > >> one. I
> > > > > > > >> > > > > >> > think it would be hard to get actual bytes sent
> > > over
> > > > > the
> > > > > > > wire
> > > > > > > >> > > (your
> > > > > > > >> > > > > #2),
> > > > > > > >> > > > > >> > since multiple records get compressed together
> > and
> > > we
> > > > > > would
> > > > > > > >> need
> > > > > > > >> > > to
> > > > > > > >> > > > > >> decide
> > > > > > > >> > > > > >> > which bytes to account to which record. So I am
> > > > > inclined
> > > > > > to
> > > > > > > >> only
> > > > > > > >> > > do
> > > > > > > >> > > > > your
> > > > > > > >> > > > > >> > #1. However, it still makes more sense to me
> just
> > > to
> > > > > > return
> > > > > > > >> > record
> > > > > > > >> > > > > size
> > > > > > > >> > > > > >> > including the header, since this is the actual
> > > record
> > > > > > size.
> > > > > > > >> > > > > >> >
> > > > > > > >> > > > > >> > Thanks,
> > > > > > > >> > > > > >> > Anna
> > > > > > > >> > > > > >> >
> > > > > > > >> > > > > >> > On Thu, Jan 28, 2016 at 11:46 AM, Becket Qin <
> > > > > > > >> > > becket.qin@gmail.com>
> > > > > > > >> > > > > >> wrote:
> > > > > > > >> > > > > >> >
> > > > > > > >> > > > > >> > > Anna,
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> > > Using CRC to do end2end auditing might be
> very
> > > > costly
> > > > > > > >> because
> > > > > > > >> > > you
> > > > > > > >> > > > > will
> > > > > > > >> > > > > >> > need
> > > > > > > >> > > > > >> > > to collect all the CRC from both producer and
> > > > > consumer.
> > > > > > > >> And it
> > > > > > > >> > > is
> > > > > > > >> > > > > >> based
> > > > > > > >> > > > > >> > on
> > > > > > > >> > > > > >> > > the assumption that broker does not modify
> the
> > > > > record.
> > > > > > > >> > > > > >> > > Can you shed some idea on how end to end
> > auditing
> > > > > will
> > > > > > be
> > > > > > > >> > using
> > > > > > > >> > > > the
> > > > > > > >> > > > > >> CRC
> > > > > > > >> > > > > >> > > before we decide to expose such low level
> > detail
> > > to
> > > > > the
> > > > > > > end
> > > > > > > >> > > user?
> > > > > > > >> > > > It
> > > > > > > >> > > > > >> > would
> > > > > > > >> > > > > >> > > also be helpful if you can compare it with
> > > > something
> > > > > > like
> > > > > > > >> > > sequence
> > > > > > > >> > > > > >> number
> > > > > > > >> > > > > >> > > based auditing.
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> > > About the record size, one thing worth notice
> > is
> > > > that
> > > > > > the
> > > > > > > >> size
> > > > > > > >> > > of
> > > > > > > >> > > > > >> Record
> > > > > > > >> > > > > >> > is
> > > > > > > >> > > > > >> > > not the actual bytes sent over the wire if we
> > use
> > > > > > > >> compression.
> > > > > > > >> > > So
> > > > > > > >> > > > > that
> > > > > > > >> > > > > >> > does
> > > > > > > >> > > > > >> > > not really tell user how much bandwidth they
> > are
> > > > > using.
> > > > > > > >> > > > Personally I
> > > > > > > >> > > > > >> > think
> > > > > > > >> > > > > >> > > two kinds of size may be useful.
> > > > > > > >> > > > > >> > > 1. The record size after serialization, i.e.
> > > > > > application
> > > > > > > >> > bytes.
> > > > > > > >> > > > (The
> > > > > > > >> > > > > >> > > uncompressed record size can be easily
> derived
> > as
> > > > > well)
> > > > > > > >> > > > > >> > > 2. The actual bytes sent over the wire.
> > > > > > > >> > > > > >> > > We can get (1) easily, but (2) is difficult
> to
> > > get
> > > > at
> > > > > > > >> Record
> > > > > > > >> > > level
> > > > > > > >> > > > > >> when
> > > > > > > >> > > > > >> > we
> > > > > > > >> > > > > >> > > use compression.
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> > > Thanks,
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> > > Jiangjie (Becket) Qin
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> > > On Thu, Jan 28, 2016 at 10:55 AM, Anna
> Povzner
> > <
> > > > > > > >> > > anna@confluent.io
> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > wrote:
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> > > > Hi Becket,
> > > > > > > >> > > > > >> > > >
> > > > > > > >> > > > > >> > > > The use-case for CRC is end-to-end audit,
> > > rather
> > > > > than
> > > > > > > >> > checking
> > > > > > > >> > > > > >> whether
> > > > > > > >> > > > > >> > a
> > > > > > > >> > > > > >> > > > single message is corrupt or not.
> > > > > > > >> > > > > >> > > >
> > > > > > > >> > > > > >> > > > Regarding record size, I was thinking to
> > > extract
> > > > > > record
> > > > > > > >> size
> > > > > > > >> > > > from
> > > > > > > >> > > > > >> > Record.
> > > > > > > >> > > > > >> > > > That will include header overhead as well.
> I
> > > > think
> > > > > > > total
> > > > > > > >> > > record
> > > > > > > >> > > > > size
> > > > > > > >> > > > > >> > will
> > > > > > > >> > > > > >> > > > tell users how much bandwidth their
> messages
> > > > take.
> > > > > > > Since
> > > > > > > >> > > header
> > > > > > > >> > > > is
> > > > > > > >> > > > > >> > > > relatively small and constant, users also
> > will
> > > > get
> > > > > an
> > > > > > > >> idea
> > > > > > > >> > of
> > > > > > > >> > > > > their
> > > > > > > >> > > > > >> > > > key/value sizes.
> > > > > > > >> > > > > >> > > >
> > > > > > > >> > > > > >> > > > Thanks,
> > > > > > > >> > > > > >> > > > Anna
> > > > > > > >> > > > > >> > > >
> > > > > > > >> > > > > >> > > > On Thu, Jan 28, 2016 at 9:29 AM, Becket
> Qin <
> > > > > > > >> > > > becket.qin@gmail.com
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > wrote:
> > > > > > > >> > > > > >> > > >
> > > > > > > >> > > > > >> > > > > I am +1 on #1.2 and #3.
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > #2: Regarding CRC, I am not sure if users
> > > care
> > > > > > about
> > > > > > > >> CRC.
> > > > > > > >> > is
> > > > > > > >> > > > > there
> > > > > > > >> > > > > >> > any
> > > > > > > >> > > > > >> > > > > specific use case? Currently we validate
> > > > messages
> > > > > > by
> > > > > > > >> > calling
> > > > > > > >> > > > > >> > > > ensureValid()
> > > > > > > >> > > > > >> > > > > to verify the checksum and throw
> exception
> > if
> > > > it
> > > > > > does
> > > > > > > >> not
> > > > > > > >> > > > match.
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > Message size would be useful. We can add
> > that
> > > > to
> > > > > > > >> > > > ConsumerRecord.
> > > > > > > >> > > > > >> Can
> > > > > > > >> > > > > >> > > you
> > > > > > > >> > > > > >> > > > > clarify the message size you are
> referring
> > > to?
> > > > > Does
> > > > > > > it
> > > > > > > >> > > include
> > > > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > message
> > > > > > > >> > > > > >> > > > > header overhead or not? From user's point
> > of
> > > > > view,
> > > > > > > they
> > > > > > > >> > > > probably
> > > > > > > >> > > > > >> > don't
> > > > > > > >> > > > > >> > > > care
> > > > > > > >> > > > > >> > > > > about header size.
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > Thanks,
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > Jiangjie (Becket) Qin
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > On Wed, Jan 27, 2016 at 8:26 PM, Neha
> > > Narkhede
> > > > <
> > > > > > > >> > > > > neha@confluent.io
> > > > > > > >> > > > > >> >
> > > > > > > >> > > > > >> > > > wrote:
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > Anna,
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > Thanks for being diligent.
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > +1 on #1.2 and sounds good on #3. I
> > > recommend
> > > > > > > adding
> > > > > > > >> > > > checksum
> > > > > > > >> > > > > >> and
> > > > > > > >> > > > > >> > > size
> > > > > > > >> > > > > >> > > > > > fields to RecordMetadata and
> > ConsumerRecord
> > > > > > instead
> > > > > > > >> of
> > > > > > > >> > > > > exposing
> > > > > > > >> > > > > >> > > > metadata
> > > > > > > >> > > > > >> > > > > > piecemeal in the interceptor APIs.
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > Neha
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > On Wed, Jan 27, 2016 at 4:10 PM, Anna
> > > > Povzner <
> > > > > > > >> > > > > >> anna@confluent.io>
> > > > > > > >> > > > > >> > > > wrote:
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > Hi All,
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > The KIP wiki page is now up-to-date
> > with
> > > > the
> > > > > > > scope
> > > > > > > >> we
> > > > > > > >> > > have
> > > > > > > >> > > > > >> agreed
> > > > > > > >> > > > > >> > > on:
> > > > > > > >> > > > > >> > > > > > > Producer and Consumer Interceptors
> > with a
> > > > > > minimal
> > > > > > > >> set
> > > > > > > >> > of
> > > > > > > >> > > > > >> mutable
> > > > > > > >> > > > > >> > > API
> > > > > > > >> > > > > >> > > > > that
> > > > > > > >> > > > > >> > > > > > > are not dependent on producer and
> > > consumer
> > > > > > > internal
> > > > > > > >> > > > > >> > implementation.
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > I have few more API details that I
> > would
> > > > like
> > > > > > to
> > > > > > > >> bring
> > > > > > > >> > > > > >> attention
> > > > > > > >> > > > > >> > to
> > > > > > > >> > > > > >> > > > > > or/and
> > > > > > > >> > > > > >> > > > > > > discuss:
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > 1. Handling exceptions
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > Exceptions can provide an additional
> > > level
> > > > of
> > > > > > > >> control.
> > > > > > > >> > > For
> > > > > > > >> > > > > >> > example,
> > > > > > > >> > > > > >> > > > we
> > > > > > > >> > > > > >> > > > > > can
> > > > > > > >> > > > > >> > > > > > > filter messages on consumer side or
> > stop
> > > > > > messages
> > > > > > > >> on
> > > > > > > >> > > > > producer
> > > > > > > >> > > > > >> if
> > > > > > > >> > > > > >> > > they
> > > > > > > >> > > > > >> > > > > > don’t
> > > > > > > >> > > > > >> > > > > > > have the right field.
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > I see two options:
> > > > > > > >> > > > > >> > > > > > > 1.1. For callbacks that can mutate
> > > records
> > > > > > > (onSend
> > > > > > > >> and
> > > > > > > >> > > > > >> > onConsume),
> > > > > > > >> > > > > >> > > > > > > propagate exceptions through the
> > original
> > > > > calls
> > > > > > > >> > > > > >> > > (KafkaProducer.send()
> > > > > > > >> > > > > >> > > > > and
> > > > > > > >> > > > > >> > > > > > > KafkaConsumer.poll() respectively).
> For
> > > > other
> > > > > > > >> > callbacks,
> > > > > > > >> > > > > catch
> > > > > > > >> > > > > >> > > > > exception,
> > > > > > > >> > > > > >> > > > > > > log, and ignore.
> > > > > > > >> > > > > >> > > > > > > 1.2. Catch exceptions from all the
> > > > > interceptor
> > > > > > > >> > callbacks
> > > > > > > >> > > > and
> > > > > > > >> > > > > >> > > ignore.
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > The issue with 1.1. is that it
> > > effectively
> > > > > > > changes
> > > > > > > >> > > > > >> > > > KafkaProducer.send()
> > > > > > > >> > > > > >> > > > > > and
> > > > > > > >> > > > > >> > > > > > > KafkaConsumer.poll() API, since now
> > they
> > > > may
> > > > > > > throw
> > > > > > > >> > > > > exceptions
> > > > > > > >> > > > > >> > that
> > > > > > > >> > > > > >> > > > are
> > > > > > > >> > > > > >> > > > > > not
> > > > > > > >> > > > > >> > > > > > > documented in KafkaProducer/Consumer
> > API.
> > > > > > Another
> > > > > > > >> > option
> > > > > > > >> > > > is
> > > > > > > >> > > > > to
> > > > > > > >> > > > > >> > > allow
> > > > > > > >> > > > > >> > > > to
> > > > > > > >> > > > > >> > > > > > > propagate some exceptions, and ignore
> > > > others.
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > I think our use-cases do not require
> > > > > > propagating
> > > > > > > >> > > > exceptions.
> > > > > > > >> > > > > >> So,
> > > > > > > >> > > > > >> > I
> > > > > > > >> > > > > >> > > > > > propose
> > > > > > > >> > > > > >> > > > > > > option 1.2. Unless someone has
> > > > > > > suggestion/use-cases
> > > > > > > >> > for
> > > > > > > >> > > > > >> > propagating
> > > > > > > >> > > > > >> > > > > > > exceptions. Please let me know.
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > 2. Intercepting record CRC and record
> > > size
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > Since we decided not to add any
> > > > intermediate
> > > > > > > >> callbacks
> > > > > > > >> > > > (such
> > > > > > > >> > > > > >> as
> > > > > > > >> > > > > >> > > > > onEnqueue
> > > > > > > >> > > > > >> > > > > > > or onReceive) to interceptors, I
> think
> > it
> > > > is
> > > > > > > still
> > > > > > > >> > > > valuable
> > > > > > > >> > > > > to
> > > > > > > >> > > > > >> > > > > intercept
> > > > > > > >> > > > > >> > > > > > > record CRC and record size in bytes
> for
> > > > > > > monitoring
> > > > > > > >> and
> > > > > > > >> > > > audit
> > > > > > > >> > > > > >> > > > use-cases.
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > I propose to add checksum and size
> > fields
> > > > to
> > > > > > > >> > > > RecordMetadata
> > > > > > > >> > > > > >> and
> > > > > > > >> > > > > >> > > > > > > ConsumerRecord. Another option would
> be
> > > to
> > > > > add
> > > > > > > >> them as
> > > > > > > >> > > > > >> parameters
> > > > > > > >> > > > > >> > > in
> > > > > > > >> > > > > >> > > > > > > onAcknowledgement() and onConsume()
> > > > > callbacks.
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > 3. Callbacks that allow to modify
> > records
> > > > > look
> > > > > > as
> > > > > > > >> > > follows:
> > > > > > > >> > > > > >> > > > > > > ProducerRecord<K, V>
> > > > onSend(ProducerRecord<K,
> > > > > > V>
> > > > > > > >> > > record);
> > > > > > > >> > > > > >> > > > > > > ConsumerRecords<K, V>
> > > > > > > onConsume(ConsumerRecords<K,
> > > > > > > >> V>
> > > > > > > >> > > > > >> records);
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > This means that interceptors can
> > > > potentially
> > > > > > > modify
> > > > > > > >> > > > > >> > topic/partition
> > > > > > > >> > > > > >> > > > in
> > > > > > > >> > > > > >> > > > > > > ProducerRecord and
> > topic/partition/offset
> > > > in
> > > > > > > >> > > > > ConsumerRecord. I
> > > > > > > >> > > > > >> > > > propose
> > > > > > > >> > > > > >> > > > > > that
> > > > > > > >> > > > > >> > > > > > > it is up to the interceptor
> > > implementation
> > > > to
> > > > > > > >> ensure
> > > > > > > >> > > that
> > > > > > > >> > > > > >> > > > > > topic/partition,
> > > > > > > >> > > > > >> > > > > > > etc is correct. KafkaProducer.send()
> > will
> > > > use
> > > > > > > >> topic,
> > > > > > > >> > > > > >> partition,
> > > > > > > >> > > > > >> > > key,
> > > > > > > >> > > > > >> > > > > and
> > > > > > > >> > > > > >> > > > > > > value from ProducerRecord returned
> from
> > > the
> > > > > > > >> onSend().
> > > > > > > >> > > > > >> Similarly,
> > > > > > > >> > > > > >> > > > > > > ConsumerRecords returned from
> > > > > > > KafkaConsumer.poll()
> > > > > > > >> > would
> > > > > > > >> > > > be
> > > > > > > >> > > > > >> the
> > > > > > > >> > > > > >> > > ones
> > > > > > > >> > > > > >> > > > > > > returned from the interceptor.
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > Please let me know if you have any
> > > > > suggestions
> > > > > > or
> > > > > > > >> > > > objections
> > > > > > > >> > > > > >> to
> > > > > > > >> > > > > >> > the
> > > > > > > >> > > > > >> > > > > > above.
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > > Anna
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > On Wed, Jan 27, 2016 at 2:56 PM, Anna
> > > > > Povzner <
> > > > > > > >> > > > > >> anna@confluent.io
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> > > > > wrote:
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > > Hi Mayuresh,
> > > > > > > >> > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > > I see why you would want to check
> for
> > > > > > messages
> > > > > > > >> left
> > > > > > > >> > in
> > > > > > > >> > > > the
> > > > > > > >> > > > > >> > > > > > > > RecordAccumulator. However, I don't
> > > think
> > > > > > this
> > > > > > > >> will
> > > > > > > >> > > > > >> completely
> > > > > > > >> > > > > >> > > > solve
> > > > > > > >> > > > > >> > > > > > the
> > > > > > > >> > > > > >> > > > > > > > problem. Messages could be
> in-flight
> > > > > > somewhere
> > > > > > > >> else,
> > > > > > > >> > > > like
> > > > > > > >> > > > > in
> > > > > > > >> > > > > >> > the
> > > > > > > >> > > > > >> > > > > > socket,
> > > > > > > >> > > > > >> > > > > > > or
> > > > > > > >> > > > > >> > > > > > > > there maybe in-flight messages on
> the
> > > > > > consumer
> > > > > > > >> side
> > > > > > > >> > of
> > > > > > > >> > > > the
> > > > > > > >> > > > > >> > > > > MirrorMaker.
> > > > > > > >> > > > > >> > > > > > > So,
> > > > > > > >> > > > > >> > > > > > > > if we go the route of checking
> > whether
> > > > > there
> > > > > > > are
> > > > > > > >> any
> > > > > > > >> > > > > >> in-flight
> > > > > > > >> > > > > >> > > > > messages
> > > > > > > >> > > > > >> > > > > > > for
> > > > > > > >> > > > > >> > > > > > > > topic deletion use-case, maybe it
> is
> > > > better
> > > > > > > count
> > > > > > > >> > them
> > > > > > > >> > > > > with
> > > > > > > >> > > > > >> > > > onSend()
> > > > > > > >> > > > > >> > > > > > and
> > > > > > > >> > > > > >> > > > > > > > onAcknowledge() -- whether all
> > messages
> > > > > sent
> > > > > > > were
> > > > > > > >> > > > > >> > acknowledged. I
> > > > > > > >> > > > > >> > > > > also
> > > > > > > >> > > > > >> > > > > > > > think that it would be better to
> > solve
> > > > this
> > > > > > > >> without
> > > > > > > >> > > > > >> > interceptors,
> > > > > > > >> > > > > >> > > > > such
> > > > > > > >> > > > > >> > > > > > as
> > > > > > > >> > > > > >> > > > > > > > fix error handling in this
> scenario.
> > > > > > However, I
> > > > > > > >> do
> > > > > > > >> > not
> > > > > > > >> > > > > have
> > > > > > > >> > > > > >> any
> > > > > > > >> > > > > >> > > > good
> > > > > > > >> > > > > >> > > > > > > > proposal right now, so these are
> just
> > > > > general
> > > > > > > >> > > thoughts.
> > > > > > > >> > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > > > Anna
> > > > > > > >> > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > > On Wed, Jan 27, 2016 at 11:18 AM,
> > > > Mayuresh
> > > > > > > >> Gharat <
> > > > > > > >> > > > > >> > > > > > > > gharatmayuresh15@gmail.com> wrote:
> > > > > > > >> > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> Calling producer.flush(), flushes
> > all
> > > > the
> > > > > > > data.
> > > > > > > >> So
> > > > > > > >> > > this
> > > > > > > >> > > > > is
> > > > > > > >> > > > > >> OK.
> > > > > > > >> > > > > >> > > But
> > > > > > > >> > > > > >> > > > > > when
> > > > > > > >> > > > > >> > > > > > > >> you
> > > > > > > >> > > > > >> > > > > > > >> are running Mirror maker, I am not
> > > sure
> > > > > > there
> > > > > > > >> is a
> > > > > > > >> > > way
> > > > > > > >> > > > to
> > > > > > > >> > > > > >> > > flush()
> > > > > > > >> > > > > >> > > > > from
> > > > > > > >> > > > > >> > > > > > > >> outside.
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > > >> Thanks,
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > > >> Mayuresh
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > > >> On Wed, Jan 27, 2016 at 11:08 AM,
> > > Becket
> > > > > > Qin <
> > > > > > > >> > > > > >> > > > becket.qin@gmail.com>
> > > > > > > >> > > > > >> > > > > > > >> wrote:
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > > >> > Mayuresh,
> > > > > > > >> > > > > >> > > > > > > >> >
> > > > > > > >> > > > > >> > > > > > > >> > Regarding your use case about
> > mirror
> > > > > > maker.
> > > > > > > >> Is it
> > > > > > > >> > > > good
> > > > > > > >> > > > > >> > enough
> > > > > > > >> > > > > >> > > as
> > > > > > > >> > > > > >> > > > > > long
> > > > > > > >> > > > > >> > > > > > > >> as we
> > > > > > > >> > > > > >> > > > > > > >> > know there is no message for the
> > > topic
> > > > > in
> > > > > > > the
> > > > > > > >> > > > producer
> > > > > > > >> > > > > >> > > anymore?
> > > > > > > >> > > > > >> > > > If
> > > > > > > >> > > > > >> > > > > > > that
> > > > > > > >> > > > > >> > > > > > > >> is
> > > > > > > >> > > > > >> > > > > > > >> > the case, call producer.flush()
> is
> > > > > > > sufficient.
> > > > > > > >> > > > > >> > > > > > > >> >
> > > > > > > >> > > > > >> > > > > > > >> > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> >
> > > > > > > >> > > > > >> > > > > > > >> > Jiangjie (Becket) Qin
> > > > > > > >> > > > > >> > > > > > > >> >
> > > > > > > >> > > > > >> > > > > > > >> > On Tue, Jan 26, 2016 at 6:18 PM,
> > > > > Mayuresh
> > > > > > > >> Gharat
> > > > > > > >> > <
> > > > > > > >> > > > > >> > > > > > > >> > gharatmayuresh15@gmail.com
> > > > > > > >> > > > > >> > > > > > > >> > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> >
> > > > > > > >> > > > > >> > > > > > > >> > > Hi Anna,
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > > Thanks a lot for summarizing
> the
> > > > > > > discussion
> > > > > > > >> on
> > > > > > > >> > > this
> > > > > > > >> > > > > >> kip.
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > > It LGTM.
> > > > > > > >> > > > > >> > > > > > > >> > > This is really nice :
> > > > > > > >> > > > > >> > > > > > > >> > > We decided not to add any
> > > callbacks
> > > > to
> > > > > > > >> producer
> > > > > > > >> > > and
> > > > > > > >> > > > > >> > consumer
> > > > > > > >> > > > > >> > > > > > > >> > > interceptors that will depend
> on
> > > > > > internal
> > > > > > > >> > > > > >> implementation
> > > > > > > >> > > > > >> > as
> > > > > > > >> > > > > >> > > > part
> > > > > > > >> > > > > >> > > > > > of
> > > > > > > >> > > > > >> > > > > > > >> this
> > > > > > > >> > > > > >> > > > > > > >> > > KIP.
> > > > > > > >> > > > > >> > > > > > > >> > > *However, it is possible to
> add
> > > them
> > > > > > later
> > > > > > > >> as
> > > > > > > >> > > part
> > > > > > > >> > > > of
> > > > > > > >> > > > > >> > > another
> > > > > > > >> > > > > >> > > > > KIP
> > > > > > > >> > > > > >> > > > > > if
> > > > > > > >> > > > > >> > > > > > > >> > there
> > > > > > > >> > > > > >> > > > > > > >> > > are good use-cases.*
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > > Do you agree with the use
> case I
> > > > > > explained
> > > > > > > >> > > earlier
> > > > > > > >> > > > > for
> > > > > > > >> > > > > >> > > knowing
> > > > > > > >> > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > > > > >> number
> > > > > > > >> > > > > >> > > > > > > >> > > of records left in the
> > > > > RecordAccumulator
> > > > > > > >> for a
> > > > > > > >> > > > > >> particular
> > > > > > > >> > > > > >> > > > topic.
> > > > > > > >> > > > > >> > > > > > It
> > > > > > > >> > > > > >> > > > > > > >> might
> > > > > > > >> > > > > >> > > > > > > >> > > be orthogonal to this KIP, but
> > > will
> > > > be
> > > > > > > >> helpful.
> > > > > > > >> > > > What
> > > > > > > >> > > > > do
> > > > > > > >> > > > > >> > you
> > > > > > > >> > > > > >> > > > > think?
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > > Mayuresh
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > > On Tue, Jan 26, 2016 at 2:46
> PM,
> > > > Todd
> > > > > > > >> Palino <
> > > > > > > >> > > > > >> > > > tpalino@gmail.com
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> wrote:
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > > > This looks good. As noted,
> > > having
> > > > > one
> > > > > > > >> mutable
> > > > > > > >> > > > > >> > interceptor
> > > > > > > >> > > > > >> > > on
> > > > > > > >> > > > > >> > > > > > each
> > > > > > > >> > > > > >> > > > > > > >> side
> > > > > > > >> > > > > >> > > > > > > >> > > > allows for the use cases we
> > can
> > > > > > envision
> > > > > > > >> > right
> > > > > > > >> > > > now,
> > > > > > > >> > > > > >> and
> > > > > > > >> > > > > >> > I
> > > > > > > >> > > > > >> > > > > think
> > > > > > > >> > > > > >> > > > > > > >> that’s
> > > > > > > >> > > > > >> > > > > > > >> > > > going to provide a great
> deal
> > of
> > > > > > > >> opportunity
> > > > > > > >> > > for
> > > > > > > >> > > > > >> > > > implementing
> > > > > > > >> > > > > >> > > > > > > things
> > > > > > > >> > > > > >> > > > > > > >> > like
> > > > > > > >> > > > > >> > > > > > > >> > > > audit, especially within a
> > > > > > multi-tenant
> > > > > > > >> > > > > environment.
> > > > > > > >> > > > > >> > > Looking
> > > > > > > >> > > > > >> > > > > > > >> forward to
> > > > > > > >> > > > > >> > > > > > > >> > > > getting this available in
> the
> > > > > clients.
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > Thanks!
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > -Todd
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > On Tue, Jan 26, 2016 at 2:36
> > PM,
> > > > > Anna
> > > > > > > >> > Povzner <
> > > > > > > >> > > > > >> > > > > > anna@confluent.io>
> > > > > > > >> > > > > >> > > > > > > >> > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > Hi All,
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > Here is meeting notes from
> > > > today’s
> > > > > > KIP
> > > > > > > >> > > meeting:
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > 1. We agreed to keep the
> > scope
> > > > of
> > > > > > this
> > > > > > > >> KIP
> > > > > > > >> > to
> > > > > > > >> > > > be
> > > > > > > >> > > > > >> > > producer
> > > > > > > >> > > > > >> > > > > and
> > > > > > > >> > > > > >> > > > > > > >> > consumer
> > > > > > > >> > > > > >> > > > > > > >> > > > > interceptors only.
> > Broker-side
> > > > > > > >> interceptor
> > > > > > > >> > > will
> > > > > > > >> > > > > be
> > > > > > > >> > > > > >> > added
> > > > > > > >> > > > > >> > > > > later
> > > > > > > >> > > > > >> > > > > > > as
> > > > > > > >> > > > > >> > > > > > > >> a
> > > > > > > >> > > > > >> > > > > > > >> > > > > separate KIP. The reasons
> > were
> > > > > > already
> > > > > > > >> > > > mentioned
> > > > > > > >> > > > > in
> > > > > > > >> > > > > >> > this
> > > > > > > >> > > > > >> > > > > > thread,
> > > > > > > >> > > > > >> > > > > > > >> but
> > > > > > > >> > > > > >> > > > > > > >> > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > summary is:
> > > > > > > >> > > > > >> > > > > > > >> > > > >  * Broker interceptor is
> > > riskier
> > > > > and
> > > > > > > >> > requires
> > > > > > > >> > > > > >> careful
> > > > > > > >> > > > > >> > > > > > > >> consideration
> > > > > > > >> > > > > >> > > > > > > >> > > about
> > > > > > > >> > > > > >> > > > > > > >> > > > > overheads, whether to
> > > intercept
> > > > > > > leaders
> > > > > > > >> vs.
> > > > > > > >> > > > > >> > > > > leaders/replicas,
> > > > > > > >> > > > > >> > > > > > > >> what to
> > > > > > > >> > > > > >> > > > > > > >> > > do
> > > > > > > >> > > > > >> > > > > > > >> > > > on
> > > > > > > >> > > > > >> > > > > > > >> > > > > leader failover and so on.
> > > > > > > >> > > > > >> > > > > > > >> > > > >  * Broker interceptors
> > > increase
> > > > > > > >> monitoring
> > > > > > > >> > > > > >> resolution,
> > > > > > > >> > > > > >> > > but
> > > > > > > >> > > > > >> > > > > not
> > > > > > > >> > > > > >> > > > > > > >> > > including
> > > > > > > >> > > > > >> > > > > > > >> > > > it
> > > > > > > >> > > > > >> > > > > > > >> > > > > in this KIP does not
> reduce
> > > > > > usefulness
> > > > > > > >> of
> > > > > > > >> > > > > producer
> > > > > > > >> > > > > >> and
> > > > > > > >> > > > > >> > > > > > consumer
> > > > > > > >> > > > > >> > > > > > > >> > > > > interceptors that enable
> > > > > end-to-end
> > > > > > > >> > > monitoring
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > 2. We agreed to scope
> > > > > > > >> ProducerInterceptor
> > > > > > > >> > and
> > > > > > > >> > > > > >> > > > > > > ConsumerInterceptor
> > > > > > > >> > > > > >> > > > > > > >> > > > callbacks
> > > > > > > >> > > > > >> > > > > > > >> > > > > to minimal set of mutable
> > API
> > > > that
> > > > > > are
> > > > > > > >> not
> > > > > > > >> > > > > >> dependent
> > > > > > > >> > > > > >> > on
> > > > > > > >> > > > > >> > > > > > producer
> > > > > > > >> > > > > >> > > > > > > >> and
> > > > > > > >> > > > > >> > > > > > > >> > > > > consumer internal
> > > > implementation.
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > ProducerInterceptor:
> > > > > > > >> > > > > >> > > > > > > >> > > > > *ProducerRecord<K, V>
> > > > > > > >> > > onSend(ProducerRecord<K,
> > > > > > > >> > > > V>
> > > > > > > >> > > > > >> > > > record);*
> > > > > > > >> > > > > >> > > > > > > >> > > > > *void
> > > > > > onAcknowledgement(RecordMetadata
> > > > > > > >> > > > metadata,
> > > > > > > >> > > > > >> > > Exception
> > > > > > > >> > > > > >> > > > > > > >> > exception);*
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > ConsumerInterceptor:
> > > > > > > >> > > > > >> > > > > > > >> > > > > *ConsumerRecords<K, V>
> > > > > > > >> > > > > >> onConsume(ConsumerRecords<K, V>
> > > > > > > >> > > > > >> > > > > > > records);*
> > > > > > > >> > > > > >> > > > > > > >> > > > > *void
> > > > onCommit(Map<TopicPartition,
> > > > > > > >> > > > > >> OffsetAndMetadata>
> > > > > > > >> > > > > >> > > > > > offsets);*
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > We will allow interceptors
> > to
> > > > > modify
> > > > > > > >> > > > > >> ProducerRecord on
> > > > > > > >> > > > > >> > > > > > producer
> > > > > > > >> > > > > >> > > > > > > >> side,
> > > > > > > >> > > > > >> > > > > > > >> > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > modify ConsumerRecords on
> > > > consumer
> > > > > > > side.
> > > > > > > >> > This
> > > > > > > >> > > > > will
> > > > > > > >> > > > > >> > > support
> > > > > > > >> > > > > >> > > > > > > >> end-to-end
> > > > > > > >> > > > > >> > > > > > > >> > > > > monitoring and auditing
> and
> > > > > support
> > > > > > > the
> > > > > > > >> > > ability
> > > > > > > >> > > > > to
> > > > > > > >> > > > > >> add
> > > > > > > >> > > > > >> > > > > > metadata
> > > > > > > >> > > > > >> > > > > > > >> for a
> > > > > > > >> > > > > >> > > > > > > >> > > > > message. This will support
> > > > Todd’s
> > > > > > > >> Auditing
> > > > > > > >> > > and
> > > > > > > >> > > > > >> Routing
> > > > > > > >> > > > > >> > > > > > > use-cases.
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > We did not find any
> use-case
> > > for
> > > > > > > >> modifying
> > > > > > > >> > > > > records
> > > > > > > >> > > > > >> in
> > > > > > > >> > > > > >> > > > > > > onConsume()
> > > > > > > >> > > > > >> > > > > > > >> > > > callback,
> > > > > > > >> > > > > >> > > > > > > >> > > > > but decided to enable
> > > > modification
> > > > > > of
> > > > > > > >> > > consumer
> > > > > > > >> > > > > >> records
> > > > > > > >> > > > > >> > > for
> > > > > > > >> > > > > >> > > > > > > >> symmetry
> > > > > > > >> > > > > >> > > > > > > >> > > with
> > > > > > > >> > > > > >> > > > > > > >> > > > > onSend().
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > 3. We agreed to ensure
> > > > > compatibility
> > > > > > > >> > when/if
> > > > > > > >> > > we
> > > > > > > >> > > > > add
> > > > > > > >> > > > > >> > new
> > > > > > > >> > > > > >> > > > > > methods
> > > > > > > >> > > > > >> > > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > ProducerInterceptor and
> > > > > > > >> ConsumerInterceptor
> > > > > > > >> > > by
> > > > > > > >> > > > > >> using
> > > > > > > >> > > > > >> > > > default
> > > > > > > >> > > > > >> > > > > > > >> methods
> > > > > > > >> > > > > >> > > > > > > >> > > with
> > > > > > > >> > > > > >> > > > > > > >> > > > > an empty implementation.
> Ok
> > to
> > > > > > assume
> > > > > > > >> Java
> > > > > > > >> > 8.
> > > > > > > >> > > > > >> (This is
> > > > > > > >> > > > > >> > > > > > Ismael’s
> > > > > > > >> > > > > >> > > > > > > >> > method
> > > > > > > >> > > > > >> > > > > > > >> > > > #2).
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > 4. We decided not to add
> any
> > > > > > callbacks
> > > > > > > >> to
> > > > > > > >> > > > > producer
> > > > > > > >> > > > > >> and
> > > > > > > >> > > > > >> > > > > > consumer
> > > > > > > >> > > > > >> > > > > > > >> > > > > interceptors that will
> > depend
> > > on
> > > > > > > >> internal
> > > > > > > >> > > > > >> > implementation
> > > > > > > >> > > > > >> > > > as
> > > > > > > >> > > > > >> > > > > > part
> > > > > > > >> > > > > >> > > > > > > >> of
> > > > > > > >> > > > > >> > > > > > > >> > > this
> > > > > > > >> > > > > >> > > > > > > >> > > > > KIP. However, it is
> possible
> > > to
> > > > > add
> > > > > > > them
> > > > > > > >> > > later
> > > > > > > >> > > > as
> > > > > > > >> > > > > >> part
> > > > > > > >> > > > > >> > > of
> > > > > > > >> > > > > >> > > > > > > another
> > > > > > > >> > > > > >> > > > > > > >> KIP
> > > > > > > >> > > > > >> > > > > > > >> > > if
> > > > > > > >> > > > > >> > > > > > > >> > > > > there are good use-cases.
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > *Reasoning.* We did not
> have
> > > > > > concrete
> > > > > > > >> > > use-cases
> > > > > > > >> > > > > >> that
> > > > > > > >> > > > > >> > > > > justified
> > > > > > > >> > > > > >> > > > > > > >> more
> > > > > > > >> > > > > >> > > > > > > >> > > > methods
> > > > > > > >> > > > > >> > > > > > > >> > > > > at this point. Some of the
> > > > > use-cases
> > > > > > > >> were
> > > > > > > >> > for
> > > > > > > >> > > > > more
> > > > > > > >> > > > > >> > > > > fine-grain
> > > > > > > >> > > > > >> > > > > > > >> latency
> > > > > > > >> > > > > >> > > > > > > >> > > > > collection, which could be
> > > done
> > > > > with
> > > > > > > >> Kafka
> > > > > > > >> > > > > Metrics.
> > > > > > > >> > > > > >> > > > Another
> > > > > > > >> > > > > >> > > > > > > >> use-case
> > > > > > > >> > > > > >> > > > > > > >> > > was
> > > > > > > >> > > > > >> > > > > > > >> > > > > encryption. However, there
> > are
> > > > > > several
> > > > > > > >> > design
> > > > > > > >> > > > > >> options
> > > > > > > >> > > > > >> > > for
> > > > > > > >> > > > > >> > > > > > > >> encryption.
> > > > > > > >> > > > > >> > > > > > > >> > > One
> > > > > > > >> > > > > >> > > > > > > >> > > > > is to do per-record
> > encryption
> > > > > which
> > > > > > > >> would
> > > > > > > >> > > > > require
> > > > > > > >> > > > > >> > > adding
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > ProducerInterceptor.onEnqueued()
> > > > > and
> > > > > > > >> > > > > >> > > > > > > >> ConsumerInterceptor.onReceive().
> > > > > > > >> > > > > >> > > > > > > >> > > One
> > > > > > > >> > > > > >> > > > > > > >> > > > > could argue that in that
> > case
> > > > > > > encryption
> > > > > > > >> > > could
> > > > > > > >> > > > be
> > > > > > > >> > > > > >> done
> > > > > > > >> > > > > >> > > by
> > > > > > > >> > > > > >> > > > > > > adding a
> > > > > > > >> > > > > >> > > > > > > >> > > custom
> > > > > > > >> > > > > >> > > > > > > >> > > > > serializer/deserializer.
> > > Another
> > > > > > > option
> > > > > > > >> is
> > > > > > > >> > to
> > > > > > > >> > > > do
> > > > > > > >> > > > > >> > > > encryption
> > > > > > > >> > > > > >> > > > > > > after
> > > > > > > >> > > > > >> > > > > > > >> > > message
> > > > > > > >> > > > > >> > > > > > > >> > > > > gets compressed, but there
> > are
> > > > > > issues
> > > > > > > >> that
> > > > > > > >> > > > arise
> > > > > > > >> > > > > >> > > regarding
> > > > > > > >> > > > > >> > > > > > > broker
> > > > > > > >> > > > > >> > > > > > > >> > doing
> > > > > > > >> > > > > >> > > > > > > >> > > > > re-compression. We decided
> > > that
> > > > it
> > > > > > is
> > > > > > > >> > better
> > > > > > > >> > > to
> > > > > > > >> > > > > >> have
> > > > > > > >> > > > > >> > > that
> > > > > > > >> > > > > >> > > > > > > >> discussion
> > > > > > > >> > > > > >> > > > > > > >> > > in a
> > > > > > > >> > > > > >> > > > > > > >> > > > > separate KIP and decide
> that
> > > > this
> > > > > is
> > > > > > > >> > > something
> > > > > > > >> > > > we
> > > > > > > >> > > > > >> want
> > > > > > > >> > > > > >> > > to
> > > > > > > >> > > > > >> > > > do
> > > > > > > >> > > > > >> > > > > > > with
> > > > > > > >> > > > > >> > > > > > > >> > > > > interceptors or by other
> > > means.
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > Todd, Mayuresh and others
> > who
> > > > > missed
> > > > > > > the
> > > > > > > >> > KIP
> > > > > > > >> > > > > >> meeting,
> > > > > > > >> > > > > >> > > > please
> > > > > > > >> > > > > >> > > > > > let
> > > > > > > >> > > > > >> > > > > > > >> me
> > > > > > > >> > > > > >> > > > > > > >> > > know
> > > > > > > >> > > > > >> > > > > > > >> > > > > your thoughts on the scope
> > we
> > > > > agreed
> > > > > > > on
> > > > > > > >> > > during
> > > > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > meeting.
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > I will update the KIP
> > proposal
> > > > > with
> > > > > > > the
> > > > > > > >> > > current
> > > > > > > >> > > > > >> > decision
> > > > > > > >> > > > > >> > > > by
> > > > > > > >> > > > > >> > > > > > end
> > > > > > > >> > > > > >> > > > > > > of
> > > > > > > >> > > > > >> > > > > > > >> > > today.
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > Anna
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > On Tue, Jan 26, 2016 at
> > 11:41
> > > > AM,
> > > > > > > >> Mayuresh
> > > > > > > >> > > > > Gharat <
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> gharatmayuresh15@gmail.com>
> > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > Hi,
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > I won't be able to make
> it
> > > to
> > > > > KIP
> > > > > > > >> hangout
> > > > > > > >> > > due
> > > > > > > >> > > > > to
> > > > > > > >> > > > > >> > > > conflict.
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > Anna, here is the use
> case
> > > > where
> > > > > > > >> knowing
> > > > > > > >> > if
> > > > > > > >> > > > > there
> > > > > > > >> > > > > >> > are
> > > > > > > >> > > > > >> > > > > > messages
> > > > > > > >> > > > > >> > > > > > > >> in
> > > > > > > >> > > > > >> > > > > > > >> > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > RecordAccumulator left
> to
> > be
> > > > > sent
> > > > > > to
> > > > > > > >> the
> > > > > > > >> > > > kafka
> > > > > > > >> > > > > >> > cluster
> > > > > > > >> > > > > >> > > > > for a
> > > > > > > >> > > > > >> > > > > > > >> topic
> > > > > > > >> > > > > >> > > > > > > >> > is
> > > > > > > >> > > > > >> > > > > > > >> > > > > > useful.
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > 1) Consider a pipeline :
> > > > > > > >> > > > > >> > > > > > > >> > > > > > A ---> Mirror-maker
> > -----> B
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > 2) We have a topic T in
> > > > cluster
> > > > > A
> > > > > > > >> > mirrored
> > > > > > > >> > > to
> > > > > > > >> > > > > >> > cluster
> > > > > > > >> > > > > >> > > B.
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > 3) Now if we delete
> topic
> > T
> > > > in A
> > > > > > and
> > > > > > > >> > > > > immediately
> > > > > > > >> > > > > >> > > proceed
> > > > > > > >> > > > > >> > > > > to
> > > > > > > >> > > > > >> > > > > > > >> delete
> > > > > > > >> > > > > >> > > > > > > >> > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > topic in cluster B, some
> > of
> > > > the
> > > > > > the
> > > > > > > >> > > > > Mirror-maker
> > > > > > > >> > > > > >> > > > machines
> > > > > > > >> > > > > >> > > > > > die
> > > > > > > >> > > > > >> > > > > > > >> > because
> > > > > > > >> > > > > >> > > > > > > >> > > > > > atleast one of the
> batches
> > > in
> > > > > > > >> > > > RecordAccumulator
> > > > > > > >> > > > > >> for
> > > > > > > >> > > > > >> > > > topic
> > > > > > > >> > > > > >> > > > > T
> > > > > > > >> > > > > >> > > > > > > >> fail to
> > > > > > > >> > > > > >> > > > > > > >> > > be
> > > > > > > >> > > > > >> > > > > > > >> > > > > > produced to cluster B.
> We
> > > have
> > > > > > seen
> > > > > > > >> this
> > > > > > > >> > > > > >> happening
> > > > > > > >> > > > > >> > in
> > > > > > > >> > > > > >> > > > our
> > > > > > > >> > > > > >> > > > > > > >> clusters.
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > If we know that there
> are
> > no
> > > > > more
> > > > > > > >> > messages
> > > > > > > >> > > > left
> > > > > > > >> > > > > >> in
> > > > > > > >> > > > > >> > the
> > > > > > > >> > > > > >> > > > > > > >> > > > RecordAccumulator
> > > > > > > >> > > > > >> > > > > > > >> > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > be produced to cluster
> B,
> > we
> > > > can
> > > > > > > >> safely
> > > > > > > >> > > > delete
> > > > > > > >> > > > > >> the
> > > > > > > >> > > > > >> > > topic
> > > > > > > >> > > > > >> > > > > in
> > > > > > > >> > > > > >> > > > > > > >> > cluster B
> > > > > > > >> > > > > >> > > > > > > >> > > > > > without disturbing the
> > > > pipeline.
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > Mayuresh
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > On Tue, Jan 26, 2016 at
> > > 10:31
> > > > > AM,
> > > > > > > Anna
> > > > > > > >> > > > Povzner
> > > > > > > >> > > > > <
> > > > > > > >> > > > > >> > > > > > > >> anna@confluent.io>
> > > > > > > >> > > > > >> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > Thanks Ismael and Todd
> > for
> > > > > your
> > > > > > > >> > feedback!
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > I agree about coming
> up
> > > with
> > > > > > lean,
> > > > > > > >> but
> > > > > > > >> > > > useful
> > > > > > > >> > > > > >> > > > interfaces
> > > > > > > >> > > > > >> > > > > > > that
> > > > > > > >> > > > > >> > > > > > > >> > will
> > > > > > > >> > > > > >> > > > > > > >> > > be
> > > > > > > >> > > > > >> > > > > > > >> > > > > > easy
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > to extend later.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > When we discuss the
> > > minimal
> > > > > set
> > > > > > of
> > > > > > > >> > > producer
> > > > > > > >> > > > > and
> > > > > > > >> > > > > >> > > > consumer
> > > > > > > >> > > > > >> > > > > > > >> > > interceptor
> > > > > > > >> > > > > >> > > > > > > >> > > > > API
> > > > > > > >> > > > > >> > > > > > > >> > > > > > in
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > today’s KIP meeting
> > > > > (discussion
> > > > > > > >> item #2
> > > > > > > >> > > in
> > > > > > > >> > > > my
> > > > > > > >> > > > > >> > > previous
> > > > > > > >> > > > > >> > > > > > > email),
> > > > > > > >> > > > > >> > > > > > > >> > lets
> > > > > > > >> > > > > >> > > > > > > >> > > > > > compare
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > two options:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > *1. Minimal set of
> > > immutable
> > > > > API
> > > > > > > for
> > > > > > > >> > > > producer
> > > > > > > >> > > > > >> and
> > > > > > > >> > > > > >> > > > > consumer
> > > > > > > >> > > > > >> > > > > > > >> > > > > interceptors*
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > ProducerInterceptor:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > public void
> > > > > > > >> onSend(ProducerRecord<K, V>
> > > > > > > >> > > > > >> record);
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > public void
> > > > > > > >> > > > onAcknowledgement(RecordMetadata
> > > > > > > >> > > > > >> > > metadata,
> > > > > > > >> > > > > >> > > > > > > >> Exception
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > exception);
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > ConsumerInterceptor:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > public void
> > > > > > > >> > onConsume(ConsumerRecords<K,
> > > > > > > >> > > V>
> > > > > > > >> > > > > >> > > records);
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > public void
> > > > > > > >> > onCommit(Map<TopicPartition,
> > > > > > > >> > > > > >> > > > > > OffsetAndMetadata>
> > > > > > > >> > > > > >> > > > > > > >> > > offsets);
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > Use-cases:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > — end-to-end
> monitoring;
> > > > > custom
> > > > > > > >> tracing
> > > > > > > >> > > and
> > > > > > > >> > > > > >> > logging
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > *2. Minimal set of
> > mutable
> > > > API
> > > > > > for
> > > > > > > >> > > producer
> > > > > > > >> > > > > and
> > > > > > > >> > > > > >> > > > consumer
> > > > > > > >> > > > > >> > > > > > > >> > > > interceptors*
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > ProducerInterceptor:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > ProducerRecord<K, V>
> > > > > > > >> > > > onSend(ProducerRecord<K,
> > > > > > > >> > > > > >> V>
> > > > > > > >> > > > > >> > > > > record);
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > void
> > > > > > > >> onAcknowledgement(RecordMetadata
> > > > > > > >> > > > > metadata,
> > > > > > > >> > > > > >> > > > > Exception
> > > > > > > >> > > > > >> > > > > > > >> > > exception);
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > ConsumerInterceptor:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > void
> > > > > > onConsume(ConsumerRecords<K,
> > > > > > > V>
> > > > > > > >> > > > > records);
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > void
> > > > > > onCommit(Map<TopicPartition,
> > > > > > > >> > > > > >> > OffsetAndMetadata>
> > > > > > > >> > > > > >> > > > > > > offsets);
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > Additional use-cases
> to
> > > #1:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > — Ability to add
> > metadata
> > > > to a
> > > > > > > >> message
> > > > > > > >> > or
> > > > > > > >> > > > > fill
> > > > > > > >> > > > > >> in
> > > > > > > >> > > > > >> > > > > standard
> > > > > > > >> > > > > >> > > > > > > >> fields
> > > > > > > >> > > > > >> > > > > > > >> > > for
> > > > > > > >> > > > > >> > > > > > > >> > > > > > audit
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > and routing.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > Implications
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > — Partition assignment
> > > will
> > > > be
> > > > > > > done
> > > > > > > >> > based
> > > > > > > >> > > > on
> > > > > > > >> > > > > >> > > modified
> > > > > > > >> > > > > >> > > > > > > >> key/value
> > > > > > > >> > > > > >> > > > > > > >> > > > instead
> > > > > > > >> > > > > >> > > > > > > >> > > > > > of
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > original key/value. If
> > > > > key/value
> > > > > > > >> > > > > >> transformation is
> > > > > > > >> > > > > >> > > not
> > > > > > > >> > > > > >> > > > > > > >> consistent
> > > > > > > >> > > > > >> > > > > > > >> > > > (same
> > > > > > > >> > > > > >> > > > > > > >> > > > > > key
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > and value does not
> > mutate
> > > to
> > > > > the
> > > > > > > >> same,
> > > > > > > >> > > but
> > > > > > > >> > > > > >> > modified,
> > > > > > > >> > > > > >> > > > > > > >> key/value),
> > > > > > > >> > > > > >> > > > > > > >> > > then
> > > > > > > >> > > > > >> > > > > > > >> > > > > log
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > compaction would not
> > work.
> > > > > > > However,
> > > > > > > >> > audit
> > > > > > > >> > > > and
> > > > > > > >> > > > > >> > > routing
> > > > > > > >> > > > > >> > > > > > > >> use-cases
> > > > > > > >> > > > > >> > > > > > > >> > > from
> > > > > > > >> > > > > >> > > > > > > >> > > > > Todd
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > will likely do
> > consistent
> > > > > > > >> > transformation.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > *Additional callbacks
> > > > > > (discussion
> > > > > > > >> item
> > > > > > > >> > #3
> > > > > > > >> > > > in
> > > > > > > >> > > > > my
> > > > > > > >> > > > > >> > > > previous
> > > > > > > >> > > > > >> > > > > > > >> email):*
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > If we want to support
> > > > > > encryption,
> > > > > > > we
> > > > > > > >> > > would
> > > > > > > >> > > > > >> want to
> > > > > > > >> > > > > >> > > be
> > > > > > > >> > > > > >> > > > > able
> > > > > > > >> > > > > >> > > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > modify
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > serialized key/value,
> > > rather
> > > > > > than
> > > > > > > >> key
> > > > > > > >> > and
> > > > > > > >> > > > > value
> > > > > > > >> > > > > >> > > > objects.
> > > > > > > >> > > > > >> > > > > > > This
> > > > > > > >> > > > > >> > > > > > > >> > will
> > > > > > > >> > > > > >> > > > > > > >> > > > add
> > > > > > > >> > > > > >> > > > > > > >> > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > following API to
> > producer
> > > > and
> > > > > > > >> consumer
> > > > > > > >> > > > > >> > interceptors:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > ProducerInterceptor:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > SerializedKeyValue
> > > > > > > >> > > > onEnqueued(TopicPartition
> > > > > > > >> > > > > >> tp,
> > > > > > > >> > > > > >> > > > > > > >> > ProducerRecord<K,
> > > > > > > >> > > > > >> > > > > > > >> > > V>
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > record,
> > SerializedKeyValue
> > > > > > > >> > > > > serializedKeyValue);
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > ConsumerInterceptor:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > SerializedKeyValue
> > > > > > > >> > > onReceive(TopicPartition
> > > > > > > >> > > > > tp,
> > > > > > > >> > > > > >> > > > > > > >> > SerializedKeyValue
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > serializedKeyValue);
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > I am leaning towards
> > > > > > implementing
> > > > > > > >> the
> > > > > > > >> > > > minimal
> > > > > > > >> > > > > >> set
> > > > > > > >> > > > > >> > of
> > > > > > > >> > > > > >> > > > > > > >> immutable or
> > > > > > > >> > > > > >> > > > > > > >> > > > > mutable
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > interfaces, making
> sure
> > > that
> > > > > we
> > > > > > > >> have a
> > > > > > > >> > > > > >> > compatibility
> > > > > > > >> > > > > >> > > > > plan
> > > > > > > >> > > > > >> > > > > > > that
> > > > > > > >> > > > > >> > > > > > > >> > > allows
> > > > > > > >> > > > > >> > > > > > > >> > > > > us
> > > > > > > >> > > > > >> > > > > > > >> > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > add more callbacks in
> > the
> > > > > future
> > > > > > > >> (per
> > > > > > > >> > > > Ismael
> > > > > > > >> > > > > >> > > comment),
> > > > > > > >> > > > > >> > > > > and
> > > > > > > >> > > > > >> > > > > > > add
> > > > > > > >> > > > > >> > > > > > > >> > more
> > > > > > > >> > > > > >> > > > > > > >> > > > > APIs
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > later. E.g., for
> > > encryption
> > > > > > > >> use-case,
> > > > > > > >> > > there
> > > > > > > >> > > > > >> could
> > > > > > > >> > > > > >> > be
> > > > > > > >> > > > > >> > > > an
> > > > > > > >> > > > > >> > > > > > > >> argument
> > > > > > > >> > > > > >> > > > > > > >> > in
> > > > > > > >> > > > > >> > > > > > > >> > > > > doing
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > encryption after
> message
> > > > > > > compression
> > > > > > > >> > vs.
> > > > > > > >> > > > > >> > per-record
> > > > > > > >> > > > > >> > > > > > > encryption
> > > > > > > >> > > > > >> > > > > > > >> > that
> > > > > > > >> > > > > >> > > > > > > >> > > > > could
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > be done using the
> above
> > > > > > additional
> > > > > > > >> API.
> > > > > > > >> > > > There
> > > > > > > >> > > > > >> is
> > > > > > > >> > > > > >> > > also
> > > > > > > >> > > > > >> > > > > more
> > > > > > > >> > > > > >> > > > > > > >> > > > implications
> > > > > > > >> > > > > >> > > > > > > >> > > > > > for
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > every API that
> modifies
> > > > > records:
> > > > > > > >> > > modifying
> > > > > > > >> > > > > >> > > serialized
> > > > > > > >> > > > > >> > > > > > > >> key/value
> > > > > > > >> > > > > >> > > > > > > >> > > will
> > > > > > > >> > > > > >> > > > > > > >> > > > > > again
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > impact partition
> > > assignment
> > > > > (we
> > > > > > > will
> > > > > > > >> > > likely
> > > > > > > >> > > > > do
> > > > > > > >> > > > > >> > that
> > > > > > > >> > > > > >> > > > > after
> > > > > > > >> > > > > >> > > > > > > >> > partition
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > assignment), which may
> > > > impact
> > > > > > log
> > > > > > > >> > > > compaction
> > > > > > > >> > > > > >> and
> > > > > > > >> > > > > >> > > > mirror
> > > > > > > >> > > > > >> > > > > > > maker
> > > > > > > >> > > > > >> > > > > > > >> > > > > > partitioning.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > Anna
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > On Tue, Jan 26, 2016
> at
> > > 7:26
> > > > > AM,
> > > > > > > >> Todd
> > > > > > > >> > > > Palino
> > > > > > > >> > > > > <
> > > > > > > >> > > > > >> > > > > > > >> tpalino@gmail.com>
> > > > > > > >> > > > > >> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > Finally got a chance
> > to
> > > > > take a
> > > > > > > >> look
> > > > > > > >> > at
> > > > > > > >> > > > > this.
> > > > > > > >> > > > > >> I
> > > > > > > >> > > > > >> > > won’t
> > > > > > > >> > > > > >> > > > > be
> > > > > > > >> > > > > >> > > > > > > >> able to
> > > > > > > >> > > > > >> > > > > > > >> > > > make
> > > > > > > >> > > > > >> > > > > > > >> > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > KIP meeting due to a
> > > > > conflict.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > I’m somewhat
> > > disappointed
> > > > in
> > > > > > > this
> > > > > > > >> > > > > proposal. I
> > > > > > > >> > > > > >> > > think
> > > > > > > >> > > > > >> > > > > that
> > > > > > > >> > > > > >> > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > explicit
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > exclusion of
> > > modification
> > > > of
> > > > > > the
> > > > > > > >> > > messages
> > > > > > > >> > > > > is
> > > > > > > >> > > > > >> > > > > > > short-sighted,
> > > > > > > >> > > > > >> > > > > > > >> and
> > > > > > > >> > > > > >> > > > > > > >> > > not
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > accounting for it
> now
> > is
> > > > > going
> > > > > > > to
> > > > > > > >> > bite
> > > > > > > >> > > us
> > > > > > > >> > > > > >> later.
> > > > > > > >> > > > > >> > > > Jay,
> > > > > > > >> > > > > >> > > > > > > aren’t
> > > > > > > >> > > > > >> > > > > > > >> > you
> > > > > > > >> > > > > >> > > > > > > >> > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > one
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > railing against
> public
> > > > > > > interfaces
> > > > > > > >> and
> > > > > > > >> > > how
> > > > > > > >> > > > > >> > > difficult
> > > > > > > >> > > > > >> > > > > they
> > > > > > > >> > > > > >> > > > > > > >> are to
> > > > > > > >> > > > > >> > > > > > > >> > > > work
> > > > > > > >> > > > > >> > > > > > > >> > > > > > with
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > when you don’t get
> > them
> > > > > right?
> > > > > > > The
> > > > > > > >> > > > “simple”
> > > > > > > >> > > > > >> > change
> > > > > > > >> > > > > >> > > > to
> > > > > > > >> > > > > >> > > > > > one
> > > > > > > >> > > > > >> > > > > > > of
> > > > > > > >> > > > > >> > > > > > > >> > > these
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > interfaces to make
> it
> > > able
> > > > > to
> > > > > > > >> return
> > > > > > > >> > a
> > > > > > > >> > > > > >> record is
> > > > > > > >> > > > > >> > > > going
> > > > > > > >> > > > > >> > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> be a
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > significant
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > change and is going
> to
> > > > > require
> > > > > > > all
> > > > > > > >> > > > clients
> > > > > > > >> > > > > to
> > > > > > > >> > > > > >> > > > rewrite
> > > > > > > >> > > > > >> > > > > > > their
> > > > > > > >> > > > > >> > > > > > > >> > > > > > interceptors.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > If we’re not willing
> > to
> > > > put
> > > > > > the
> > > > > > > >> time
> > > > > > > >> > to
> > > > > > > >> > > > > think
> > > > > > > >> > > > > >> > > > through
> > > > > > > >> > > > > >> > > > > > > >> > > manipulation
> > > > > > > >> > > > > >> > > > > > > >> > > > > now,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > then this KIP should
> > be
> > > > > > shelved
> > > > > > > >> until
> > > > > > > >> > > we
> > > > > > > >> > > > > are.
> > > > > > > >> > > > > >> > > > > > Implementing
> > > > > > > >> > > > > >> > > > > > > >> > > > something
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > halfway is going to
> be
> > > > worse
> > > > > > > than
> > > > > > > >> > > taking
> > > > > > > >> > > > a
> > > > > > > >> > > > > >> > little
> > > > > > > >> > > > > >> > > > > > longer.
> > > > > > > >> > > > > >> > > > > > > In
> > > > > > > >> > > > > >> > > > > > > >> > > > > addition,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > I
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > don’t believe that
> > > > > > manipulation
> > > > > > > >> > > requires
> > > > > > > >> > > > > >> > anything
> > > > > > > >> > > > > >> > > > more
> > > > > > > >> > > > > >> > > > > > > than
> > > > > > > >> > > > > >> > > > > > > >> > > > > > interceptors
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > receive the full
> > record,
> > > > and
> > > > > > > then
> > > > > > > >> to
> > > > > > > >> > > > return
> > > > > > > >> > > > > >> it.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > There are 3 use
> case I
> > > can
> > > > > > think
> > > > > > > >> of
> > > > > > > >> > > right
> > > > > > > >> > > > > now
> > > > > > > >> > > > > >> > > > without
> > > > > > > >> > > > > >> > > > > > any
> > > > > > > >> > > > > >> > > > > > > >> deep
> > > > > > > >> > > > > >> > > > > > > >> > > > > > discussion
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > that can make use of
> > > > > > > interceptors
> > > > > > > >> > with
> > > > > > > >> > > > > >> > > modification:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > 1. Auditing. The
> > ability
> > > > to
> > > > > > add
> > > > > > > >> > > metadata
> > > > > > > >> > > > > to a
> > > > > > > >> > > > > >> > > > message
> > > > > > > >> > > > > >> > > > > > for
> > > > > > > >> > > > > >> > > > > > > >> > > auditing
> > > > > > > >> > > > > >> > > > > > > >> > > > is
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > critical. Hostname,
> > > > service
> > > > > > > name,
> > > > > > > >> > > > > timestamps,
> > > > > > > >> > > > > >> > etc.
> > > > > > > >> > > > > >> > > > are
> > > > > > > >> > > > > >> > > > > > all
> > > > > > > >> > > > > >> > > > > > > >> > pieces
> > > > > > > >> > > > > >> > > > > > > >> > > > of
> > > > > > > >> > > > > >> > > > > > > >> > > > > > data
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > that can be used on
> > the
> > > > > other
> > > > > > > >> side of
> > > > > > > >> > > the
> > > > > > > >> > > > > >> > pipeline
> > > > > > > >> > > > > >> > > > to
> > > > > > > >> > > > > >> > > > > > > >> > categorize
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > messages,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > determine loss and
> > > > transport
> > > > > > > time,
> > > > > > > >> > and
> > > > > > > >> > > > pin
> > > > > > > >> > > > > >> down
> > > > > > > >> > > > > >> > > > > issues.
> > > > > > > >> > > > > >> > > > > > > You
> > > > > > > >> > > > > >> > > > > > > >> may
> > > > > > > >> > > > > >> > > > > > > >> > > say
> > > > > > > >> > > > > >> > > > > > > >> > > > > > that
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > these things can
> just
> > be
> > > > > part
> > > > > > of
> > > > > > > >> the
> > > > > > > >> > > > > message
> > > > > > > >> > > > > >> > > schema,
> > > > > > > >> > > > > >> > > > > but
> > > > > > > >> > > > > >> > > > > > > >> anyone
> > > > > > > >> > > > > >> > > > > > > >> > > who
> > > > > > > >> > > > > >> > > > > > > >> > > > > has
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > worked with a
> > multi-user
> > > > > data
> > > > > > > >> system
> > > > > > > >> > > > > >> (especially
> > > > > > > >> > > > > >> > > > those
> > > > > > > >> > > > > >> > > > > > who
> > > > > > > >> > > > > >> > > > > > > >> have
> > > > > > > >> > > > > >> > > > > > > >> > > > been
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > involved with
> > LinkedIn)
> > > > know
> > > > > > how
> > > > > > > >> > > > difficult
> > > > > > > >> > > > > >> it is
> > > > > > > >> > > > > >> > > to
> > > > > > > >> > > > > >> > > > > > > maintain
> > > > > > > >> > > > > >> > > > > > > >> > > > > consistent
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > message schemas and
> to
> > > get
> > > > > > other
> > > > > > > >> > people
> > > > > > > >> > > > to
> > > > > > > >> > > > > >> put
> > > > > > > >> > > > > >> > in
> > > > > > > >> > > > > >> > > > > fields
> > > > > > > >> > > > > >> > > > > > > for
> > > > > > > >> > > > > >> > > > > > > >> > your
> > > > > > > >> > > > > >> > > > > > > >> > > > > use.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > 2. Encryption. This
> is
> > > > > > probably
> > > > > > > >> the
> > > > > > > >> > > most
> > > > > > > >> > > > > >> obvious
> > > > > > > >> > > > > >> > > > case
> > > > > > > >> > > > > >> > > > > > for
> > > > > > > >> > > > > >> > > > > > > >> > record
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > manipulation on both
> > > > sides.
> > > > > > The
> > > > > > > >> > ability
> > > > > > > >> > > > to
> > > > > > > >> > > > > >> tie
> > > > > > > >> > > > > >> > in
> > > > > > > >> > > > > >> > > > end
> > > > > > > >> > > > > >> > > > > to
> > > > > > > >> > > > > >> > > > > > > end
> > > > > > > >> > > > > >> > > > > > > >> > > > > encryption
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > is
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > important for data
> > that
> > > > > > requires
> > > > > > > >> > > external
> > > > > > > >> > > > > >> > > compliance
> > > > > > > >> > > > > >> > > > > > (PCI,
> > > > > > > >> > > > > >> > > > > > > >> > HIPAA,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > etc.).
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > 3. Routing. By being
> > > able
> > > > to
> > > > > > > add a
> > > > > > > >> > bit
> > > > > > > >> > > of
> > > > > > > >> > > > > >> > > > information
> > > > > > > >> > > > > >> > > > > > > about
> > > > > > > >> > > > > >> > > > > > > >> the
> > > > > > > >> > > > > >> > > > > > > >> > > > > source
> > > > > > > >> > > > > >> > > > > > > >> > > > > > or
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > destination of a
> > message
> > > > to
> > > > > > the
> > > > > > > >> > > metadata,
> > > > > > > >> > > > > you
> > > > > > > >> > > > > >> > can
> > > > > > > >> > > > > >> > > > > easily
> > > > > > > >> > > > > >> > > > > > > >> > > construct
> > > > > > > >> > > > > >> > > > > > > >> > > > an
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > intelligent mirror
> > maker
> > > > > that
> > > > > > > can
> > > > > > > >> > > prevent
> > > > > > > >> > > > > >> loops.
> > > > > > > >> > > > > >> > > > This
> > > > > > > >> > > > > >> > > > > > has
> > > > > > > >> > > > > >> > > > > > > >> the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > opportunity
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > to result in
> > significant
> > > > > > > >> operational
> > > > > > > >> > > > > >> savings, as
> > > > > > > >> > > > > >> > > you
> > > > > > > >> > > > > >> > > > > can
> > > > > > > >> > > > > >> > > > > > > get
> > > > > > > >> > > > > >> > > > > > > >> > rid
> > > > > > > >> > > > > >> > > > > > > >> > > of
> > > > > > > >> > > > > >> > > > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > need for tiered
> > clusters
> > > > in
> > > > > > > order
> > > > > > > >> to
> > > > > > > >> > > > > prevent
> > > > > > > >> > > > > >> > loops
> > > > > > > >> > > > > >> > > > in
> > > > > > > >> > > > > >> > > > > > > >> mirroring
> > > > > > > >> > > > > >> > > > > > > >> > > > > > messages.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > All three of these
> > share
> > > > the
> > > > > > > >> feature
> > > > > > > >> > > that
> > > > > > > >> > > > > >> they
> > > > > > > >> > > > > >> > add
> > > > > > > >> > > > > >> > > > > > > metadata
> > > > > > > >> > > > > >> > > > > > > >> to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > messages.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > With the pushback on
> > > > having
> > > > > > > >> arbitrary
> > > > > > > >> > > > > >> metadata
> > > > > > > >> > > > > >> > as
> > > > > > > >> > > > > >> > > an
> > > > > > > >> > > > > >> > > > > > > >> “envelope”
> > > > > > > >> > > > > >> > > > > > > >> > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > message, this is a
> way
> > > to
> > > > > > > provide
> > > > > > > >> it
> > > > > > > >> > > and
> > > > > > > >> > > > > >> make it
> > > > > > > >> > > > > >> > > the
> > > > > > > >> > > > > >> > > > > > > >> > > responsibility
> > > > > > > >> > > > > >> > > > > > > >> > > > > of
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > client, and not the
> > > Kafka
> > > > > > broker
> > > > > > > >> and
> > > > > > > >> > > > system
> > > > > > > >> > > > > >> > > itself.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > -Todd
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > On Tue, Jan 26, 2016
> > at
> > > > 2:30
> > > > > > AM,
> > > > > > > >> > Ismael
> > > > > > > >> > > > > Juma
> > > > > > > >> > > > > >> <
> > > > > > > >> > > > > >> > > > > > > >> > ismael@juma.me.uk>
> > > > > > > >> > > > > >> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > Hi Anna and Neha,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > I think it makes a
> > lot
> > > > of
> > > > > > > sense
> > > > > > > >> to
> > > > > > > >> > > try
> > > > > > > >> > > > > and
> > > > > > > >> > > > > >> > keep
> > > > > > > >> > > > > >> > > > the
> > > > > > > >> > > > > >> > > > > > > >> interface
> > > > > > > >> > > > > >> > > > > > > >> > > > lean
> > > > > > > >> > > > > >> > > > > > > >> > > > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > add more methods
> > later
> > > > > > when/if
> > > > > > > >> > there
> > > > > > > >> > > > is a
> > > > > > > >> > > > > >> > need.
> > > > > > > >> > > > > >> > > > What
> > > > > > > >> > > > > >> > > > > > is
> > > > > > > >> > > > > >> > > > > > > >> the
> > > > > > > >> > > > > >> > > > > > > >> > > > current
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > thinking with
> > regards
> > > to
> > > > > > > >> > > compatibility
> > > > > > > >> > > > > >> when/if
> > > > > > > >> > > > > >> > > we
> > > > > > > >> > > > > >> > > > > add
> > > > > > > >> > > > > >> > > > > > > new
> > > > > > > >> > > > > >> > > > > > > >> > > > methods?
> > > > > > > >> > > > > >> > > > > > > >> > > > > A
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > few
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > options come to
> > mind:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > 1. Change the
> > > interface
> > > > to
> > > > > > an
> > > > > > > >> > > abstract
> > > > > > > >> > > > > >> class
> > > > > > > >> > > > > >> > > with
> > > > > > > >> > > > > >> > > > > > empty
> > > > > > > >> > > > > >> > > > > > > >> > > > > > implementations
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > for
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > all the methods.
> > This
> > > > > means
> > > > > > > that
> > > > > > > >> > the
> > > > > > > >> > > > path
> > > > > > > >> > > > > >> to
> > > > > > > >> > > > > >> > > > adding
> > > > > > > >> > > > > >> > > > > > new
> > > > > > > >> > > > > >> > > > > > > >> > methods
> > > > > > > >> > > > > >> > > > > > > >> > > > is
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > clear.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > 2. Hope we have
> > moved
> > > to
> > > > > > Java
> > > > > > > 8
> > > > > > > >> by
> > > > > > > >> > > the
> > > > > > > >> > > > > >> time we
> > > > > > > >> > > > > >> > > > need
> > > > > > > >> > > > > >> > > > > to
> > > > > > > >> > > > > >> > > > > > > add
> > > > > > > >> > > > > >> > > > > > > >> > new
> > > > > > > >> > > > > >> > > > > > > >> > > > > > methods
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > use default
> methods
> > > with
> > > > > an
> > > > > > > >> empty
> > > > > > > >> > > > > >> > implementation
> > > > > > > >> > > > > >> > > > for
> > > > > > > >> > > > > >> > > > > > any
> > > > > > > >> > > > > >> > > > > > > >> new
> > > > > > > >> > > > > >> > > > > > > >> > > > method
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > (and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > potentially make
> > > > existing
> > > > > > > >> methods
> > > > > > > >> > > > default
> > > > > > > >> > > > > >> > > methods
> > > > > > > >> > > > > >> > > > > too
> > > > > > > >> > > > > >> > > > > > at
> > > > > > > >> > > > > >> > > > > > > >> that
> > > > > > > >> > > > > >> > > > > > > >> > > > point
> > > > > > > >> > > > > >> > > > > > > >> > > > > > for
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > consistency)
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > 3. Introduce a new
> > > > > interface
> > > > > > > >> that
> > > > > > > >> > > > > inherits
> > > > > > > >> > > > > >> > from
> > > > > > > >> > > > > >> > > > the
> > > > > > > >> > > > > >> > > > > > > >> existing
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > Interceptor
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > interface when we
> > need
> > > > to
> > > > > > add
> > > > > > > >> new
> > > > > > > >> > > > > methods.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > Option 1 is the
> > > easiest
> > > > > and
> > > > > > it
> > > > > > > >> also
> > > > > > > >> > > > means
> > > > > > > >> > > > > >> that
> > > > > > > >> > > > > >> > > > > > > interceptor
> > > > > > > >> > > > > >> > > > > > > >> > > users
> > > > > > > >> > > > > >> > > > > > > >> > > > > only
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > need
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > to override the
> > > methods
> > > > > that
> > > > > > > >> they
> > > > > > > >> > are
> > > > > > > >> > > > > >> > interested
> > > > > > > >> > > > > >> > > > > (more
> > > > > > > >> > > > > >> > > > > > > >> useful
> > > > > > > >> > > > > >> > > > > > > >> > > if
> > > > > > > >> > > > > >> > > > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > number
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > of methods grows).
> > The
> > > > > > > downside
> > > > > > > >> is
> > > > > > > >> > > that
> > > > > > > >> > > > > >> > > > interceptor
> > > > > > > >> > > > > >> > > > > > > >> > > > implementations
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > cannot
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > inherit from
> another
> > > > class
> > > > > > (a
> > > > > > > >> > > > > >> straightforward
> > > > > > > >> > > > > >> > > > > > workaround
> > > > > > > >> > > > > >> > > > > > > >> is
> > > > > > > >> > > > > >> > > > > > > >> > to
> > > > > > > >> > > > > >> > > > > > > >> > > > make
> > > > > > > >> > > > > >> > > > > > > >> > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > interceptor a
> > > forwarder
> > > > > that
> > > > > > > >> calls
> > > > > > > >> > > > > another
> > > > > > > >> > > > > >> > > class).
> > > > > > > >> > > > > >> > > > > > Also,
> > > > > > > >> > > > > >> > > > > > > >> our
> > > > > > > >> > > > > >> > > > > > > >> > > > > existing
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > callbacks are
> > > > interfaces,
> > > > > so
> > > > > > > >> seems
> > > > > > > >> > a
> > > > > > > >> > > > bit
> > > > > > > >> > > > > >> > > > > inconsistent.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > Option 2 may be
> the
> > > most
> > > > > > > >> appealing
> > > > > > > >> > > one
> > > > > > > >> > > > as
> > > > > > > >> > > > > >> both
> > > > > > > >> > > > > >> > > > users
> > > > > > > >> > > > > >> > > > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > ourselves
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > retain
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > flexibility. The
> > main
> > > > > > downside
> > > > > > > >> is
> > > > > > > >> > > that
> > > > > > > >> > > > it
> > > > > > > >> > > > > >> > relies
> > > > > > > >> > > > > >> > > > on
> > > > > > > >> > > > > >> > > > > us
> > > > > > > >> > > > > >> > > > > > > >> moving
> > > > > > > >> > > > > >> > > > > > > >> > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > Java
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > 8,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > which may be more
> > > than a
> > > > > > year
> > > > > > > >> away
> > > > > > > >> > > > > >> potentially
> > > > > > > >> > > > > >> > > (if
> > > > > > > >> > > > > >> > > > > we
> > > > > > > >> > > > > >> > > > > > > >> support
> > > > > > > >> > > > > >> > > > > > > >> > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > last
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > 2
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > Java releases).
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > Thoughts?
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > Ismael
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > On Tue, Jan 26,
> 2016
> > > at
> > > > > 4:59
> > > > > > > AM,
> > > > > > > >> > Neha
> > > > > > > >> > > > > >> > Narkhede <
> > > > > > > >> > > > > >> > > > > > > >> > > > neha@confluent.io>
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > Anna,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > I'm also in
> favor
> > of
> > > > > > > including
> > > > > > > >> > just
> > > > > > > >> > > > the
> > > > > > > >> > > > > >> APIs
> > > > > > > >> > > > > >> > > for
> > > > > > > >> > > > > >> > > > > > which
> > > > > > > >> > > > > >> > > > > > > >> we
> > > > > > > >> > > > > >> > > > > > > >> > > have
> > > > > > > >> > > > > >> > > > > > > >> > > > a
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > clear
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > use
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > case. If more
> use
> > > > cases
> > > > > > for
> > > > > > > >> finer
> > > > > > > >> > > > > >> monitoring
> > > > > > > >> > > > > >> > > > show
> > > > > > > >> > > > > >> > > > > up
> > > > > > > >> > > > > >> > > > > > > in
> > > > > > > >> > > > > >> > > > > > > >> the
> > > > > > > >> > > > > >> > > > > > > >> > > > > future,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > we
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > can
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > always update
> the
> > > > > > interface.
> > > > > > > >> > Would
> > > > > > > >> > > > you
> > > > > > > >> > > > > >> > please
> > > > > > > >> > > > > >> > > > > > > highlight
> > > > > > > >> > > > > >> > > > > > > >> in
> > > > > > > >> > > > > >> > > > > > > >> > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > KIP
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > APIs
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > that you think
> we
> > > have
> > > > > an
> > > > > > > >> > immediate
> > > > > > > >> > > > use
> > > > > > > >> > > > > >> for?
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > Joel,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > Broker-side
> > > monitoring
> > > > > > > makes a
> > > > > > > >> > lot
> > > > > > > >> > > of
> > > > > > > >> > > > > >> sense
> > > > > > > >> > > > > >> > in
> > > > > > > >> > > > > >> > > > the
> > > > > > > >> > > > > >> > > > > > > long
> > > > > > > >> > > > > >> > > > > > > >> > term
> > > > > > > >> > > > > >> > > > > > > >> > > > > > though I
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > don't
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > think it is a
> > > > > requirement
> > > > > > > for
> > > > > > > >> > > > > end-to-end
> > > > > > > >> > > > > >> > > > > monitoring.
> > > > > > > >> > > > > >> > > > > > > >> With
> > > > > > > >> > > > > >> > > > > > > >> > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > producer
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > consumer
> > > interceptors,
> > > > > you
> > > > > > > >> have
> > > > > > > >> > the
> > > > > > > >> > > > > >> ability
> > > > > > > >> > > > > >> > to
> > > > > > > >> > > > > >> > > > get
> > > > > > > >> > > > > >> > > > > > > full
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > publish-to-subscribe
> > > > > > > >> end-to-end
> > > > > > > >> > > > > >> monitoring.
> > > > > > > >> > > > > >> > > The
> > > > > > > >> > > > > >> > > > > > broker
> > > > > > > >> > > > > >> > > > > > > >> > > > > interceptor
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > certainly
> improves
> > > the
> > > > > > > >> resolution
> > > > > > > >> > > of
> > > > > > > >> > > > > >> > > monitoring
> > > > > > > >> > > > > >> > > > > but
> > > > > > > >> > > > > >> > > > > > it
> > > > > > > >> > > > > >> > > > > > > >> is
> > > > > > > >> > > > > >> > > > > > > >> > > also
> > > > > > > >> > > > > >> > > > > > > >> > > > a
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > riskier
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > change. I prefer
> > an
> > > > > > > >> incremental
> > > > > > > >> > > > > approach
> > > > > > > >> > > > > >> > over
> > > > > > > >> > > > > >> > > a
> > > > > > > >> > > > > >> > > > > > > big-bang
> > > > > > > >> > > > > >> > > > > > > >> > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > recommend
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > taking
> baby-steps.
> > > > Let's
> > > > > > > first
> > > > > > > >> > make
> > > > > > > >> > > > > sure
> > > > > > > >> > > > > >> the
> > > > > > > >> > > > > >> > > > > > > >> > > producer/consumer
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > interceptors
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > are successful.
> > And
> > > > then
> > > > > > > come
> > > > > > > >> > back
> > > > > > > >> > > > and
> > > > > > > >> > > > > >> add
> > > > > > > >> > > > > >> > the
> > > > > > > >> > > > > >> > > > > > broker
> > > > > > > >> > > > > >> > > > > > > >> > > > interceptor
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > carefully.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > Having said
> that,
> > it
> > > > > would
> > > > > > > be
> > > > > > > >> > great
> > > > > > > >> > > > to
> > > > > > > >> > > > > >> > > > understand
> > > > > > > >> > > > > >> > > > > > your
> > > > > > > >> > > > > >> > > > > > > >> > > proposal
> > > > > > > >> > > > > >> > > > > > > >> > > > > for
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > broker
> interceptor
> > > > > > > >> independently.
> > > > > > > >> > > We
> > > > > > > >> > > > > can
> > > > > > > >> > > > > >> > > either
> > > > > > > >> > > > > >> > > > > add
> > > > > > > >> > > > > >> > > > > > an
> > > > > > > >> > > > > >> > > > > > > >> > > > > interceptor
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > on-append or
> > > > on-commit.
> > > > > If
> > > > > > > >> people
> > > > > > > >> > > > want
> > > > > > > >> > > > > to
> > > > > > > >> > > > > >> > use
> > > > > > > >> > > > > >> > > > this
> > > > > > > >> > > > > >> > > > > > for
> > > > > > > >> > > > > >> > > > > > > >> > > > > monitoring,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > then
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > possibly
> on-commit
> > > > might
> > > > > > be
> > > > > > > >> more
> > > > > > > >> > > > > useful?
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > Neha
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > On Mon, Jan 25,
> > 2016
> > > > at
> > > > > > 6:47
> > > > > > > >> PM,
> > > > > > > >> > > Jay
> > > > > > > >> > > > > >> Kreps <
> > > > > > > >> > > > > >> > > > > > > >> > jay@confluent.io
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > Hey Joel,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > What is the
> > > > interface
> > > > > > you
> > > > > > > >> are
> > > > > > > >> > > > > thinking
> > > > > > > >> > > > > >> of?
> > > > > > > >> > > > > >> > > > > > Something
> > > > > > > >> > > > > >> > > > > > > >> like
> > > > > > > >> > > > > >> > > > > > > >> > > > this:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > >  onAppend(String
> > > > > > topic,
> > > > > > > >> int
> > > > > > > >> > > > > >> partition,
> > > > > > > >> > > > > >> > > > > Records
> > > > > > > >> > > > > >> > > > > > > >> > records,
> > > > > > > >> > > > > >> > > > > > > >> > > > long
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > time)
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > ?
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > One challenge
> > > right
> > > > > now
> > > > > > is
> > > > > > > >> that
> > > > > > > >> > > we
> > > > > > > >> > > > > are
> > > > > > > >> > > > > >> > still
> > > > > > > >> > > > > >> > > > > using
> > > > > > > >> > > > > >> > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > old
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > Message/MessageSet
> > > > > > classes
> > > > > > > >> on
> > > > > > > >> > the
> > > > > > > >> > > > > >> broker
> > > > > > > >> > > > > >> > > which
> > > > > > > >> > > > > >> > > > > I'm
> > > > > > > >> > > > > >> > > > > > > not
> > > > > > > >> > > > > >> > > > > > > >> > sure
> > > > > > > >> > > > > >> > > > > > > >> > > > if
> > > > > > > >> > > > > >> > > > > > > >> > > > > > we'd
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > want
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > support over
> the
> > > > long
> > > > > > haul
> > > > > > > >> but
> > > > > > > >> > it
> > > > > > > >> > > > > >> might be
> > > > > > > >> > > > > >> > > > okay
> > > > > > > >> > > > > >> > > > > > just
> > > > > > > >> > > > > >> > > > > > > >> to
> > > > > > > >> > > > > >> > > > > > > >> > > > create
> > > > > > > >> > > > > >> > > > > > > >> > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > records
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > instance for
> > this
> > > > > > > interface.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > -Jay
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > On Mon, Jan
> 25,
> > > 2016
> > > > > at
> > > > > > > >> 12:37
> > > > > > > >> > PM,
> > > > > > > >> > > > > Joel
> > > > > > > >> > > > > >> > > Koshy <
> > > > > > > >> > > > > >> > > > > > > >> > > > > > jjkoshy.w@gmail.com>
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > I'm
> definitely
> > > in
> > > > > > favor
> > > > > > > of
> > > > > > > >> > > having
> > > > > > > >> > > > > >> such
> > > > > > > >> > > > > >> > > hooks
> > > > > > > >> > > > > >> > > > > in
> > > > > > > >> > > > > >> > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > produce/consume
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > life-cycle.
> > Not
> > > > sure
> > > > > > if
> > > > > > > >> > people
> > > > > > > >> > > > > >> remember
> > > > > > > >> > > > > >> > > this
> > > > > > > >> > > > > >> > > > > but
> > > > > > > >> > > > > >> > > > > > > in
> > > > > > > >> > > > > >> > > > > > > >> > Kafka
> > > > > > > >> > > > > >> > > > > > > >> > > > 0.7
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > this
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > was
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > pretty much
> > how
> > > it
> > > > > > was:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> >
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > >
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> >
> > > > > > > >> > > > > >>
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/0.7/core/src/main/scala/kafka/producer/async/CallbackHandler.scala
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > i.e., we had
> > > > > something
> > > > > > > >> > similar
> > > > > > > >> > > to
> > > > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > > > interceptor
> > > > > > > >> > > > > >> > > > > > > >> > > proposal
> > > > > > > >> > > > > >> > > > > > > >> > > > > for
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > various
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > stages of
> the
> > > > > producer
> > > > > > > >> > request.
> > > > > > > >> > > > The
> > > > > > > >> > > > > >> > > producer
> > > > > > > >> > > > > >> > > > > > > >> provided
> > > > > > > >> > > > > >> > > > > > > >> > > > > > call-backs
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > for
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> beforeEnqueue,
> > > > > > > >> afterEnqueue,
> > > > > > > >> > > > > >> > > afterDequeuing,
> > > > > > > >> > > > > >> > > > > > > >> > > beforeSending,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > etc.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > So
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > at
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > LinkedIn we
> in
> > > > fact
> > > > > > did
> > > > > > > >> > > auditing
> > > > > > > >> > > > > >> within
> > > > > > > >> > > > > >> > > > these
> > > > > > > >> > > > > >> > > > > > > >> > call-backs
> > > > > > > >> > > > > >> > > > > > > >> > > > (and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > not
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > explicitly
> in
> > > the
> > > > > > > >> wrapper).
> > > > > > > >> > > Over
> > > > > > > >> > > > > time
> > > > > > > >> > > > > >> > and
> > > > > > > >> > > > > >> > > > with
> > > > > > > >> > > > > >> > > > > > 0.8
> > > > > > > >> > > > > >> > > > > > > >> we
> > > > > > > >> > > > > >> > > > > > > >> > > moved
> > > > > > > >> > > > > >> > > > > > > >> > > > > > that
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > out
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > wrapper
> > > libraries.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > On a
> side-note
> > > > while
> > > > > > > audit
> > > > > > > >> > and
> > > > > > > >> > > > > other
> > > > > > > >> > > > > >> > > > > monitoring
> > > > > > > >> > > > > >> > > > > > > can
> > > > > > > >> > > > > >> > > > > > > >> be
> > > > > > > >> > > > > >> > > > > > > >> > > done
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > internally
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > in a
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > convenient
> > way I
> > > > > think
> > > > > > > it
> > > > > > > >> > > should
> > > > > > > >> > > > be
> > > > > > > >> > > > > >> > > > clarified
> > > > > > > >> > > > > >> > > > > > that
> > > > > > > >> > > > > >> > > > > > > >> > > having a
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > wrapper
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > is
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > in
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > general not
> a
> > > bad
> > > > > idea
> > > > > > > >> and I
> > > > > > > >> > > > would
> > > > > > > >> > > > > >> even
> > > > > > > >> > > > > >> > > > > consider
> > > > > > > >> > > > > >> > > > > > > it
> > > > > > > >> > > > > >> > > > > > > >> to
> > > > > > > >> > > > > >> > > > > > > >> > > be a
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > best-practice.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > Even with
> 0.7
> > we
> > > > > still
> > > > > > > >> had a
> > > > > > > >> > > > > wrapper
> > > > > > > >> > > > > >> > > library
> > > > > > > >> > > > > >> > > > > and
> > > > > > > >> > > > > >> > > > > > > >> that
> > > > > > > >> > > > > >> > > > > > > >> > API
> > > > > > > >> > > > > >> > > > > > > >> > > > has
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > largely
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > stayed the
> > same
> > > > and
> > > > > > has
> > > > > > > >> > helped
> > > > > > > >> > > > > >> protect
> > > > > > > >> > > > > >> > > > against
> > > > > > > >> > > > > >> > > > > > > >> > (sometimes
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > backwards
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> incompatible)
> > > > > changes
> > > > > > in
> > > > > > > >> open
> > > > > > > >> > > > > source.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > While we are
> > on
> > > > this
> > > > > > > >> topic I
> > > > > > > >> > > have
> > > > > > > >> > > > > one
> > > > > > > >> > > > > >> > > > comment
> > > > > > > >> > > > > >> > > > > > and
> > > > > > > >> > > > > >> > > > > > > >> Anna,
> > > > > > > >> > > > > >> > > > > > > >> > > you
> > > > > > > >> > > > > >> > > > > > > >> > > > > may
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > have
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > already
> > > considered
> > > > > > this
> > > > > > > >> but I
> > > > > > > >> > > > don't
> > > > > > > >> > > > > >> see
> > > > > > > >> > > > > >> > > > > mention
> > > > > > > >> > > > > >> > > > > > of
> > > > > > > >> > > > > >> > > > > > > >> it
> > > > > > > >> > > > > >> > > > > > > >> > in
> > > > > > > >> > > > > >> > > > > > > >> > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > KIP:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > Add a custom
> > > > message
> > > > > > > >> > > > > >> > interceptor/validator
> > > > > > > >> > > > > >> > > > on
> > > > > > > >> > > > > >> > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > broker
> > > > > > > >> > > > > >> > > > > > > >> > > on
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > message
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > arrival.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > We
> decompress
> > > and
> > > > do
> > > > > > > basic
> > > > > > > >> > > > > >> validation of
> > > > > > > >> > > > > >> > > > > > messages
> > > > > > > >> > > > > >> > > > > > > on
> > > > > > > >> > > > > >> > > > > > > >> > > > > arrival. I
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > think
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > there
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > is value in
> > > > > supporting
> > > > > > > >> custom
> > > > > > > >> > > > > >> validation
> > > > > > > >> > > > > >> > > and
> > > > > > > >> > > > > >> > > > > > > expand
> > > > > > > >> > > > > >> > > > > > > >> it
> > > > > > > >> > > > > >> > > > > > > >> > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > support
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > custom
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > on-arrival
> > > > > processing.
> > > > > > > >> Here
> > > > > > > >> > is
> > > > > > > >> > > a
> > > > > > > >> > > > > >> > specific
> > > > > > > >> > > > > >> > > > > > > use-case I
> > > > > > > >> > > > > >> > > > > > > >> > have
> > > > > > > >> > > > > >> > > > > > > >> > > > in
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > mind.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > The
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > blog
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > that James
> > > > > referenced
> > > > > > > >> > describes
> > > > > > > >> > > > our
> > > > > > > >> > > > > >> > > auditing
> > > > > > > >> > > > > >> > > > > > > >> > > > infrastructure.
> > > > > > > >> > > > > >> > > > > > > >> > > > > In
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > order
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > audit the
> > Kafka
> > > > > > cluster
> > > > > > > >> > itself
> > > > > > > >> > > we
> > > > > > > >> > > > > >> need
> > > > > > > >> > > > > >> > to
> > > > > > > >> > > > > >> > > > run
> > > > > > > >> > > > > >> > > > > a
> > > > > > > >> > > > > >> > > > > > > >> > "console
> > > > > > > >> > > > > >> > > > > > > >> > > > > > auditor"
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > service
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > that
> consumes
> > > > > > everything
> > > > > > > >> and
> > > > > > > >> > > > spits
> > > > > > > >> > > > > >> out
> > > > > > > >> > > > > >> > > audit
> > > > > > > >> > > > > >> > > > > > > events
> > > > > > > >> > > > > >> > > > > > > >> > back
> > > > > > > >> > > > > >> > > > > > > >> > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > cluster.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > I
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > would prefer
> > not
> > > > > > having
> > > > > > > to
> > > > > > > >> > run
> > > > > > > >> > > > this
> > > > > > > >> > > > > >> > > service
> > > > > > > >> > > > > >> > > > > > > because:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >    - Well,
> it
> > is
> > > > one
> > > > > > > more
> > > > > > > >> > > service
> > > > > > > >> > > > > >> that
> > > > > > > >> > > > > >> > we
> > > > > > > >> > > > > >> > > > have
> > > > > > > >> > > > > >> > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> run
> > > > > > > >> > > > > >> > > > > > > >> > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > monitor
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >    -
> Consuming
> > > > > > > everything
> > > > > > > >> > takes
> > > > > > > >> > > > up
> > > > > > > >> > > > > >> > > bandwidth
> > > > > > > >> > > > > >> > > > > > which
> > > > > > > >> > > > > >> > > > > > > >> can
> > > > > > > >> > > > > >> > > > > > > >> > be
> > > > > > > >> > > > > >> > > > > > > >> > > > > > avoided
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >    - The
> > console
> > > > > > auditor
> > > > > > > >> > > consumer
> > > > > > > >> > > > > >> itself
> > > > > > > >> > > > > >> > > can
> > > > > > > >> > > > > >> > > > > lag
> > > > > > > >> > > > > >> > > > > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > cause
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > temporary
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > audit
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > discrepancies
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > One way we
> can
> > > > > > mitigate
> > > > > > > >> this
> > > > > > > >> > is
> > > > > > > >> > > > by
> > > > > > > >> > > > > >> > having
> > > > > > > >> > > > > >> > > > > > > >> mirror-makers
> > > > > > > >> > > > > >> > > > > > > >> > > in
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > between
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > clusters
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > emit audit
> > > events.
> > > > > The
> > > > > > > >> > problem
> > > > > > > >> > > is
> > > > > > > >> > > > > >> that
> > > > > > > >> > > > > >> > the
> > > > > > > >> > > > > >> > > > > very
> > > > > > > >> > > > > >> > > > > > > last
> > > > > > > >> > > > > >> > > > > > > >> > > > cluster
> > > > > > > >> > > > > >> > > > > > > >> > > > > in
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > pipeline
> will
> > > not
> > > > > have
> > > > > > > any
> > > > > > > >> > > audit
> > > > > > > >> > > > > >> which
> > > > > > > >> > > > > >> > is
> > > > > > > >> > > > > >> > > > why
> > > > > > > >> > > > > >> > > > > we
> > > > > > > >> > > > > >> > > > > > > >> need
> > > > > > > >> > > > > >> > > > > > > >> > to
> > > > > > > >> > > > > >> > > > > > > >> > > > have
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > something
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > audit the
> > > cluster.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > If we had a
> > > custom
> > > > > > > message
> > > > > > > >> > > > > validator
> > > > > > > >> > > > > >> > then
> > > > > > > >> > > > > >> > > > the
> > > > > > > >> > > > > >> > > > > > > audit
> > > > > > > >> > > > > >> > > > > > > >> can
> > > > > > > >> > > > > >> > > > > > > >> > > be
> > > > > > > >> > > > > >> > > > > > > >> > > > > done
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > on-arrival
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > and we won't
> > > need
> > > > a
> > > > > > > >> console
> > > > > > > >> > > > > auditor.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > One
> potential
> > > > issue
> > > > > in
> > > > > > > >> this
> > > > > > > >> > > > > approach
> > > > > > > >> > > > > >> and
> > > > > > > >> > > > > >> > > any
> > > > > > > >> > > > > >> > > > > > > >> elaborate
> > > > > > > >> > > > > >> > > > > > > >> > > > > > on-arrival
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > processing
> for
> > > > that
> > > > > > > >> matter is
> > > > > > > >> > > > that
> > > > > > > >> > > > > >> you
> > > > > > > >> > > > > >> > may
> > > > > > > >> > > > > >> > > > > need
> > > > > > > >> > > > > >> > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > deserialize
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > message
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > as well
> which
> > > can
> > > > > > drive
> > > > > > > up
> > > > > > > >> > > > produce
> > > > > > > >> > > > > >> > request
> > > > > > > >> > > > > >> > > > > > > handling
> > > > > > > >> > > > > >> > > > > > > >> > > times.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > However
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > I'm
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > not
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > terribly
> > > concerned
> > > > > > about
> > > > > > > >> that
> > > > > > > >> > > > > >> especially
> > > > > > > >> > > > > >> > > if
> > > > > > > >> > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > > > > >> audit
> > > > > > > >> > > > > >> > > > > > > >> > > > header
> > > > > > > >> > > > > >> > > > > > > >> > > > > > can
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > be
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > separated
> out
> > > > easily
> > > > > > or
> > > > > > > >> even
> > > > > > > >> > > > > >> > deserialized
> > > > > > > >> > > > > >> > > > > > > partially
> > > > > > > >> > > > > >> > > > > > > >> as
> > > > > > > >> > > > > >> > > > > > > >> > > this
> > > > > > > >> > > > > >> > > > > > > >> > > > > > Avro
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > thread
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > touches on
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> >
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > >
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> >
> > > > > > > >> > > > > >>
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://search-hadoop.com/m/F2svI1HDLY12W8tnH1&subj=Re+any+optimization+in+reading+a+partial+schema+in+the+decoder+
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > Joel
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > On Mon, Jan
> > 25,
> > > > 2016
> > > > > > at
> > > > > > > >> 12:02
> > > > > > > >> > > PM,
> > > > > > > >> > > > > >> > Mayuresh
> > > > > > > >> > > > > >> > > > > > Gharat
> > > > > > > >> > > > > >> > > > > > > <
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> gharatmayuresh15@gmail.com>
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > Nice KIP.
> > > > > Excellent
> > > > > > > >> idea.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > Was just
> > > > thinking
> > > > > if
> > > > > > > we
> > > > > > > >> can
> > > > > > > >> > > add
> > > > > > > >> > > > > >> > > onDequed()
> > > > > > > >> > > > > >> > > > > to
> > > > > > > >> > > > > >> > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > ProducerIterceptor
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > interface.
> > > Since
> > > > > we
> > > > > > > have
> > > > > > > >> > the
> > > > > > > >> > > > > >> > > onEnqueued(),
> > > > > > > >> > > > > >> > > > > it
> > > > > > > >> > > > > >> > > > > > > will
> > > > > > > >> > > > > >> > > > > > > >> > help
> > > > > > > >> > > > > >> > > > > > > >> > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > client
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > or
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > tools to
> > know
> > > > how
> > > > > > much
> > > > > > > >> time
> > > > > > > >> > > the
> > > > > > > >> > > > > >> > message
> > > > > > > >> > > > > >> > > > > spent
> > > > > > > >> > > > > >> > > > > > in
> > > > > > > >> > > > > >> > > > > > > >> the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> RecordAccumulator.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > Also an
> API
> > to
> > > > > check
> > > > > > > if
> > > > > > > >> > there
> > > > > > > >> > > > are
> > > > > > > >> > > > > >> any
> > > > > > > >> > > > > >> > > > > messages
> > > > > > > >> > > > > >> > > > > > > >> left
> > > > > > > >> > > > > >> > > > > > > >> > > for a
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > particular
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > topic
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > in the
> > > > > > > RecordAccumulator
> > > > > > > >> > > would
> > > > > > > >> > > > > >> help.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > Mayuresh
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > On Mon,
> Jan
> > > 25,
> > > > > 2016
> > > > > > > at
> > > > > > > >> > 11:29
> > > > > > > >> > > > AM,
> > > > > > > >> > > > > >> Todd
> > > > > > > >> > > > > >> > > > > Palino
> > > > > > > >> > > > > >> > > > > > <
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > tpalino@gmail.com
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > Great
> > idea.
> > > > I’ve
> > > > > > > been
> > > > > > > >> > > talking
> > > > > > > >> > > > > >> about
> > > > > > > >> > > > > >> > > this
> > > > > > > >> > > > > >> > > > > > for 2
> > > > > > > >> > > > > >> > > > > > > >> > years,
> > > > > > > >> > > > > >> > > > > > > >> > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > I’m
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > glad
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > someone
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > is
> finally
> > > > > picking
> > > > > > > it
> > > > > > > >> up.
> > > > > > > >> > > > Will
> > > > > > > >> > > > > >> take
> > > > > > > >> > > > > >> > a
> > > > > > > >> > > > > >> > > > look
> > > > > > > >> > > > > >> > > > > > at
> > > > > > > >> > > > > >> > > > > > > >> the
> > > > > > > >> > > > > >> > > > > > > >> > KIP
> > > > > > > >> > > > > >> > > > > > > >> > > > at
> > > > > > > >> > > > > >> > > > > > > >> > > > > > some
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > point
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > shortly.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > -Todd
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > On Mon,
> > Jan
> > > > 25,
> > > > > > 2016
> > > > > > > >> at
> > > > > > > >> > > 11:24
> > > > > > > >> > > > > AM,
> > > > > > > >> > > > > >> > Jay
> > > > > > > >> > > > > >> > > > > Kreps
> > > > > > > >> > > > > >> > > > > > <
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > jay@confluent.io>
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > Hey
> > > Becket,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > Yeah
> > this
> > > is
> > > > > > > really
> > > > > > > >> > > similar
> > > > > > > >> > > > > to
> > > > > > > >> > > > > >> the
> > > > > > > >> > > > > >> > > > > > callback.
> > > > > > > >> > > > > >> > > > > > > >> The
> > > > > > > >> > > > > >> > > > > > > >> > > > > > difference
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > really
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > in
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > who
> sets
> > > the
> > > > > > > >> behavior.
> > > > > > > >> > > The
> > > > > > > >> > > > > >> idea of
> > > > > > > >> > > > > >> > > the
> > > > > > > >> > > > > >> > > > > > > >> > interceptor
> > > > > > > >> > > > > >> > > > > > > >> > > is
> > > > > > > >> > > > > >> > > > > > > >> > > > > > that
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > it
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > doesn't
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > >
> require
> > > any
> > > > > code
> > > > > > > >> change
> > > > > > > >> > > in
> > > > > > > >> > > > > >> apps so
> > > > > > > >> > > > > >> > > you
> > > > > > > >> > > > > >> > > > > can
> > > > > > > >> > > > > >> > > > > > > >> > globally
> > > > > > > >> > > > > >> > > > > > > >> > > > add
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > behavior
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > your
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > Kafka
> > > usage
> > > > > > > without
> > > > > > > >> > > > changing
> > > > > > > >> > > > > >> app
> > > > > > > >> > > > > >> > > code.
> > > > > > > >> > > > > >> > > > > > > Whereas
> > > > > > > >> > > > > >> > > > > > > >> > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > callback
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > added
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > by
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > app.
> The
> > > > idea
> > > > > is
> > > > > > > to
> > > > > > > >> > kind
> > > > > > > >> > > of
> > > > > > > >> > > > > >> > obviate
> > > > > > > >> > > > > >> > > > the
> > > > > > > >> > > > > >> > > > > > need
> > > > > > > >> > > > > >> > > > > > > >> for
> > > > > > > >> > > > > >> > > > > > > >> > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > wrapper
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > code
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > that
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > e.g.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > >
> LinkedIn
> > > > > > maintains
> > > > > > > >> to
> > > > > > > >> > > hold
> > > > > > > >> > > > > this
> > > > > > > >> > > > > >> > kind
> > > > > > > >> > > > > >> > > > of
> > > > > > > >> > > > > >> > > > > > > stuff.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > -Jay
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > On
> Sun,
> > > Jan
> > > > > 24,
> > > > > > > >> 2016 at
> > > > > > > >> > > > 4:21
> > > > > > > >> > > > > >> PM,
> > > > > > > >> > > > > >> > > > Becket
> > > > > > > >> > > > > >> > > > > > Qin
> > > > > > > >> > > > > >> > > > > > > <
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > becket.qin@gmail.com>
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > This
> > > could
> > > > > be
> > > > > > a
> > > > > > > >> > useful
> > > > > > > >> > > > > >> feature.
> > > > > > > >> > > > > >> > > And
> > > > > > > >> > > > > >> > > > I
> > > > > > > >> > > > > >> > > > > > > think
> > > > > > > >> > > > > >> > > > > > > >> > there
> > > > > > > >> > > > > >> > > > > > > >> > > > are
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > some
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > use
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > cases
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> mutate
> > > the
> > > > > > data
> > > > > > > >> like
> > > > > > > >> > > > > rejected
> > > > > > > >> > > > > >> > > > > > alternative
> > > > > > > >> > > > > >> > > > > > > >> one
> > > > > > > >> > > > > >> > > > > > > >> > > > > > mentioned.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > I am
> > > > > wondering
> > > > > > > if
> > > > > > > >> > there
> > > > > > > >> > > > is
> > > > > > > >> > > > > >> > > > functional
> > > > > > > >> > > > > >> > > > > > > >> > overlapping
> > > > > > > >> > > > > >> > > > > > > >> > > > > > between
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > ProducerInterceptor.onAcknowledgement()
> > > > > > > >> > > > > >> > > > > > > and
> > > > > > > >> > > > > >> > > > > > > >> the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > producer
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > callback?
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > I
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > can
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > see
> > that
> > > > the
> > > > > > > >> Callback
> > > > > > > >> > > > could
> > > > > > > >> > > > > >> be a
> > > > > > > >> > > > > >> > > per
> > > > > > > >> > > > > >> > > > > > > record
> > > > > > > >> > > > > >> > > > > > > >> > > setting
> > > > > > > >> > > > > >> > > > > > > >> > > > > > while
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> onAcknowledgement()
> > > > > > > >> > is
> > > > > > > >> > > a
> > > > > > > >> > > > > >> > producer
> > > > > > > >> > > > > >> > > > > level
> > > > > > > >> > > > > >> > > > > > > >> > setting.
> > > > > > > >> > > > > >> > > > > > > >> > > > > Other
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > than
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > that,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > is
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > there
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > any
> > > > > difference
> > > > > > > >> > between
> > > > > > > >> > > > > them?
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > Jiangjie
> > > > > > > (Becket)
> > > > > > > >> Qin
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > On
> > Fri,
> > > > Jan
> > > > > > 22,
> > > > > > > >> 2016
> > > > > > > >> > at
> > > > > > > >> > > > > 6:21
> > > > > > > >> > > > > >> PM,
> > > > > > > >> > > > > >> > > > Neha
> > > > > > > >> > > > > >> > > > > > > >> Narkhede
> > > > > > > >> > > > > >> > > > > > > >> > <
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > neha@confluent.io
> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > James,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> That
> > > is
> > > > > one
> > > > > > of
> > > > > > > >> the
> > > > > > > >> > > many
> > > > > > > >> > > > > >> > > monitoring
> > > > > > > >> > > > > >> > > > > use
> > > > > > > >> > > > > >> > > > > > > >> cases
> > > > > > > >> > > > > >> > > > > > > >> > > for
> > > > > > > >> > > > > >> > > > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > interceptor
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > > interface.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> Neha
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > On
> > > Fri,
> > > > > Jan
> > > > > > > 22,
> > > > > > > >> > 2016
> > > > > > > >> > > at
> > > > > > > >> > > > > >> 6:18
> > > > > > > >> > > > > >> > PM,
> > > > > > > >> > > > > >> > > > > James
> > > > > > > >> > > > > >> > > > > > > >> Cheng
> > > > > > > >> > > > > >> > > > > > > >> > <
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > jcheng@tivo.com
> >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > Anna,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > I'm
> > > > > trying
> > > > > > > to
> > > > > > > >> > > > > understand
> > > > > > > >> > > > > >> a
> > > > > > > >> > > > > >> > > > > concrete
> > > > > > > >> > > > > >> > > > > > > use
> > > > > > > >> > > > > >> > > > > > > >> > case.
> > > > > > > >> > > > > >> > > > > > > >> > > > It
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > sounds
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > like
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > interceptors
> > > > > > > >> > could
> > > > > > > >> > > be
> > > > > > > >> > > > > >> used
> > > > > > > >> > > > > >> > to
> > > > > > > >> > > > > >> > > > > > > implement
> > > > > > > >> > > > > >> > > > > > > >> > part
> > > > > > > >> > > > > >> > > > > > > >> > > of
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > LinkedIn's
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > Kafak
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > Audit
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > tool?
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> >
> > > > > > https://engineering.linkedin.com/kafka/running-kafka-scale
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > Part
> > > > of
> > > > > > that
> > > > > > > >> is
> > > > > > > >> > > done
> > > > > > > >> > > > > by a
> > > > > > > >> > > > > >> > > > wrapper
> > > > > > > >> > > > > >> > > > > > > >> library
> > > > > > > >> > > > > >> > > > > > > >> > > > around
> > > > > > > >> > > > > >> > > > > > > >> > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > kafka
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > that
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > keeps
> > > > a
> > > > > > > count
> > > > > > > >> of
> > > > > > > >> > > the
> > > > > > > >> > > > > >> number
> > > > > > > >> > > > > >> > of
> > > > > > > >> > > > > >> > > > > > > messages
> > > > > > > >> > > > > >> > > > > > > >> > > > produced,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > then
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > sends
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > that
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > count
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > to a
> > > > > > > >> side-topic.
> > > > > > > >> > It
> > > > > > > >> > > > > >> sounds
> > > > > > > >> > > > > >> > > like
> > > > > > > >> > > > > >> > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > producer
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > interceptors
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > could
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > possibly
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > be
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > used
> > > > to
> > > > > > > >> implement
> > > > > > > >> > > > that?
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > -James
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > On
> > > > Jan
> > > > > > 22,
> > > > > > > >> > 2016,
> > > > > > > >> > > at
> > > > > > > >> > > > > >> 4:33
> > > > > > > >> > > > > >> > PM,
> > > > > > > >> > > > > >> > > > > Anna
> > > > > > > >> > > > > >> > > > > > > >> > Povzner <
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > anna@confluent.io
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > Hi,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > I
> > > > just
> > > > > > > >> created
> > > > > > > >> > a
> > > > > > > >> > > > > KIP-42
> > > > > > > >> > > > > >> > for
> > > > > > > >> > > > > >> > > > > adding
> > > > > > > >> > > > > >> > > > > > > >> > producer
> > > > > > > >> > > > > >> > > > > > > >> > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > consumer
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > >
> > > interceptors
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> for
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > > > > > intercepting
> > > > > > > >> > > > messages
> > > > > > > >> > > > > >> at
> > > > > > > >> > > > > >> > > > > different
> > > > > > > >> > > > > >> > > > > > > >> points
> > > > > > > >> > > > > >> > > > > > > >> > > on
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > producer
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> consumer.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> >
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > >
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> >
> > > > > > > >> > > > > >>
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > > > Comments
> > > > > > > and
> > > > > > > >> > > > > >> suggestions
> > > > > > > >> > > > > >> > are
> > > > > > > >> > > > > >> > > > > > > welcome!
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> >
> > > Anna
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > ________________________________
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > This
> > > > > email
> > > > > > > and
> > > > > > > >> > any
> > > > > > > >> > > > > >> > attachments
> > > > > > > >> > > > > >> > > > may
> > > > > > > >> > > > > >> > > > > > > >> contain
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > confidential
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > >
> > privileged
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > material
> > > > > > for
> > > > > > > >> the
> > > > > > > >> > > sole
> > > > > > > >> > > > > >> use of
> > > > > > > >> > > > > >> > > the
> > > > > > > >> > > > > >> > > > > > > >> intended
> > > > > > > >> > > > > >> > > > > > > >> > > > > > recipient.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > Any
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > review,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > copying,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> or
> > > > > > > >> distribution
> > > > > > > >> > of
> > > > > > > >> > > > this
> > > > > > > >> > > > > >> > email
> > > > > > > >> > > > > >> > > > (or
> > > > > > > >> > > > > >> > > > > > any
> > > > > > > >> > > > > >> > > > > > > >> > > > > attachments)
> > > > > > > >> > > > > >> > > > > > > >> > > > > > by
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > others
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > is
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > prohibited.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> If
> > > you
> > > > > are
> > > > > > > not
> > > > > > > >> > the
> > > > > > > >> > > > > >> intended
> > > > > > > >> > > > > >> > > > > > recipient,
> > > > > > > >> > > > > >> > > > > > > >> > please
> > > > > > > >> > > > > >> > > > > > > >> > > > > > contact
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > sender
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > immediately
> > > > > > > >> and
> > > > > > > >> > > > > >> permanently
> > > > > > > >> > > > > >> > > > delete
> > > > > > > >> > > > > >> > > > > > > this
> > > > > > > >> > > > > >> > > > > > > >> > email
> > > > > > > >> > > > > >> > > > > > > >> > > > and
> > > > > > > >> > > > > >> > > > > > > >> > > > > > any
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> attachments.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > No
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > employee
> > > > > > or
> > > > > > > >> agent
> > > > > > > >> > > of
> > > > > > > >> > > > > TiVo
> > > > > > > >> > > > > >> > Inc.
> > > > > > > >> > > > > >> > > > is
> > > > > > > >> > > > > >> > > > > > > >> > authorized
> > > > > > > >> > > > > >> > > > > > > >> > > to
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > conclude
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > any
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > binding
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > agreement
> > > > > > on
> > > > > > > >> > behalf
> > > > > > > >> > > > of
> > > > > > > >> > > > > >> TiVo
> > > > > > > >> > > > > >> > > Inc.
> > > > > > > >> > > > > >> > > > > by
> > > > > > > >> > > > > >> > > > > > > >> email.
> > > > > > > >> > > > > >> > > > > > > >> > > > > Binding
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > agreements
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > with
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > TiVo
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > Inc.
> > > > may
> > > > > > > only
> > > > > > > >> be
> > > > > > > >> > > made
> > > > > > > >> > > > > by
> > > > > > > >> > > > > >> a
> > > > > > > >> > > > > >> > > > signed
> > > > > > > >> > > > > >> > > > > > > >> written
> > > > > > > >> > > > > >> > > > > > > >> > > > > > agreement.
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > --
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> Neha
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > --
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > *—-*
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > *Todd
> > > Palino*
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > Staff
> Site
> > > > > > > Reliability
> > > > > > > >> > > > Engineer
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > > Data
> > > > > > Infrastructure
> > > > > > > >> > > Streaming
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > linkedin.com/in/toddpalino
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > --
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > -Regards,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > Mayuresh
> R.
> > > > Gharat
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > > (862)
> > 250-7125
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > --
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > Neha
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > --
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > *—-*
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > *Todd Palino*
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > Staff Site
> Reliability
> > > > > > Engineer
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > > Data Infrastructure
> > > > > Streaming
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > linkedin.com/in/toddpalino
> > > > > > > >> > > > > >> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > > > --
> > > > > > > >> > > > > >> > > > > > > >> > > > > > -Regards,
> > > > > > > >> > > > > >> > > > > > > >> > > > > > Mayuresh R. Gharat
> > > > > > > >> > > > > >> > > > > > > >> > > > > > (862) 250-7125
> > > > > > > >> > > > > >> > > > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > >
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > --
> > > > > > > >> > > > > >> > > > > > > >> > > > *—-*
> > > > > > > >> > > > > >> > > > > > > >> > > > *Todd Palino*
> > > > > > > >> > > > > >> > > > > > > >> > > > Staff Site Reliability
> > Engineer
> > > > > > > >> > > > > >> > > > > > > >> > > > Data Infrastructure
> Streaming
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > > > linkedin.com/in/toddpalino
> > > > > > > >> > > > > >> > > > > > > >> > > >
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> > > --
> > > > > > > >> > > > > >> > > > > > > >> > > -Regards,
> > > > > > > >> > > > > >> > > > > > > >> > > Mayuresh R. Gharat
> > > > > > > >> > > > > >> > > > > > > >> > > (862) 250-7125
> > > > > > > >> > > > > >> > > > > > > >> > >
> > > > > > > >> > > > > >> > > > > > > >> >
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > > >> --
> > > > > > > >> > > > > >> > > > > > > >> -Regards,
> > > > > > > >> > > > > >> > > > > > > >> Mayuresh R. Gharat
> > > > > > > >> > > > > >> > > > > > > >> (862) 250-7125
> > > > > > > >> > > > > >> > > > > > > >>
> > > > > > > >> > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > > >
> > > > > > > >> > > > > >> > > > > > >
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > > > --
> > > > > > > >> > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > >> > > > > > Neha
> > > > > > > >> > > > > >> > > > > >
> > > > > > > >> > > > > >> > > > >
> > > > > > > >> > > > > >> > > >
> > > > > > > >> > > > > >> > >
> > > > > > > >> > > > > >> >
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> --
> > > > > > > >> > > > > >> *—-*
> > > > > > > >> > > > > >> *Todd Palino*
> > > > > > > >> > > > > >> Staff Site Reliability Engineer
> > > > > > > >> > > > > >> Data Infrastructure Streaming
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >> linkedin.com/in/toddpalino
> > > > > > > >> > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > --
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Neha
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Neha
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Neha
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message