kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renuka M <renumetuk...@gmail.com>
Subject Re: [ DISCUSS ] KIP-512:Adding headers to RecordMetaData
Date Wed, 28 Aug 2019 00:06:53 GMT
Hi  Gwen/Team

Can you please review the KIP. Hope we have clarified the question you have
regarding proposal.

Thanks
Renuka M

On Mon, Aug 26, 2019 at 3:35 PM Renuka M <renumetukuru@gmail.com> wrote:

> Hi Eric,
>
> We thought about that but we didn't find the strong  enough reason for
> having record itself in Acknowledgement.
> Headers are supposed to carry metadata and that is the reason headers are
> added to producer/consumer records.
> Also we feel having headers information in record metadata is good enough
> to bridge the gap and link the record to its metadata.
> Its simple change since we are not adding any new method signatures.
> Adding new method signatures requires adoption and deprecation of old ones
> to reduce duplication.
> If we get enough votes on adding new method signature, we are open to add
> it.
>
> Thanks
> Renuka M
>
> On Mon, Aug 26, 2019 at 10:54 AM Eric Azama <eazama001@gmail.com> wrote:
>
>> Have you considered adding a new onAcknowledgement method to the
>> ProducerInterceptor with the signature onAcknowledgement(RecordMetadata
>> metadata, Exception exception, ProducerRecord record)? I would also
>> consider adding this to Producer Callbacks as well, since linking a
>> Callback to a specific record currently requires creating a new Callback
>> for every ProducerRecord sent.
>>
>> This seems like a more robust strategy compared to using Headers. Headers
>> don't necessarily contain anything that connects them to the original
>> ProducerRecord, and forcibly including information in the Headers seems
>> like unnecessary bloat. If your goal is to link a RecordMetadata to a
>> specific ProducerRecord, it seems simpler to make sure the original
>> ProducerRecord is accessible at the same time as the RecordMetadata
>>
>> On Mon, Aug 26, 2019 at 10:26 AM Renuka M <renumetukuru@gmail.com> wrote:
>>
>> > Hi Gwen,
>> >
>> > 1.We are not doing any changes on the broker side. This change is only
>> on
>> > Kafka clients library.
>> > 2. RecordMetaData is created by client library while appending record to
>> > ProducerBatch where offset alone returned by broker. Here we are adding
>> > headers to RecordMetaData while creating FutureRecordMetaData to create
>> > context between record and its metadata. I have updated the snippet in
>> KIP
>> > proposed changes in step 3.
>> > 3. As we mentioned in alternatives, client side we can link record and
>> its
>> > metadata using callback, but Interceptors having same RecordMetadata
>> will
>> > not have context on for which record this MetaData belongs to. To fill
>> that
>> > Gap, we are proposing these changes.
>> > Please let us know if we are not clear.
>> >
>> > Thanks
>> > Renuka M
>> >
>> >
>> >
>> >
>> > On Fri, Aug 23, 2019 at 7:08 PM Gwen Shapira <gwen@confluent.io> wrote:
>> >
>> > > I am afraid I don't understand the proposal. The RecordMetadata is
>> > > information returned from the broker regarding the record. The
>> > > producer already has the record (including the headers), so why would
>> > > the broker need to send the headers back as part of the metadata?
>> > >
>> > > On Fri, Aug 23, 2019 at 4:22 PM Renuka M <renumetukuru@gmail.com>
>> wrote:
>> > > >
>> > > > Hi All,
>> > > >
>> > > > I am starting this thread to discuss
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3AAdding+headers+to+RecordMetaData
>> > > > .
>> > > >
>> > > > Please provide the feedback.
>> > > >
>> > > > Thanks
>> > > > Renuka M
>> > >
>> > >
>> > >
>> > > --
>> > > Gwen Shapira
>> > > Product Manager | Confluent
>> > > 650.450.2760 | @gwenshap
>> > > Follow us: Twitter | blog
>> > >
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message