kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismael Juma <ism...@juma.me.uk>
Subject Re: ProducerRecord/Consumer MetaData/Headers
Date Thu, 22 Sep 2016 08:42:08 GMT
Sorry for the delay, you should have access now.

Ismael

On Thu, Sep 22, 2016 at 8:15 AM, Michael Pearce <Michael.Pearce@ig.com>
wrote:

> Hi again,
>
> Sorry to be nudging this, but it seems I'm still unable to create a page
> in the KIP Proposal area. Just don't want to be forgotten about between all
> the emails.
>
> Cheers
> Mike
> ________________________________________
> From: Michael Pearce <Michael.Pearce@ig.com>
> Sent: Monday, September 19, 2016 6:19 PM
> To: dev@kafka.apache.org
> Subject: Re: ProducerRecord/Consumer MetaData/Headers
>
> Hi Again,
>
> I went to the wiki this afternoon to start writing it up, and seems I
> cannot create a page under the KIP area still. If someone could assist.
>
> Cheers
> Mike
>
>
> On 9/18/16, 7:07 PM, "Michael Pearce" <Michael.Pearce@ig.com> wrote:
>
>     Hi Ismaelj
>
>     Thanks, my wiki user is michael.andre.pearce.
>
>     Re the link thanks again, actually indeed we started off trying to do
> this after we lost the ability to use the key to hold metadata once the
> compaction feature came, but it actually abusing the payload isn't imo a
> great solution, and has some issues that cannot be overcome and stopping us
> from using in some of our data / message flows. As such I think a solution
> in the broker/message/client needs to be made and formalised. Also then an
> ecosystems of tools could rely on such.
>
>     I will add all details in KIP proposal, once I have access.
>
>     Cheers
>     Mike
>
>
>     ________________________________________
>     From: ismaelj@gmail.com <ismaelj@gmail.com> on behalf of Ismael Juma <
> ismael@juma.me.uk>
>     Sent: Sunday, September 18, 2016 9:01:22 AM
>     To: dev@kafka.apache.org
>     Subject: Re: ProducerRecord/Consumer MetaData/Headers
>
>     Hi Mike,
>
>     If you give me your wiki user name, I can give you the required
> permissions
>     to post a KIP. This is definitely a big change and there is no clear
>     consensus if changing the Kafka message format is the right way (it
> would
>     be good not to pay the cost if you don't need it) or if it should be
> done
>     via schemas, for example. Gwen shared some thoughts in the following
>     message:
>
>     http://search-hadoop.com/m/uyzND1OXS8EoGCU2
>
>     Ismael
>
>     On Sun, Sep 18, 2016 at 7:11 AM, Michael Pearce <Michael.Pearce@ig.com
> >
>     wrote:
>
>     > Hi All, (again)
>     >
>     > If it helps the discussion, and almost ready patch implementing this
> is
>     > available here:
>     >
>     > https://github.com/michaelandrepearce/kafka
>     >
>     >
>     > The biggest/most core change is obviously the kafka.message.Message
> object.
>     >
>     >
>     >
>     > Some key bits in this implementation is the server side, and
> submodules
>     > (connect, mirrormaker, streams) all updated to be aware of the new
> “headers”
>     >
>     >
>     >
>     > As a big API change have to use new feature on the client side, you
> use
>     > the ConsumerRecord and ProducerRecord (which now extend the new
> Enhanced
>     > versions) for K,V records without any code changes, to use the
> headers you
>     > use the Enhanced versions HeadersConsumerRecord and
> HeadersProducerRecord.
>     > This was needed to avoid causing code compilation failure just by
>     > upgrading. If the patch were accepted I would imagine it as a way to
>     > transition.
>     >
>     >
>     >
>     > I am guessing this needs a KIP rather than just myself raising a
> JIRA as
>     > fairly substantial api change but unsure whom can raise these so
> assistance
>     > in the process would be gratefully accepted..
>     >
>     >
>     >
>     > Cheers
>     >
>     > Mike
>     >
>     >
>     >
>     >
>     >
>     >
>     > From: Michael Pearce <Michael.Pearce@ig.com>
>     > Date: Saturday, September 17, 2016 at 6:40 AM
>     > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
>     > Subject: ProducerRecord/Consumer MetaData/Headers
>     >
>     > Hi All,
>     >
>     > First of all apologies if this has been previously discussed I have
> just
>     > joined the mail list (I cannot find a JIRA or KIP related nor via
> good old
>     > google search)
>     >
>     > In our company we are looking to replace some of our more traditional
>     > message flows with Kafka.
>     >
>     > One thing we have found lacking though compared with most messaging
>     > systems is the ability to set header/metadata separate from our
> payload. We
>     > did think about the key, but as this is used for compaction we
> cannot have
>     > changing values here which metadata/header values will obviously be.
>     >
>     > e.g. these headers/metadata are useful for audit data or platform
> data
>     > that is not business payload related e.g. storing
>     > the clientId that generated the message, correlation id of a
>     > request/response, cluster id where the message was generate (case for
>     > MirrorMakers), message uuid etc this list is endless.
>     >
>     > We would like to propose extending the Record from
>     > ProducerRecord/ConsumerRecord<K, V> to
> ProducerRecord/ConsumerRecord<K,V,M>
>     > where M is metadata/header again being like the key and value a
> simple
>     > byte[] so that it is completely upto the end users how to serialize /
>     > deserialize it.
>     >
>     > What our people’s thoughts?
>     > Any other ideas how to add headers/metadata.
>     >
>     > How can I progress this?
>     >
>     > Cheers
>     > Mike
>     > The information contained in this email is strictly confidential and
> for
>     > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > the intended recipient, please do not read, copy, use or disclose to
> others
>     > this message or any attachment. Please also notify the sender by
> replying
>     > to this email or by telephone (+44(020 7896 0011) and then delete
> the
>     > email and any copies of it. Opinions, conclusion (etc) that do not
> relate
>     > to the official business of this company shall be understood as
> neither
>     > given nor endorsed by it. IG is a trading name of IG Markets Limited
> (a
>     > company registered in England and Wales, company number 04008957)
> and IG
>     > Index Limited (a company registered in England and Wales, company
> number
>     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > Index Limited (register number 114059) are authorised and regulated
> by the
>     > Financial Conduct Authority.
>     >
>     The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message