Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
In-Reply-To: <483538BD-65F9-4060-849C-27530129CBB4@ig.com>
References: <CAJDuW=BA=zvKWTLOA0nyoOHCW5=D-qZuQc_u-9TQ20By6Lf2Zg@mail.gmail.com>
 <126E4021-9FC6-4773-9548-24BEE306B1A6@ig.com> <CAD5tkZZXBxPm4n+mNU+ZLAGJVP8vYRmN54JhsX4GEmeBnMcrzQ@mail.gmail.com>
 <E4120400-7969-4E88-A422-76034DE5E69A@ig.com> <CAJDuW=DNzasT22FpM_0bS8JQm0Zt=mDJugc2hG=54DbKGAUEnw@mail.gmail.com>
 <93e47558e38a4323b05f164da234fab2@BMPRDEXC142.IGI.IG.LOCAL>
 <CAJDuW=DeF4yAUj1pdzvEYFPD+x=EcSeAp-eFUCncqC0wqtiPxQ@mail.gmail.com>
 <1487802174980.8025@ig.com> <CAJDuW=AJb6hzET6AHRxqox5ua+eQ1G51nWAKGnofgDtjUez1Tw@mail.gmail.com>
 <489178A5-189C-4FCA-B0D9-52FF7DCFE801@ig.com> <483538BD-65F9-4060-849C-27530129CBB4@ig.com>
From: Jason Gustafson <jason@confluent.io>
Date: Wed, 22 Feb 2017 16:37:30 -0800
Message-ID: <CAJDuW=Bi4vvgAx78ztqQV2GYXJhKJLcGc+fk-MdLEHCmcXFgEg@mail.gmail.com>
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
To: dev@kafka.apache.org
Content-Type: multipart/alternative; boundary=f403045e1e0087cb14054927d1a2
archived-at: Thu, 23 Feb 2017 00:37:52 -0000

--f403045e1e0087cb14054927d1a2
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

The point about usability is fair. It's also reasonable to expect that
common use cases such as appending headers should be done efficiently.

Perhaps we could compromise with something like this?

class Headers {
 Headers append(Iterable<Header> headers);
 Headers prepend(Iterable<Header> headers);
}

That retains ease of use while still giving ourselves some flexibility in
the implementation.

-Jason


On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <Michael.Pearce@ig.com>
wrote:

> I wasn=E2=80=99t referring to the headers needing to be copied, im meanin=
g the
> fact we=E2=80=99d be forcing a new producer record to be created, with al=
l the
> contents copied.
>
> i.e what will happen is utility method will be created or end up being
> used, which does this, and returns the new ProducerRecord instance.
>
> ProducerRecord  addHeader(ProducerRecord record, Header header){
> Return New ProducerRecord(record.key, record.value, record.timestamp=E2=
=80=A6..,
> record.headers.concat(header))
> }
>
> To me this seems ugly, but will be inevitable if we don=E2=80=99t make ad=
ding
> headers to existing records a simple clean method call.
>
>
>
> On 22/02/2017, 22:57, "Michael Pearce" <Michael.Pearce@ig.com> wrote:
>
>     Lazy init can achieve/avoid that.
>
>     Re the concat, why don=E2=80=99t we implement that inside the Headers=
 rather
> than causing everyone to implement this as adding headers in interceptors
> will be a dominant use case. We want a user friendly API. Having as a use=
r
> having to code this instead of having the headers handle this for me seem=
s
> redundant.
>
>     On 22/02/2017, 22:34, "Jason Gustafson" <jason@confluent.io> wrote:
>
>         I thought the argument was against creating the extra objects
> unnecessarily
>         (i.e. if they were not accessed). And note that making the Header=
s
>         immutable doesn't necessarily mean that they need to be copied:
> you can do
>         a trick like Guava's Iterables.concat to add additional headers
> without
>         changing the underlying collections.
>
>         -Jason
>
>         On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> Michael.Pearce@ig.com>
>         wrote:
>
>         > If the argument for not having a map holding the key, value
> pairs is due
>         > to garbage creation of HashMap entry's, forcing the creation of
> a whole new
>         > producer record to simply add a head, surely is creating a-lot
> more?
>         > ________________________________________
>         > From: Jason Gustafson <jason@confluent.io>
>         > Sent: Wednesday, February 22, 2017 10:09 PM
>         > To: dev@kafka.apache.org
>         > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>         >
>         > The current producer interceptor API is this:
>         >
>         > ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
>         >
>         > So adding a header means creating a new ProducerRecord with a
> new header
>         > added to the current headers and returning it. Would that not
> work?
>         >
>         > -Jason
>         >
>         > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> Michael.Pearce@ig.com>
>         > wrote:
>         >
>         > > So how would you have this work if not mutable where
> interceptors would
>         > > add headers?
>         > >
>         > > Sent using OWA for iPhone
>         > > ________________________________________
>         > > From: Jason Gustafson <jason@confluent.io>
>         > > Sent: Wednesday, February 22, 2017 8:42:27 PM
>         > > To: dev@kafka.apache.org
>         > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>         > >
>         > > I think the point on the mutability of Headers is worth
> discussing a
>         > little
>         > > more. As far as I can tell, once the ProducerRecord (or
> ConsumerRecord)
>         > is
>         > > constructed, there should be no need to further change the
> headers. Is
>         > that
>         > > correct? If so, then why not enforce that that is the case
> through the
>         > API?
>         > > One problem with mutability it that it constrains the
> implementation of
>         > > Headers. For example, if we were backing with a byte slice,
> would we
>         > recopy
>         > > the bytes if a header is added or would we maintain a satelli=
te
>         > collection
>         > > of added records. Seems not great either way. If we really
> think
>         > mutability
>         > > is needed, perhaps we could add a method to headers to conver=
t
> it to a
>         > > mutable type (e.g. a Headers.toMap)?
>         > >
>         > > I'm also with Ismael about exposing Headers.get(). I thought
> it might
>         > make
>         > > sense to have a method like this instead:
>         > >
>         > > Iterable<Header> findMatching(Pattern pattern);
>         > >
>         > > This makes the (potential) need to scan the headers clear in
> the API. I'd
>         > > also be fine exposing no getter at all. In general, =C3=8F th=
ink
> it's good to
>         > > start with a minimalistic API and work from there since it's
> always
>         > easier
>         > > to add methods than remove them.
>         > >
>         > > -Jason
>         > >
>         > > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <
> Michael.Pearce@ig.com>
>         > > wrote:
>         > >
>         > > >
>         > > > Hi Ismael
>         > > >
>         > > > On point 1,
>         > > >
>         > > > Sure makes sense will update shortly.
>         > > >
>         > > > On point 2,
>         > > >
>         > > > Setter/getter typical to properties/headers api=E2=80=99s
> traditionally are map
>         > > > styled interfaces and what I believe is most expected style=
d
> thus the
>         > > Key,
>         > > > Value setter.
>         > > > Also it would mean rather than an interface, we would be
> making our
>         > > > internal header impl object we have for the array, exposed.
> E.g. if we
>         > > had
>         > > > a Map really this would be Map.Entry interface, this is the
> same
>         > reasons
>         > > on
>         > > > the map interface I cannot actually make the underlying Nod=
e
> object
>         > > that=E2=80=99s
>         > > > the implementation for HashMap, so that internals can be
> changed.
>         > > >
>         > > >
>         > > > On point 3,
>         > > >
>         > > > I think it people do expect it to be performant, thus why
> originally
>         > > > concern I raised with using an array for to me is an early
> memory
>         > > > optimisation. I think the user experience of
> properties/headers is on a
>         > > > get/set model. This is why its important we have
> encapsulated logic
>         > that
>         > > > then allows us to change this to a map, if this becomes and
> issue, and
>         > > the
>         > > > memory overhead of hashmap is less so.
>         > > >
>         > > >
>         > > >
>         > > >
>         > > > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of
> Ismael Juma" <
>         > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
>         > > >
>         > > >     Hi all,
>         > > >
>         > > >     Great to see the progress that has been achieved on thi=
s
> one. :) A
>         > > few
>         > > >     comments regarding the APIs (I'm still reviewing the
> message format
>         > > >     changes):
>         > > >
>         > > >     1. Nit: `getHeaders` in `ProducerRecord` and
> `ConsumerRecord`
>         > should
>         > > be
>         > > >     named `headers` (we avoid the `get` prefix in Kafka)
>         > > >
>         > > >     2. The `Headers` class is mutable (there's an `add`
> method). Does
>         > it
>         > > > need
>         > > >     to be? If so, it would be good to explain why. Related
> to that, we
>         > > > should
>         > > >     also explain the thinking around thread-safety. If we
> keep the
>         > `add`
>         > > >     method, it may make sense for it to take a `Header`
> (that way we
>         > can
>         > > > add
>         > > >     things to `Header` without changing the interface).
>         > > >
>         > > >     3. Do we need the `Headers.get()` method? People usuall=
y
> assume
>         > that
>         > > > `get`
>         > > >     would be efficient, but depending on the implementation
> (the
>         > current
>         > > >     proposal states that an array would be used), it may no=
t
> be. If we
>         > > > expect
>         > > >     the number of headers to be small, it doesn't matter
> though.
>         > > >
>         > > >     Ismael
>         > > >
>         > > >     On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
>         > > Michael.Pearce@ig.com
>         > > > >
>         > > >     wrote:
>         > > >
>         > > >     > Hi Jason,
>         > > >     >
>         > > >     > Have converted the interface/api bullets into
> interface code
>         > > > snippets.
>         > > >     >
>         > > >     > Agreed implementation won=E2=80=99t take too long. We=
 have
> early versions
>         > > > already.
>         > > >     > Maybe a week before you think about merging I would
> assume it
>         > would
>         > > > be more
>         > > >     > stabilised? I was thinking then we could fork from
> your confluent
>         > > > branch,
>         > > >     > making and then holding KIP-82 changes in a patch
> file, that we
>         > can
>         > > > then
>         > > >     > re-fork from apache once KIP98 final is merged, and
> apply patch
>         > > with
>         > > > last
>         > > >     > minute changes.
>         > > >     >
>         > > >     > Cheers
>         > > >     > Mike
>         > > >     >
>         > > >     >
>         > > >     > On 22/02/2017, 00:56, "Jason Gustafson" <
> jason@confluent.io>
>         > > wrote:
>         > > >     >
>         > > >     >     Hey Michael,
>         > > >     >
>         > > >     >     Awesome. I have a minor request. The APIs are
> currently
>         > > > documented as a
>         > > >     >     wiki list. Would you mind adding a code snippet
> instead?
>         > It's a
>         > > > bit
>         > > >     > easier
>         > > >     >     to process.
>         > > >     >
>         > > >     >     How will be best to manage this, as we will
> obviously build
>         > off
>         > > > your
>         > > >     > KIP=E2=80=99s
>         > > >     >     > protocol changes, to avoid a merge hell, should
> we branch
>         > > from
>         > > > your
>         > > >     > branch
>         > > >     >     > in the confluent repo or is it worth having a
> KIP-98
>         > special
>         > > > branch
>         > > >     > in the
>         > > >     >     > apache git, that we can branch/fork from?
>         > > >     >
>         > > >     >
>         > > >     >     I was thinking about this also. Ideally we'd like
> to get the
>         > > > changes
>         > > >     > in as
>         > > >     >     close together as possible since we only want one
> magic bump
>         > > and
>         > > > some
>         > > >     > users
>         > > >     >     deploy trunk. The level of effort to change the
> format for
>         > > > headers
>         > > >     > seems
>         > > >     >     not too high. Do you agree? My guess is that the
> KIP-98
>         > message
>         > > > format
>         > > >     >     patch will take 2-3 weeks to review before we
> merge to trunk,
>         > > so
>         > > > you
>         > > >     > could
>         > > >     >     hold off implementing until that patch has somewh=
at
>         > stabilized.
>         > > > That
>         > > >     > would
>         > > >     >     save some potential rebase pain.
>         > > >     >
>         > > >     >     -Jason
>         > > >     >
>         > > >     >
>         > > >     > The information contained in this email is strictly
> confidential
>         > > and
>         > > > for
>         > > >     > the use of the addressee only, unless otherwise
> indicated. If you
>         > > > are not
>         > > >     > the intended recipient, please do not read, copy, use
> or disclose
>         > > to
>         > > > others
>         > > >     > this message or any attachment. Please also notify th=
e
> sender by
>         > > > replying
>         > > >     > to this email or by telephone (+44(020 7896 0011) and
> then delete
>         > > the
>         > > >     > email and any copies of it. Opinions, conclusion (etc=
)
> that do
>         > not
>         > > > relate
>         > > >     > to the official business of this company shall be
> understood as
>         > > > neither
>         > > >     > given nor endorsed by it. IG is a trading name of IG
> Markets
>         > > Limited
>         > > > (a
>         > > >     > company registered in England and Wales, company
> number 04008957)
>         > > > and IG
>         > > >     > Index Limited (a company registered in England and
> Wales, company
>         > > > number
>         > > >     > 01190902). Registered address at Cannon Bridge House,
> 25 Dowgate
>         > > > Hill,
>         > > >     > London EC4R 2YA. Both IG Markets Limited (register
> number 195355)
>         > > > and IG
>         > > >     > Index Limited (register number 114059) are authorised
> and
>         > regulated
>         > > > by the
>         > > >     > Financial Conduct Authority.
>         > > >     >
>         > > >
>         > > >
>         > > > The information contained in this email is strictly
> confidential and
>         > for
>         > > > the use of the addressee only, unless otherwise indicated.
> If you are
>         > not
>         > > > the intended recipient, please do not read, copy, use or
> disclose to
>         > > others
>         > > > this message or any attachment. Please also notify the
> sender by
>         > replying
>         > > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
>         > > email
>         > > > and any copies of it. Opinions, conclusion (etc) that do no=
t
> relate to
>         > > the
>         > > > official business of this company shall be understood as
> neither given
>         > > nor
>         > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a company
>         > > > registered in England and Wales, company number 04008957)
> and IG Index
>         > > > Limited (a company registered in England and Wales, company
> number
>         > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
>         > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
>         > IG
>         > > > Index Limited (register number 114059) are authorised and
> regulated by
>         > > the
>         > > > Financial Conduct Authority.
>         > > >
>         > > The information contained in this email is strictly
> confidential and for
>         > > the use of the addressee only, unless otherwise indicated. If
> you are not
>         > > the intended recipient, please do not read, copy, use or
> disclose to
>         > others
>         > > this message or any attachment. Please also notify the sender
> by replying
>         > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
>         > email
>         > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
>         > the
>         > > official business of this company shall be understood as
> neither given
>         > nor
>         > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>         > > registered in England and Wales, company number 04008957) and
> IG Index
>         > > Limited (a company registered in England and Wales, company
> number
>         > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
>         > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
>         > > Index Limited (register number 114059) are authorised and
> regulated by
>         > the
>         > > Financial Conduct Authority.
>         > >
>         > The information contained in this email is strictly confidentia=
l
> and for
>         > the use of the addressee only, unless otherwise indicated. If
> you are not
>         > the intended recipient, please do not read, copy, use or
> disclose to others
>         > this message or any attachment. Please also notify the sender b=
y
> replying
>         > to this email or by telephone (+44(020 7896 0011) and then
> delete the email
>         > and any copies of it. Opinions, conclusion (etc) that do not
> relate to the
>         > official business of this company shall be understood as neithe=
r
> given nor
>         > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
>         > registered in England and Wales, company number 04008957) and I=
G
> Index
>         > Limited (a company registered in England and Wales, company
> number
>         > 01190902). Registered address at Cannon Bridge House, 25 Dowgat=
e
> Hill,
>         > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
>         > Index Limited (register number 114059) are authorised and
> regulated by the
>         > Financial Conduct Authority.
>         >
>
>
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to othe=
rs
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the ema=
il
> and any copies of it. Opinions, conclusion (etc) that do not relate to th=
e
> official business of this company shall be understood as neither given no=
r
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by th=
e
> Financial Conduct Authority.
>

--f403045e1e0087cb14054927d1a2--