kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Magesh Nandakumar <mage...@confluent.io>
Subject Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics
Date Wed, 29 Aug 2018 03:53:01 GMT
I was wondering if it would be much simpler to just do a pass-through so
that we can support any topic setting added in Kafka without any code
changes in connect. Since these are for topics that will have the actual
data stream, users might possibly need more flexibility in terms of how the
topics get created.

Thanks
Magesh

On Tue, Aug 28, 2018 at 4:56 PM Randall Hauch <rhauch@gmail.com> wrote:

> Do you think we should support name-value pairs, too?
>
> On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar <mageshn@confluent.io>
> wrote:
>
> > Randall,
> >
> > Thanks a lot for the KIP. I think this would be a great addition for many
> > source connectors.
> > One clarification I had was regarding the topic settings that can be
> > configured. Is it limited to the setting exposed in the TopicSettings
> > interface?
> >
> > Thanks
> > Magesh
> >
> > On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch <rhauch@gmail.com> wrote:
> >
> > > Okay, after much delay let's try this again for AK 2.1. Has anyone
> found
> > > any concerns? Stephane suggested that we allow updating topic
> > > configurations (everything but partition count). I'm unconvinced that
> > it's
> > > worth the additional complexity in the implementation and the
> > documentation
> > > to explain the behavior. Changing several of the topic-specific
> > > configurations have significant impact on broker behavior /
> > functionality,
> > > so IMO we need to proceed more cautiously.
> > >
> > > Stephane, do you have a particular use case in mind for updating topic
> > > configurations on an existing topic?
> > >
> > > Randall
> > >
> > >
> > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <rhauch@gmail.com>
> wrote:
> > >
> > > > The KIP deadline for 1.1 has already passed, but I'd like to restart
> > this
> > > > discussion so that we make the next release. I've not yet addressed
> the
> > > > previous comment about *existing* topics, but I'll try to do that
> over
> > > the
> > > > next few weeks. Any other comments/suggestions/questions?
> > > >
> > > > Best regards,
> > > >
> > > > Randall
> > > >
> > > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rhauch@gmail.com>
> > wrote:
> > > >
> > > >> Oops. Yes, I meant “replication factor”.
> > > >>
> > > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhihong@gmail.com>
wrote:
> > > >> >
> > > >> > Randall:
> > > >> > bq. AdminClient currently allows changing the replication factory.
> > > >> >
> > > >> > By 'replication factory' did you mean 'replication factor' ?
> > > >> >
> > > >> > Cheers
> > > >> >
> > > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rhauch@gmail.com>
> > > >> wrote:
> > > >> >>
> > > >> >> Currently the KIP's scope is only topics that don't yet exist,
> and
> > we
> > > >> have
> > > >> >> to cognizant of race conditions between tasks with the same
> > > connector.
> > > >> I
> > > >> >> think it is worthwhile to consider whether the KIP's scope
should
> > > >> expand to
> > > >> >> also address *existing* partitions, though it may not be
> > appropriate
> > > to
> > > >> >> have as much control when changing the topic settings for
an
> > existing
> > > >> >> topic. For example, changing the number of partitions (which
the
> > KIP
> > > >> >> considers a "topic-specific setting" even though technically
it
> is
> > > not)
> > > >> >> shouldn't be done blindly due to the partitioning impacts,
and
> IIRC
> > > you
> > > >> >> can't reduce them (which we could verify before applying).
Also,
> I
> > > >> don't
> > > >> >> think the AdminClient currently allows changing the replication
> > > >> factory. I
> > > >> >> think changing the topic configs is less problematic both
from
> what
> > > >> makes
> > > >> >> sense for connectors to verify/change and from what the
> AdminClient
> > > >> >> supports.
> > > >> >>
> > > >> >> Even if we decide that it's not appropriate to change the
> settings
> > on
> > > >> an
> > > >> >> existing topic, I do think it's advantageous to at least
notify
> the
> > > >> >> connector (or task) prior to the first record sent to a given
> topic
> > > so
> > > >> that
> > > >> >> the connector can fail or issue a warning if it doesn't meet
its
> > > >> >> requirements.
> > > >> >>
> > > >> >> Best regards,
> > > >> >>
> > > >> >> Randall
> > > >> >>
> > > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek <
> > > >> >> stephane@simplemachines.com.au> wrote:
> > > >> >>
> > > >> >>> Hi Randall,
> > > >> >>>
> > > >> >>> Thanks for the KIP. I like it
> > > >> >>> What happens when the target topic is already created
but the
> > > configs
> > > >> do
> > > >> >>> not match?
> > > >> >>> i.e. wrong RF, num partitions, or missing / additional
configs?
> > Will
> > > >> you
> > > >> >>> attempt to apply the necessary changes or throw an error?
> > > >> >>>
> > > >> >>> Thanks!
> > > >> >>> Stephane
> > > >> >>>
> > > >> >>>
> > > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" <
> > > mathieu.fenniak@replicon.com
> > > >> >
> > > >> >>> wrote:
> > > >> >>>
> > > >> >>>    Ah, yes, I see you a highlighted part that should've
made
> this
> > > >> clear
> > > >> >>>    to me the first read. :-)  Much clearer now!
> > > >> >>>
> > > >> >>>    By the way, enjoyed your Debezium talk in NYC.
> > > >> >>>
> > > >> >>>    Looking forward to this Kafka Connect change; it will
allow
> me
> > to
> > > >> >>>    remove a post-deployment tool that I hacked together
for the
> > > >> purpose
> > > >> >>>    of ensuring auto-created topics have the right config.
> > > >> >>>
> > > >> >>>    Mathieu
> > > >> >>>
> > > >> >>>
> > > >> >>>    On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <
> > > rhauch@gmail.com>
> > > >> >>> wrote:
> > > >> >>>> Thanks for the quick feedback, Mathieu. Yes, the
first
> > > >> >> configuration
> > > >> >>> rule
> > > >> >>>> whose regex matches will be applied, and no other
rules will be
> > > >> >>> used. I've
> > > >> >>>> updated the KIP to try to make this more clear, but
let me know
> > if
> > > >> >>> it's
> > > >> >>>> still not clear.
> > > >> >>>>
> > > >> >>>> Best regards,
> > > >> >>>>
> > > >> >>>> Randall
> > > >> >>>>
> > > >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak
<
> > > >> >>>> mathieu.fenniak@replicon.com> wrote:
> > > >> >>>>
> > > >> >>>>> Hi Randall,
> > > >> >>>>>
> > > >> >>>>> Awesome, very much looking forward to this.
> > > >> >>>>>
> > > >> >>>>> It isn't 100% clear from the KIP how multiple
config-based
> rules
> > > >> >>> would
> > > >> >>>>> be applied; it looks like the first configuration
rule whose
> > regex
> > > >> >>>>> matches the topic name will be used, and no other
rules will
> be
> > > >> >>>>> applied.  Is that correct?  (I wasn't sure if
it might cascade
> > > >> >>>>> together multiple matching rules...)
> > > >> >>>>>
> > > >> >>>>> Looks great,
> > > >> >>>>>
> > > >> >>>>> Mathieu
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch
<
> > rhauch@gmail.com>
> > > >> >>> wrote:
> > > >> >>>>>> Hi, all.
> > > >> >>>>>>
> > > >> >>>>>> We recently added the ability for Kafka Connect
to create
> > > >> >>> *internal*
> > > >> >>>>> topics
> > > >> >>>>>> using the new AdminClient, but it still would
be great if
> Kafka
> > > >> >>> Connect
> > > >> >>>>>> could do this for new topics that result
from source
> connector
> > > >> >>> records.
> > > >> >>>>>> I've outlined an approach to do this in "KIP-158
Kafka
> Connect
> > > >> >>> should
> > > >> >>>>> allow
> > > >> >>>>>> source connectors to set topic-specific settings
for new
> > > >> >> topics".
> > > >> >>>>>>
> > > >> >>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+
> > > >> >>>>> set+topic-specific+settings+for+new+topics
> > > >> >>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+
> > > >> >>>>> set+topic-specific+settings+for+new+topics>*
> > > >> >>>>>>
> > > >> >>>>>> Please take a look and provide feedback.
Thanks!
> > > >> >>>>>>
> > > >> >>>>>> Best regards,
> > > >> >>>>>>
> > > >> >>>>>> Randall
> > > >> >>>>>
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>
> > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message