kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "McCaig, Rhys" <Rhys_McC...@comcast.com>
Subject Re: [EXTERNAL] [DISCUSS] KIP-310: Add a Kafka Source Connector to Kafka Connect
Date Wed, 01 Aug 2018 07:32:28 GMT
Hi All,

I’ve updated the proposal to include the improvements suggested by Stephane.

I have also submitted a PR to implement this functionality into Kafka. https://github.com/apache/kafka/pull/5438

I don’t have a benchmark against MirrorMaker yet, as I only currently have a local docker
stack available to me, though I have seen very good performance in that test stack (200k messages/sec@100bytes
on limited compute resource containers). Further benchmarking might take a few days.

Review and comments would be appreciated.

Cheers,
Rhys


On Jun 18, 2018, at 9:00 AM, McCaig, Rhys <Rhys_McCaig@cable.comcast.com<mailto:Rhys_McCaig@cable.comcast.com>>
wrote:

Hi Stephane,

Thanks for your feedback and apologies for the delay in my response.

Are there any performance benchmarks against Mirror Maker available? I'm
interested to know if this is more performant / scalable.
Regarding the implementation, here's some feedback:


Currently I don’t have any performance benchmarks, but I think this is a great idea, ill
see if I can set up something one the next week or so.

- I think it's worth mentioning that this solution does not rely on
consumer groups, and therefore tracking progress may be tricky. Can you
think of a way to expose that?

This is a reasonable concern. I’m not sure how to track this other than looking at the Kafka
connect offsets. Once a messages is passed to the framework, I'm unaware of a way to get at
the commit offsets on the producer side. Any thoughts?

- Some code can be in config Validator I believe:
https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47

- I think your kip mentions `source.admin.` and `source.consumer.` but I
don't see it reflected yet in the code

- Is there a way to be flexible and merge list and regex, or offer the two
simultaneously ? source_topics=my_static_topic,prefix.* ?

Agree on all of the above - I will incorporate into the code later this week as ill get some
time back to work on this.

Cheers,
Rhys



On Jun 6, 2018, at 7:16 PM, Stephane Maarek <stephane@simplemachines.com.au<mailto:stephane@simplemachines.com.au>>
wrote:

Hi Rhys,

I think this will be a great addition.

Are there any performance benchmarks against Mirror Maker available? I'm
interested to know if this is more performant / scalable.
Regarding the implementation, here's some feedback:

- I think it's worth mentioning that this solution does not rely on
consumer groups, and therefore tracking progress may be tricky. Can you
think of a way to expose that?


- Some code can be in config Validator I believe:
https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47

- I think your kip mentions `source.admin.` and `source.consumer.` but I
don't see it reflected yet in the code

- Is there a way to be flexible and merge list and regex, or offer the two
simultaneously ? source_topics=my_static_topic,prefix.* ?

Hope that helps
Stephane

Kind regards,
Stephane

[image: Simple Machines]

Stephane Maarek | Developer

+61 416 575 980
stephane@simplemachines.com.au
simplemachines.com.au
Level 2, 145 William Street, Sydney NSW 2010

On 5 June 2018 at 09:04, McCaig, Rhys <Rhys_McCaig@comcast.com> wrote:

Hi All,

As I didn’t get any comment on this KIP and there has since been an
additional 2 KIP’s created numbered 308 since, I'm bumping this and
renaming the KIP to 310 to remove the duplication:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-
310%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect

Let me know if you have any comments or feedback, would love to hear them.

Cheers,
Rhys

On May 28, 2018, at 10:23 PM, McCaig, Rhys <rhys_mccaig@comcast.com>
wrote:

Sorry for the bad link to the KIP, here it is: https://cwiki.apache.org/
confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+
Connector+to+Kafka+Connect

On May 28, 2018, at 10:19 PM, McCaig, Rhys <Rhys_McCaig@comcast.com>
wrote:

Hi All,

I added a KIP to include a Kafka Source Connector with Kafka Connect.
Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
308%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect<htt
ps://cwiki.apache.org/confluence/display/KAFKA/KIP-
308:+Add+a+Kafka+Source+Connector+to+Kafka+Connect>

Looking forward to your feedback and suggestions.

Cheers,
Rhys







Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message