kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: [DISCUSS] KIP-238: Expose Kafka cluster ID in Connect REST API
Date Tue, 12 Dec 2017 04:21:20 GMT
On Mon, Dec 11, 2017 at 4:01 PM, Gwen Shapira <gwen@confluent.io> wrote:

> Thanks, Ewen :)
>
> One thing that wasn't clear to me from the wiki: Will standalone connect
> also have a Kafka cluster ID? While it is true that only tasks have
> producers and consumers, I think we assumed that all tasks on one
> stand-alone will use one Kafka cluster?
>

Yeah, maybe not clear enough in the KIP, but this is what I was getting at
-- while I think it's possible to use different clusters for worker,
producer, and consumer, I don't think this is really expected or a use case
worth bending backwards to support perfectly. In standalone mode,
technically a value is not required because a default is included and we
only utilize the value currently for the producers/consumers in tasks. But
I don't think it is unreasonable to require a valid setting at the worker
level, even if you override the bootstrap.servers for producer and consumer.


>
> Another suggestion is not to block the REST API on the connection, but
> rather not return the cluster ID until we know it (return null instead). So
> clients will need to poll rather than block. Not sure this is better, but
> you didn't really discuss this, so wanted to raise the option.
>

It's mentioned briefly in
https://cwiki.apache.org/confluence/display/KAFKA/KIP-238%3A+Expose+Kafka+cluster+ID+in+Connect+REST+API#KIP-238:ExposeKafkaclusterIDinConnectRESTAPI-ProposedChanges
I think the tradeoff of blocking the server from being "started" until we
can at least make one request to the cluster isn't unreasonable since if
you can't do that, you're not going to be able to do any useful work
anyway. Anyone who might otherwise be using this endpoint to monitor health
(which it is useful for since it doesn't require any other external
services to be running just to give a response) can just interpret
connection refused or timeouts as an unhealthy state, as they should anyway.

-Ewen


>
> Gwen
>
>
> On Mon, Dec 11, 2017 at 3:42 PM Ewen Cheslack-Postava <ewen@confluent.io>
> wrote:
>
> > I'd like to start discussion on a simple KIP to expose Kafka cluster ID
> > info in the Connect REST API:
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 238%3A+Expose+Kafka+cluster+ID+in+Connect+REST+API
> >
> > Hopefully straightforward, though there are some details on how this
> > affects startup behavior that might warrant discussion.
> >
> > -Ewen
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message