kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Tan <c...@confluent.io>
Subject Re: [DISCUSS] KIP-601: Configurable socket connection timeout
Date Fri, 15 May 2020 19:21:05 GMT
Dear Rajini,


Thanks for the reply. 

> e have a lot of these and I want to
> understand the benefits of the proposed timeout in this case alone. We
> currently have a request timeout of 30s. Would you consider adding a 10s
> connection timeout? 

A shorter timeout (10s) at the transportation level will help clients detect dead nodes faster.
“request.timeout.ms” is too general and applies to all the requests whose complexity at
the application level varies.  It’s risky to set “request.timeout.ms” to a lower value
for detecting dead nodes quicker because of the involvement of the application layer.

After “socket.connection.setup.timeout.ms” hits, NetworkClient will fail the request in
the exact approach as it handles “request.timeout.ms”. That is to say, the response will
constructed upon a RetriableException. Producer, Consumer, and KafkaAdminClient can then perform
their retry logic as a request timeout happens.

> We have KIP-612 that is proposing to throttle connection set up on the one
> hand and this KIP that is dramatically reducing default connection timeout
> on the other. Not sure if that is a good idea.

The default of the broker connection creation rate limit is Int.MaxValue. The KIP also proposes
per-IP throttle configuration. Thus, I don’t expect the combination of the broker connection
throttle and a shorter client transportation timeout will have a side effect.

Does the reasons above make sense to you? 

Best, - Cheng




> On May 15, 2020, at 4:49 AM, Rajini Sivaram <rajinisivaram@gmail.com> wrote:
> 
> Hi Cheng,
> 
> Let me rephrase my question. Let's say we didn't have the case of
> leastLoadedNode. We are only talking about connections to a specific node
> (i.e. leader or controller). We have a lot of these and I want to
> understand the benefits of the proposed timeout in this case alone. We
> currently have a request timeout of 30s. Would you consider adding a 10s
> connection timeout? And if you did, what would you expect the 10s timeout
> to do?
> 
> a) We could fail a request if connection didn't complete within 10s. If we
> always expect connections to succeed within 10s, this would be considered
> reasonable behaviour. But this would be changing the current default, which
> allows you up to 30 seconds to connect and process a request.
> b) We retry the connection. What would be the point? We were waiting in a
> queue for connecting, but we decide to stop and join the back of the queue.
> 
> We have KIP-612 that is proposing to throttle connection set up on the one
> hand and this KIP that is dramatically reducing default connection timeout
> on the other. Not sure if that is a good idea.
> 
> 
> On Fri, May 15, 2020 at 1:26 AM Cheng Tan <ctan@confluent.io> wrote:


Mime
View raw message