kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandor Murakozi <smurak...@gmail.com>
Subject Producer timeout and retry behavior
Date Tue, 27 Mar 2018 10:26:19 GMT
Hi All,

I have a question about intended producer behavior if the broker is lost -
do I see a bug or the code works as specified?

What I do and see using trunk:

*a) No message send timeout at all if there is no available broker*
- no broker is started
- consoleproducer is started using --broker-list localhost:9092 --topic
test --request-timeout-ms 2000 --property retries=3
- I try to send some messages. The request doesn't time out, I just see an
endless amount of log entries like this:
[2018-03-27 10:04:42,084] WARN [Producer clientId=console-producer]
Connection to node -1 could not be established. Broker may not be
available. (org.apache.kafka.clients.NetworkClient)

I would expect that the producer eventually times out, preferably retrying
sending 3 times.

*b) No retries if the last broker disappears after successful communication*
- broker is started
- consoleproducer is started with the same config as in a)
- 1 message is sent successfully
- broker is stopped - producer starts to add log entries like
[2018-03-27 10:09:11,597] WARN [Producer clientId=console-producer]
Connection to node 0 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
- I try to send another message
- it times out after ~2 seconds
[2018-03-27 10:09:17,442] ERROR Error when sending message to topic test
with key: null, value: 1 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for
test-0: 2018 ms has passed since batch creation plus linger time
- the producer *does not retry* sending

Some debugging have shown that after the connection was broken the producer
will attempt to fetch metadata. As there are no more available brokers this
will not be possible.
In this case it does not retry fetching of the metadata but it considers
the batch expired.

My expectation would be that the producer times out but it retries sending.

I've spent quite some time trying to derive what's the correct behavior
according to KIP-19
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient>
and
jiras/pull requests related to the topic (e.g. KAFKA-2805
<https://issues.apache.org/jira/browse/KAFKA-2805> and KAFKA-3388
<https://issues.apache.org/jira/browse/KAFKA-3388>) but it's still not 100%
clear to me if the code works as intended.

Could someone help me decide this question?

Many thanks in advance,
Sandor

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message