kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Canes Kelly (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-5826) Transactional Producer needs another way to control transactional expiration
Date Sun, 03 Sep 2017 11:44:02 GMT
Canes Kelly created KAFKA-5826:

             Summary: Transactional Producer needs another way to control transactional expiration
                 Key: KAFKA-5826
                 URL: https://issues.apache.org/jira/browse/KAFKA-5826
             Project: Kafka
          Issue Type: Improvement
          Components: clients, core
    Affects Versions:
            Reporter: Canes Kelly
            Priority: Minor

Since transactional producer can send message to several TopicPartitions, it's hard to customize
transaction timeout by the user when starts a transaction.

In current version of kafka, users who want to send message with transaction should set the
timeout configuration which indicates the max time interval between the beginning transaction
time with the expiration time. But it's hard to say how long is the most appropriate time
to wait for expiration. If timeout is too short, normal appending could be abort because of
the large amount of TopicPartitions needed to append. If timeout is too long, consumer must
wait for the LSO updated, which means the decline of performance. So the timeout is hard to
be determined by users.

However, connection expiration time of producer and cluster is easier to determine.So I propose
to try the other way to calculate the timeout of a transaction.
1. Use heartbeat timeout instead of transaction timeout. Like consumer, we can make producer
to send heartbeat to prove it's alive. Then we can use the way of consumer heartbeat to calculate
2. update the timestamp of a transaction when producer sends a ProduceRequest to append message.
Then we can make *latest timestamp + timeout < now * as the timout flag

This message was sent by Atlassian JIRA

View raw message