kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5212) Consumer ListOffsets request can starve group heartbeats
Date Wed, 11 Oct 2017 04:05:02 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199778#comment-16199778

Richard Yu commented on KAFKA-5212:

During some study after the failure of my most recent commit, there is this approach which
one would note:

1. Due to hierarchy design, heartbeat is only accessible by ConsumerCoordinator.
2. In order to pass a variable to Fetcher, ConsumerCoordinator must pass the necessary information
through ConsumerNetworkClient.

One of the easiest ways to is to define a variable with the time of the next heartbeat (e.g.
long nextHeartbeat = now + remainingMs). The variable is defined as a private field of ConsumerNetworkClient
but is defined by ConsumerCoordinator's methods. This way, Fetcher could have access to the
next heartbeat time.

However, this is when we encounter a problem: If fetcher's poll() is concluded (it could not
yield, thus we must wait), and if the time.milliseconds() exceeds nextHeartbeat, we must resend
a heartbeat. However, the poll() method is currently only accessible in ConsumerCoordinator.
Is there a way to get around this?

> Consumer ListOffsets request can starve group heartbeats
> --------------------------------------------------------
>                 Key: KAFKA-5212
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5212
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>            Reporter: Jason Gustafson
>            Assignee: Richard Yu
>             Fix For: 1.1.0, 1.0.1
> The consumer is not able to send heartbeats while it is awaiting a ListOffsets response.
Typically this is not a problem because ListOffsets requests are handled quickly, but in the
worst case if the request takes longer than the session timeout, the consumer will fall out
of the group.

This message was sent by Atlassian JIRA

View raw message