kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5697) StreamThread.shutdown() need to interrupt the stream threads to break the loop
Date Thu, 12 Apr 2018 00:08:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434745#comment-16434745
] 

ASF GitHub Bot commented on KAFKA-5697:
---------------------------------------

vvcephei opened a new pull request #4855: KAFKA-5697: prevent poll() from blocking forever
URL: https://github.com/apache/kafka/pull/4855
 
 
   As the name says, currently `KafkaConsumer#poll` may block forever
   despite having a timeout parameter, since that timeout only applies to
   the `ConsumerNetworkClient#poll` and not the `ConsumerCoordinator#poll`.
   
   This change applies the timeout to the entire `poll` operation, allowing us to
   ensure threads won't hang forever in the event, e.g., that we can't talk to the 
   coordinator. I've tried tomake the change in a 'private' fashion so as not to 
   change any public APIs, but let me know if you think it still needs a KIP.
   
   Several tests depended on being able to send a timeout of 0, but still
   have the coordinator poll take non-zero time to do its work. I updated
   them to send a long enough timeout for the coordinator to to the
   required work.
   
   I believe our testing suite should be sufficient to validate this change.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> StreamThread.shutdown() need to interrupt the stream threads to break the loop
> ------------------------------------------------------------------------------
>
>                 Key: KAFKA-5697
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5697
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: Guozhang Wang
>            Assignee: John Roesler
>            Priority: Major
>              Labels: newbie
>
> In {{StreamThread.shutdown()}} we currently do nothing but set the state, hoping the
stream thread may eventually check it and shutdown itself. However, under certain scenarios
the thread may get blocked within a single loop and hence will never check on this state enum.
For example, it's {{consumer.poll}} call trigger {{ensureCoordinatorReady()}} which will block
until the coordinator can be found. If the coordinator broker is never up and running then
the Stream instance will be blocked forever.
> A simple way to produce this issue is to start the work count demo without starting the
ZK / Kafka broker, and then it will get stuck in a single loop and even `ctrl-C` will not
stop it since its set state will never be read by the thread:
> {code:java}
> [2017-08-03 15:17:39,981] WARN Connection to node -1 could not be established. Broker
may not be available. (org.apache.kafka.clients.NetworkClient)
> [2017-08-03 15:17:40,046] WARN Connection to node -1 could not be established. Broker
may not be available. (org.apache.kafka.clients.NetworkClient)
> [2017-08-03 15:17:40,101] WARN Connection to node -1 could not be established. Broker
may not be available. (org.apache.kafka.clients.NetworkClient)
> [2017-08-03 15:17:40,206] WARN Connection to node -1 could not be established. Broker
may not be available. (org.apache.kafka.clients.NetworkClient)
> [2017-08-03 15:17:40,261] WARN Connection to node -1 could not be established. Broker
may not be available. (org.apache.kafka.clients.NetworkClient)
> [2017-08-03 15:17:40,366] WARN Connection to node -1 could not be established. Broker
may not be available. (org.apache.kafka.clients.NetworkClient)
> [2017-08-03 15:17:40,472] WARN Connection to node -1 could not be established. Broker
may not be available. (org.apache.kafka.clients.NetworkClient)
> ^C[2017-08-03 15:17:40,580] WARN Connection to node -1 could not be established. Broker
may not be available. (org.apache.kafka.clients.NetworkClient)
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message