kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5154) Kafka Streams throws NPE during rebalance
Date Tue, 02 May 2017 18:00:05 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993408#comment-15993408
] 

Matthias J. Sax commented on KAFKA-5154:
----------------------------------------

Thanks for reporting this [~Lukas Gemela]. I just double checked the code: https://github.com/apache/kafka/blob/0.10.2.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L618-L619

A {{NullPointerException}} can only occur if {{task}} is {{null}} (if {{records}} would be
{{null}} this would have happened in line 617 already). Thus, it seems, that there is no active
task for a record. Not sure how this could happen atm. What triggers your rebalance? How many
topics do you process and what it the number of partitions per topic?

> Kafka Streams throws NPE during rebalance
> -----------------------------------------
>
>                 Key: KAFKA-5154
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5154
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 0.10.2.0
>            Reporter: Lukas Gemela
>
> please see attached log, Kafka streams throws NullPointerException during rebalance,
which is caught by our custom exception handler
> {noformat}
> 2017-04-30T17:44:17,675 INFO  kafka-coordinator-heartbeat-thread | hades org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
@618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) dead for group
hades
> 2017-04-30T17:44:27,395 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess()
@573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) for group hades.
> 2017-04-30T17:44:27,941 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare()
@393 - Revoking previously assigned partitions [poseidonIncidentFeed-27, poseidonIncidentFeed-29,
poseidonIncidentFeed-30, poseidonIncidentFeed-18] for group hades
> 2017-04-30T17:44:27,947 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
@407 - (Re-)joining group hades
> 2017-04-30T17:44:48,468 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
@407 - (Re-)joining group hades
> 2017-04-30T17:44:53,628 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
@407 - (Re-)joining group hades
> 2017-04-30T17:45:09,587 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
@407 - (Re-)joining group hades
> 2017-04-30T17:45:11,961 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess()
@375 - Successfully joined group hades with generation 99
> 2017-04-30T17:45:13,126 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete()
@252 - Setting newly assigned partitions [poseidonIncidentFeed-11, poseidonIncidentFeed-27,
poseidonIncidentFeed-25, poseidonIncidentFeed-29, poseidonIncidentFeed-19, poseidonIncidentFeed-18]
for group hades
> 2017-04-30T17:46:37,254 INFO  kafka-coordinator-heartbeat-thread | hades org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
@618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) dead for group
hades
> 2017-04-30T18:04:25,993 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess()
@573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) for group hades.
> 2017-04-30T18:04:29,401 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare()
@393 - Revoking previously assigned partitions [poseidonIncidentFeed-11, poseidonIncidentFeed-27,
poseidonIncidentFeed-25, poseidonIncidentFeed-29, poseidonIncidentFeed-19, poseidonIncidentFeed-18]
for group hades
> 2017-04-30T18:05:10,877 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
@407 - (Re-)joining group hades
> 2017-05-01T00:01:55,707 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
@618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) dead for group
hades
> 2017-05-01T00:01:59,027 INFO  StreamThread-1 org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess()
@573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) for group hades.
> 2017-05-01T00:01:59,031 ERROR StreamThread-1 org.apache.kafka.streams.processor.internals.StreamThread.run()
@376 - stream-thread [StreamThread-1] Streams application error during processing:
>  java.lang.NullPointerException
> 	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:619)
~[kafka-streams-0.10.2.0.jar!/:?]
> 	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
[kafka-streams-0.10.2.0.jar!/:?]
> 2017-05-01T00:02:00,038 INFO  StreamThread-1 org.apache.kafka.clients.producer.KafkaProducer.close()
@689 - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
> 2017-05-01T00:02:00,949 WARN  StreamThread-1 org.apache.kafka.streams.processor.internals.StreamThread.setState()
@160 - Unexpected state transition from PARTITIONS_REVOKED to NOT_RUNNING
> 2017-05-01T00:02:00,951 ERROR StreamThread-1 com.williamhill.trading.platform.hades.kafka.KafkaStreamManager.uncaughtException()
@104 - UncaughtException in thread StreamThread-1, stopping kafka streams
>  java.lang.NullPointerException
> 	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:619)
~[kafka-streams-0.10.2.0.jar!/:?]
> 	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
~[kafka-streams-0.10.2.0.jar!/:?]
> 2017-05-01T00:02:01,076 WARN  kafka-streams-close-thread org.apache.kafka.streams.processor.internals.StreamThread.setState()
@160 - Unexpected state transition from NOT_RUNNING to PENDING_SHUTDOWN
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message