kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6212) Kafka Streams - Incorrect partition rebalancing
Date Thu, 16 Nov 2017 18:36:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255754#comment-16255754
] 

Matthias J. Sax commented on KAFKA-6212:
----------------------------------------

That's weird. No idea atm what could cause this. But it should help to debug it. Thx.

> Kafka Streams - Incorrect partition rebalancing
> -----------------------------------------------
>
>                 Key: KAFKA-6212
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6212
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 0.10.0.1
>         Environment: Linux
>            Reporter: Ivan Atanasov
>
> Trying to use streaming with version 0.10.0.1 of kafka but it is not working how I'd
expect. I realize that this is a fairly old version now but it is what we are running and
are not in a position to upgrade right now.
> The particular problem I am having is when an extra instance of the streaming app is
run using the same application ID. What seems to happen is the newly introduced instance takes
half of the partitions available, which is expected but the original instance drops all the
partitions it was reading from. therefore from then on, data is only read from half the partitions.
> Strangely offsets are still being committed for the other partitions but the data from
them is not consumed as expected.
> My topology is very simple for now, all it does is a print of the message. Also I have
tried making both instances use different client IDs and state directories.
> Is this a known bug in 0.10.0.1?
> *Logs Below:*
> Instance 1:
> {quote}[2017-11-15 10:41:41,597] INFO [StreamThread-1] Setting newly assigned partitions
[rawEvents-5, rawEvents-6, rawEvents-3, rawEvents-4, rawEvents-9, rawEvents-7, rawEvents-8,
rawEvents-1, rawEvents-2, rawEvents-0] for group kafka-stream-test (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-11-15 10:41:41,616] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_0 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,645] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_1 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,645] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_2 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_3 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_4 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_5 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_6 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_7 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_8 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,648] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_9 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,682] INFO [StreamThread-1] Revoking previously assigned partitions
[rawEvents-5, rawEvents-6, rawEvents-3, rawEvents-4, rawEvents-9, rawEvents-7, rawEvents-8,
rawEvents-1, rawEvents-2, rawEvents-0] for group kafka-stream-test (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-11-15 10:42:08,682] INFO [StreamThread-1] Removing a task 0_0 (org.apache.kafka.streams.processor.internals.StreamThread){quote}
> Instance 2:
> {quote}[2017-11-15 10:42:08,827] INFO [StreamThread-1] Successfully joined group kafka-stream-test
with generation 2 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2017-11-15 10:42:08,829] INFO [StreamThread-1] Setting newly assigned partitions [rawEvents-5,
rawEvents-3, rawEvents-1, rawEvents-2, rawEvents-0] for group kafka-stream-test (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-11-15 10:42:08,840] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_0 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,869] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_1 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,870] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_2 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,871] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_3 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,872] INFO [StreamThread-1] Creating restoration consumer client
for stream task #0_5 (org.apache.kafka.streams.processor.internals.StreamTask){quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message