kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Worsley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5825) Streams not processing when exactly once is set
Date Sun, 03 Sep 2017 19:21:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151923#comment-16151923
] 

Ryan Worsley commented on KAFKA-5825:
-------------------------------------

I've updated the tests to use the {{EmbeddedKafkaCluster}} and removed the dependency on the
Kafka test library that I was using - it took a while to solve various library dependency
issues, but I can now start a 3 node cluster.

[~mjsax] it's exhibiting the same behaviour unfortunately, there's no errors in the broker
logs, just a lot of this:

{{
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition - Partition [__transaction_state,11]
on broker 2: Skipping update high watermark since new hw 0 [0 : 0] is not larger than old
hw 0 [0 : 0].All LEOs are 0 [0 : 0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition - Partition [__transaction_state,26]
on broker 2: Skipping update high watermark since new hw 0 [0 : 0] is not larger than old
hw 0 [0 : 0].All LEOs are 0 [0 : 0]
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition - Partition [__transaction_state,11]
on broker 2: Recorded replica 0 log end offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition - Partition [__transaction_state,26]
on broker 2: Recorded replica 1 log end offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager - [Replica Manager
on Broker 2]: Request key __transaction_state-26 unblocked 0 producer requests.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager - [Replica Manager
on Broker 2]: Request key __transaction_state-11 unblocked 0 producer requests.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition - Partition [__transaction_state,11]
on broker 2: Skipping update high watermark since new hw 0 [0 : 0] is not larger than old
hw 0 [0 : 0].All LEOs are 0 [0 : 0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition - Partition [__transaction_state,11]
on broker 2: Recorded replica 1 log end offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition - Partition [__transaction_state,8]
on broker 2: Skipping update high watermark since new hw 0 [0 : 0] is not larger than old
hw 0 [0 : 0].All LEOs are 0 [0 : 0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager - [Replica Manager
on Broker 2]: Request key __transaction_state-11 unblocked 0 producer requests.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition - Partition [__transaction_state,8]
on broker 2: Recorded replica 0 log end offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition - Partition [__transaction_state,8]
on broker 2: Skipping update high watermark since new hw 0 [0 : 0] is not larger than old
hw 0 [0 : 0].All LEOs are 0 [0 : 0]
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager - [Replica Manager
on Broker 2]: Request key __transaction_state-8 unblocked 0 producer requests.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition - Partition [__transaction_state,8]
on broker 2: Recorded replica 1 log end offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager - [Replica Manager
on Broker 2]: Request key __transaction_state-8 unblocked 0 producer requests.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition - Partition [__transaction_state,41]
on broker 2: Skipping update high watermark since new hw 0 [0 : 0] is not larger than old
hw 0 [0 : 0].All LEOs are 0 [0 : 0]
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition - Partition [__transaction_state,41]
on broker 2: Recorded replica 0 log end offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager - [Replica Manager
on Broker 2]: Request key __transaction_state-41 unblocked 0 producer requests.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition - Partition [__transaction_state,41]
on broker 2: Skipping update high watermark since new hw 0 [0 : 0] is not larger than old
hw 0 [0 : 0].All LEOs are 0 [0 : 0]
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition - Partition [__transaction_state,41]
on broker 2: Recorded replica 1 log end offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition - Partition [__transaction_state,17]
on broker 2: Skipping update high watermark since new hw 3 [0 : 474] is not larger than old
hw 3 [0 : 474].All LEOs are 3 [0 : 474]
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager - [Replica Manager
on Broker 2]: Request key __transaction_state-41 unblocked 0 producer requests.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition - Partition [__transaction_state,17]
on broker 2: Recorded replica 0 log end offset (LEO) position 3.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition - Partition [__transaction_state,17]
on broker 2: Skipping update high watermark since new hw 3 [0 : 474] is not larger than old
hw 3 [0 : 474].All LEOs are 3 [0 : 474]
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager - [Replica Manager
on Broker 2]: Request key __transaction_state-17 unblocked 0 producer requests.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition - Partition [__transaction_state,17]
on broker 2: Recorded replica 1 log end offset (LEO) position 3.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager - [Replica Manager
on Broker 2]: Request key __transaction_state-17 unblocked 0 producer requests.}}

The streams task never processes any messages, do you have any further suggestions?

> Streams not processing when exactly once is set
> -----------------------------------------------
>
>                 Key: KAFKA-5825
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5825
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 0.11.0.0
>         Environment: EmbeddedKafka running on Windows.  Relevant files attached.
>            Reporter: Ryan Worsley
>         Attachments: build.sbt, log4j.properties, Tests.scala
>
>
> +Set-up+
> I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] for ScalaTest.
> This spins up a single broker internally on a random port.
> I've written two tests - the first without transactions, the second with.  They're nearly
identical apart from the config and the transactional semantics.  I've written the transactional
version based on Neha's [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/]
which is the closest thing I could find to instructions.
> The tests wait until a single message is processed by the streams topology, they use
this message to complete a promise that the test is waiting on.  Once the promise completes
the test verifies the value of the promise as being the expected value of the message.
> +Observed behaviour+
> The first test passes fine, the second test times out, the stream processor never seems
to read the transactional message.
> +Notes+
> I've attached my build.sbt, log4j.properties and my Tests.scala file in order to make
it as easy as possible for someone to re-create.  I'm running on Windows and using Scala as
this reflects my workplace.  I completely expect there to be some configuration issue that's
causing this, but am unable to proceed at this time.
> Related information: https://github.com/manub/scalatest-embedded-kafka/issues/82



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message