From dev-return-97120-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Tue Aug 14 14:37:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4A6CF180654 for ; Tue, 14 Aug 2018 14:37:05 +0200 (CEST) Received: (qmail 49760 invoked by uid 500); 14 Aug 2018 12:37:04 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 49740 invoked by uid 99); 14 Aug 2018 12:37:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Aug 2018 12:37:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 942011800C5 for ; Tue, 14 Aug 2018 12:37:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id twqYigFK1UN9 for ; Tue, 14 Aug 2018 12:37:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 677D75F300 for ; Tue, 14 Aug 2018 12:37:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 92098E00D3 for ; Tue, 14 Aug 2018 12:37:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 5194623F9D for ; Tue, 14 Aug 2018 12:37:00 +0000 (UTC) Date: Tue, 14 Aug 2018 12:37:00 +0000 (UTC) From: "Tim Van Laer (JIRA)" To: dev@kafka.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (KAFKA-7290) Kafka Streams application fails to rebalance and is stuck in "Updated cluster metadata version" MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Tim Van Laer created KAFKA-7290: ----------------------------------- Summary: Kafka Streams application fails to rebalance and is stuck in "Updated cluster metadata version" Key: KAFKA-7290 URL: https://issues.apache.org/jira/browse/KAFKA-7290 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 0.11.0.3, 0.10.2.2, 0.10.2.1 Reporter: Tim Van Laer Attachments: cg_metadata_failure.txt Our kafka streams application crashed due to a RocksDBException, after that the consumer group basically became unusable. Every consumer in the group went from RUNNING to REBALANCING and were stuck to that state. The application was still on an older version of Kafka Streams (0.10.2.1), but an upgrade of the library didn't got the consumer group back active. We tried: * adding and removing consumers to the group, no luck, none of the consumers starts processing * stopping all consumers and restarted the application, no luck * stopping all consumer, reset the consumer group (using the kafka-streams-application-reset tool), no luck * replaced the underlying machines, no luck * Upgrading our application from Kafka Streams 0.10.2.1 to 0.10.2.2 and 0.11.0.3 after it got stuck, no luck We finally got the application back running by changing the applicationId (we could afford to loose the state in this particular case). See attachment for debug logs of the application. The application can reach the Kafka cluster but fails to join the group. The RocksDBException that triggered this state (I lost the container, so unfortunately I don't have more logging): {code} 2018-08-14 01:40:39 ERROR StreamThread:813 - stream-thread [StreamThread-1] Failed to commit StreamTask 1_1 state: org.apache.kafka.streams.errors.ProcessorStateException: task [1_1] Failed to flush state store firehose_subscriptions at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:337) ~[firechief.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:72) ~[firechief.jar:?] at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188) ~[firechief.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:280) ~[firechief.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.commitOne(StreamThread.java:807) [firechief.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:794) [firechief.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:769) [firechief.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:647) [firechief.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:361) [firechief.jar:?] Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error while executing flush from store firehose_subscriptions at org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:354) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.RocksDBStore.flush(RocksDBStore.java:345) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.WrappedStateStore$AbstractWrappedStateStore.flush(WrappedStateStore.java:80) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.MeteredKeyValueStore$6.run(MeteredKeyValueStore.java:92) ~[firechief.jar:?] at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.flush(MeteredKeyValueStore.java:186) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.CachingKeyValueStore.flush(CachingKeyValueStore.java:113) ~[firechief.jar:?] at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:335) ~[firechief.jar:?] ... 8 more Caused by: org.rocksdb.RocksDBException: _ at org.rocksdb.RocksDB.flush(Native Method) ~[firechief.jar:?] at org.rocksdb.RocksDB.flush(RocksDB.java:1642) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:352) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.RocksDBStore.flush(RocksDBStore.java:345) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.WrappedStateStore$AbstractWrappedStateStore.flush(WrappedStateStore.java:80) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.MeteredKeyValueStore$6.run(MeteredKeyValueStore.java:92) ~[firechief.jar:?] at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.flush(MeteredKeyValueStore.java:186) ~[firechief.jar:?] at org.apache.kafka.streams.state.internals.CachingKeyValueStore.flush(CachingKeyValueStore.java:113) ~[firechief.jar:?] at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:335) ~[firechief.jar:?] ... 8 more {code} Any ideas on what is wrong or what we can do to workaround this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)