Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id DD139200CD3 for ; Fri, 28 Jul 2017 22:46:57 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id DB80F16D564; Fri, 28 Jul 2017 20:46:57 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2D37216D527 for ; Fri, 28 Jul 2017 22:46:57 +0200 (CEST) Received: (qmail 91371 invoked by uid 500); 28 Jul 2017 20:46:56 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 91361 invoked by uid 99); 28 Jul 2017 20:46:56 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Jul 2017 20:46:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 9BF55C35B5 for ; Fri, 28 Jul 2017 20:46:55 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.487 X-Spam-Level: *** X-Spam-Status: No, score=3.487 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, NML_ADSP_CUSTOM_MED=1.2, SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001, URI_HEX=1.313] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 9meg174Be4pb for ; Fri, 28 Jul 2017 20:46:46 +0000 (UTC) Received: from mwork.nabble.com (mwork.nabble.com [162.253.133.43]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 1A94B5FB32 for ; Fri, 28 Jul 2017 20:46:46 +0000 (UTC) Received: from mjoe.nabble.com (unknown [162.253.133.57]) by mwork.nabble.com (Postfix) with ESMTP id C33395703A9B0 for ; Fri, 28 Jul 2017 13:46:45 -0700 (MST) Date: Fri, 28 Jul 2017 13:46:44 -0700 (PDT) From: ninad To: user@flink.apache.org Message-ID: <1501274804643-14522.post@n4.nabble.com> In-Reply-To: References: <1496759887655-13527.post@n4.nabble.com> <1496940767596-13597.post@n4.nabble.com> <1497623264174-13805.post@n4.nabble.com> <1499880741612-14224.post@n4.nabble.com> Subject: Re: Fink: KafkaProducer Data Loss MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit archived-at: Fri, 28 Jul 2017 20:46:58 -0000 Hi Gordon, I was able to reproduce the data loss on standalone flink cluster also. I have stripped down version of our code with here: Environment: Flink standalone 1.3.0 Kafka 0.9 *What the code is doing:* -consume messages from kafka topic ('event.filter.topic' property in application.properties) -group them by key -analyze the events in a window and filter some messages. -send remaining messages to kafka topc ('sep.http.topic' property in application.properties) To build: ./gradlew clean assemble The jar needs path to 'application.properties' file to run Important properties in application.properties: window.session.interval.sec kafka.brokers event.filter.topic --> source topic sep.http.topic --> destination topic To test: -Use 'EventGenerator' class to publish messages to source kafka topic The data published won't be filtered by the logic. If you publish 10 messages to the source topic, those 10 messages should be sent to the destination topic. -Once we see that flink has received all the messages, bring down all kafka brokers -Let Flink jobs fail for 2-3 times. -Restart kafka brokers. Note: Data loss isn't observed frequently. 1/4 times or so. Thanks for all your help. eventFilter.zip -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14522.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.