Return-Path: X-Original-To: apmail-flink-dev-archive@www.apache.org Delivered-To: apmail-flink-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F136818F0C for ; Thu, 23 Apr 2015 12:54:27 +0000 (UTC) Received: (qmail 8972 invoked by uid 500); 23 Apr 2015 12:54:27 -0000 Delivered-To: apmail-flink-dev-archive@flink.apache.org Received: (qmail 8920 invoked by uid 500); 23 Apr 2015 12:54:27 -0000 Mailing-List: contact dev-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list dev@flink.apache.org Received: (qmail 8909 invoked by uid 99); 23 Apr 2015 12:54:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Apr 2015 12:54:27 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: message received from 54.191.145.13 which is an MX secondary for dev@flink.apache.org) Received: from [54.191.145.13] (HELO mx1-us-west.apache.org) (54.191.145.13) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Apr 2015 12:54:22 +0000 Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with SMTP id 3EF9320659 for ; Thu, 23 Apr 2015 12:54:02 +0000 (UTC) Received: (qmail 87058 invoked by uid 99); 23 Apr 2015 12:53:04 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Apr 2015 12:53:04 +0000 Date: Thu, 23 Apr 2015 12:52:55 +0000 (UTC) From: "Robert Metzger (JIRA)" To: dev@flink.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (FLINK-1935) Reimplement PersistentKafkaSource using high level Kafka API MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org Robert Metzger created FLINK-1935: ------------------------------------- Summary: Reimplement PersistentKafkaSource using high level Kafka API Key: FLINK-1935 URL: https://issues.apache.org/jira/browse/FLINK-1935 Project: Flink Issue Type: Improvement Components: Kafka Connector, Streaming Affects Versions: 0.9 Reporter: Robert Metzger Assignee: Robert Metzger Fix For: 0.9 The current PersistentKafkaSource in Flink has some limitations that I seek to overcome by reimplementing it using Kafka's high level API (and manually committing the offsets to ZK). This approach only works when the offsets are committed to ZK directly. The current PersistentKafkaSource does not integrate with existing Kafka tools (for example for monitoring the "lag"). All the communication with Zookeeper is implemented manually in our current code. This is prone to errors and inefficiencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)