Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1B402107DE for ; Tue, 3 Feb 2015 01:28:02 +0000 (UTC) Received: (qmail 91821 invoked by uid 500); 3 Feb 2015 01:27:45 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 91741 invoked by uid 500); 3 Feb 2015 01:27:44 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 91731 invoked by uid 99); 3 Feb 2015 01:27:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Feb 2015 01:27:44 +0000 X-ASF-Spam-Status: No, hits=2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dibyendu.bhattachary@gmail.com designates 209.85.218.43 as permitted sender) Received: from [209.85.218.43] (HELO mail-oi0-f43.google.com) (209.85.218.43) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Feb 2015 01:27:41 +0000 Received: by mail-oi0-f43.google.com with SMTP id z81so47020857oif.2 for ; Mon, 02 Feb 2015 17:25:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=I2OLua15DY5lJPDSPx39Erz2KQ1FWoFhnf7N/DfJzhA=; b=Oaq15oj6GPpXJcf3l72ZKYsJHxcLa3M/2nytPlBkNfWMyHjE5dgdxu9n38q3YRhAuT Zfsyh/GDoRgvrJHY0eCmEdScnks9xl0A7DkxcckvNYDHufPp88glGmKVnW1+cF03ngmy y7zt0jrIyNxq1zgqqV/iFBo5gSdl7QxmULdV0Wz6rbLI0GSrhjfoGmo3JCvgw5zr0+QU vSdLuLbUAjO1mbs+kw7cfP9ctcnx+7r1F4W25M4DV8z7rS4i7s7Qbx3oiCtXwYPoF0TY dlbQHALbmNbq4lnm29YitOnrTUq6ZcHssvs4NsezZu20TbTpXgA4Tgx47ZcVa8zhgIH2 w9Kg== MIME-Version: 1.0 X-Received: by 10.60.158.134 with SMTP id wu6mr13780979oeb.34.1422926705230; Mon, 02 Feb 2015 17:25:05 -0800 (PST) Received: by 10.76.158.163 with HTTP; Mon, 2 Feb 2015 17:25:04 -0800 (PST) In-Reply-To: References: <1422911225278-21471.post@n3.nabble.com> Date: Tue, 3 Feb 2015 06:55:04 +0530 Message-ID: Subject: Re: Error when Spark streaming consumes from Kafka From: Dibyendu Bhattacharya To: Tathagata Das Cc: Greg Temchenko , "user@spark.apache.org" Content-Type: multipart/alternative; boundary=047d7bd6b674ce88d3050e24f045 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bd6b674ce88d3050e24f045 Content-Type: text/plain; charset=UTF-8 Or you can use this Low Level Kafka Consumer for Spark : https://github.com/dibbhatt/kafka-spark-consumer This is now part of http://spark-packages.org/ and is running successfully for past few months in Pearson production environment . Being Low Level consumer, it does not have this re-balancing issue which High Level consumer have. Also I know there are few who has shifted to this Low Level Consumer which started giving them a better robust fault tolerant Kafka Receiver for Spark. Regards, Dibyendu On Tue, Feb 3, 2015 at 3:57 AM, Tathagata Das wrote: > This is an issue that is hard to resolve without rearchitecting the whole > Kafka Receiver. There are some workarounds worth looking into. > > > http://mail-archives.apache.org/mod_mbox/kafka-users/201312.mbox/%3CCAFbh0Q38qQ0aAg_cj=jZk-kbi8XWf+1M6XLj+fzf6EEtj9zQAw@mail.gmail.com%3E > > On Mon, Feb 2, 2015 at 1:07 PM, Greg Temchenko wrote: > >> Hi, >> >> This seems not fixed yet. >> I filed an issue in jira: >> https://issues.apache.org/jira/browse/SPARK-5505 >> >> Greg >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Error-when-Spark-streaming-consumes-from-Kafka-tp19570p21471.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org >> For additional commands, e-mail: user-help@spark.apache.org >> >> > --047d7bd6b674ce88d3050e24f045 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On Tue, Feb 3, 2015 at 3:57 AM, T= athagata Das <tathagata.das1565@gmail.com> wrote:<= br>
This is an issue that is= hard to resolve without rearchitecting the whole Kafka Receiver. There are= some workarounds worth looking into.=C2=A0


On Mon, Feb 2, 2015 at 1:07 PM, Gre= g Temchenko <soid@dicefield.com> wrote:
Hi,

This seems not fixed yet.
I filed an issue in jira: https://issues.apache.org/jira/browse/SPARK-5= 505

Greg



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble= .com/Error-when-Spark-streaming-consumes-from-Kafka-tp19570p21471.html<= br> Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org



--047d7bd6b674ce88d3050e24f045--