spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dibyendu Bhattacharya <dibyendu.bhattach...@gmail.com>
Subject Re: Error when Spark streaming consumes from Kafka
Date Tue, 03 Feb 2015 01:25:04 GMT
Or you can use this Low Level Kafka Consumer for Spark :
https://github.com/dibbhatt/kafka-spark-consumer

This is now part of http://spark-packages.org/ and is running successfully
for past few months in Pearson production environment . Being Low Level
consumer, it does not have this re-balancing issue which High Level
consumer have.

Also I know there are few who has shifted to this Low Level Consumer which
started giving them a better robust fault tolerant Kafka Receiver for Spark.

Regards,
Dibyendu

On Tue, Feb 3, 2015 at 3:57 AM, Tathagata Das <tathagata.das1565@gmail.com>
wrote:

> This is an issue that is hard to resolve without rearchitecting the whole
> Kafka Receiver. There are some workarounds worth looking into.
>
>
> http://mail-archives.apache.org/mod_mbox/kafka-users/201312.mbox/%3CCAFbh0Q38qQ0aAg_cj=jZk-kbi8XWf+1M6XLj+fzf6EEtj9zQAw@mail.gmail.com%3E
>
> On Mon, Feb 2, 2015 at 1:07 PM, Greg Temchenko <soid@dicefield.com> wrote:
>
>> Hi,
>>
>> This seems not fixed yet.
>> I filed an issue in jira:
>> https://issues.apache.org/jira/browse/SPARK-5505
>>
>> Greg
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Error-when-Spark-streaming-consumes-from-Kafka-tp19570p21471.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Mime
View raw message