spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: Kafaka 0.8, 0.9 in Structured Streaming
Date Fri, 07 Oct 2016 20:15:33 GMT
Does Kafka 0.10 work on a Kafka 0.8/0.9 cluster?


On Fri, Oct 7, 2016 at 1:14 PM, Jeremy Smith <jeremy.smith@acorns.com>
wrote:

> +1
>
> We're on CDH, and it will probably be a while before they support Kafka
> 0.10. At the same time, we don't use their Spark and we're looking forward
> to upgrading to 2.0.x and using structured streaming.
>
> I was just going to write our own Kafka Source implementation which uses
> the existing KafkaRDD but it would be much easier to get buy-in for an
> official Spark module.
>
> Jeremy
>
> On Fri, Oct 7, 2016 at 12:41 PM, Michael Armbrust <michael@databricks.com>
> wrote:
>
>> We recently merged support for Kafak 0.10.0 in Structured Streaming, but
>> I've been hearing a few people tell me that they are stuck on an older
>> version of Kafka and cannot upgrade.  I'm considering revisiting
>> SPARK-17344 <https://issues.apache.org/jira/browse/SPARK-17344>, but it
>> would be good to have more information.
>>
>> Could people please vote or comment on the above ticket if a lack of
>> support for older versions of kafka would block you from trying out
>> structured streaming?
>>
>> Thanks!
>>
>> Michael
>>
>
>

Mime
View raw message