spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong Zhu (JIRA)" <>
Subject [jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0
Date Fri, 10 Mar 2017 23:46:04 GMT


Shixiong Zhu commented on SPARK-18057:

>  Based on previous kafka client upgrades I wouldn't expect them to be binary compatible,
so it's likely to cause someone problems if they were also making use of kafka client libraries
in their spark job. Still may be the path of least resistance.

I can confirm the APIs used by Kafka sink is source compatible since I didn't change any core
source codes (test codes have to be changed because of the server APIs are changed). Since
these APIs are Java APIs. I'm pretty sure they are binary compatible. So for the user, even
if we upgrade the Kafka client version, they can still downgrade the Kafka client version
if they want, and just repackage the codes with the kafka client. It's a bit annoying that
"--packages" probably won't work but it's acceptable.

> Update structured streaming kafka from 10.0.1 to 10.2.0
> -------------------------------------------------------
>                 Key: SPARK-18057
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>            Reporter: Cody Koeninger
> There are a couple of relevant KIPs here,

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message