spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marco Gaido (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24189) Spark Strcutured Streaming not working with the Kafka Transactions
Date Sat, 05 May 2018 14:45:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464798#comment-16464798
] 

Marco Gaido commented on SPARK-24189:
-------------------------------------

You need to add also:

{code}
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.0.0</version>
</dependency>
{code}

> Spark Strcutured Streaming not working with the Kafka Transactions
> ------------------------------------------------------------------
>
>                 Key: SPARK-24189
>                 URL: https://issues.apache.org/jira/browse/SPARK-24189
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.3.0
>            Reporter: bharath kumar avusherla
>            Priority: Major
>
> Was trying to read kafka transactional topic using Spark Structured Streaming 2.3.0 with
the  kafka option isolation-level = "read_committed", but spark reading the data immediately
without waiting for the data in topic to be committed. In spark documentation it was mentioned
as Structured Streaming supports Kafka version 0.10 or higher. I am using below command to
test the scenario.
> val df = spark
>  .readStream
>  .format("kafka")
>  .option("kafka.bootstrap.servers", "localhost:9092")
>  .option("subscribe", "test-topic")
>  .option("isolation-level","read_committed")
>  .load()
> Can you please let me know if the transactional read is supported in SPark 2.3.0 strcutured
Streaming or am i missing anything.
>  
> Thank you.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message