spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-22824) Spark Structured Streaming Source trait breaking change
Date Mon, 18 Dec 2017 22:05:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295745#comment-16295745
] 

Apache Spark commented on SPARK-22824:
--------------------------------------

User 'joseph-torres' has created a pull request for this issue:
https://github.com/apache/spark/pull/20012

> Spark Structured Streaming Source trait breaking change
> -------------------------------------------------------
>
>                 Key: SPARK-22824
>                 URL: https://issues.apache.org/jira/browse/SPARK-22824
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.3.0
>         Environment: Azure Databricks 3.4
>            Reporter: Matthieu Maitre
>            Assignee: Jose Torres
>
> org.apache.spark.sql.execution.streaming.Offset was moved to org.apache.spark.sql.sources.v2.reader.Offset
on the Source trait:
> https://github.com/apache/spark/commit/f8c7c1f21aa9d1fd38b584ca8c4adf397966e9f7#diff-56453fdb4dc2d7666c7ab237cb102189
> This broke our custom sources that are called in Spark jobs running on Azure Databricks
3.4 since the Maven package does not match the hosted Spark bits. We use the following Maven
version to be able to deploy on Azure Databricks:
>         <dependency>
>             <groupId>org.apache.spark</groupId>
>             <artifactId>spark-sql_2.11</artifactId>
>             <version>2.3.0-SNAPSHOT</version>
>         </dependency>



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message