spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Armbrust (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-22824) Spark Structured Streaming Source trait breaking change
Date Mon, 18 Dec 2017 21:19:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295676#comment-16295676
] 

Michael Armbrust commented on SPARK-22824:
------------------------------------------

This is technically an internal API (as is all of [org.apache.spark.sql.execution|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/package.scala]).
 That said, I think we should do our best to preserve compatibility until there is a {{@Stable}}
public alternative (which is the goal of the V2 project).

We'll open a PR to try and fix this.

> Spark Structured Streaming Source trait breaking change
> -------------------------------------------------------
>
>                 Key: SPARK-22824
>                 URL: https://issues.apache.org/jira/browse/SPARK-22824
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.3.0
>         Environment: Azure Databricks 3.4
>            Reporter: Matthieu Maitre
>
> org.apache.spark.sql.execution.streaming.Offset was moved to org.apache.spark.sql.sources.v2.reader.Offset
on the Source trait:
> https://github.com/apache/spark/commit/f8c7c1f21aa9d1fd38b584ca8c4adf397966e9f7#diff-56453fdb4dc2d7666c7ab237cb102189
> This broke our custom sources that are called in Spark jobs running on Azure Databricks
3.4 since the Maven package does not match the hosted Spark bits. We use the following Maven
version to be able to deploy on Azure Databricks:
>         <dependency>
>             <groupId>org.apache.spark</groupId>
>             <artifactId>spark-sql_2.11</artifactId>
>             <version>2.3.0-SNAPSHOT</version>
>         </dependency>



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message