spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Armbrust (JIRA)" <>
Subject [jira] [Commented] (SPARK-11855) Catalyst breaks backwards compatibility in branch-1.6
Date Thu, 19 Nov 2015 19:12:10 GMT


Michael Armbrust commented on SPARK-11855:

Hey, very sorry for the confusion here.  Catalyst is not a public API and is explicitly hidden
from the public documentation and MIMA for this reason.  We also document this fact [here|].
 I don't know of any database that exposes this level of detail about the inner workings in
a stable way.

We avoid using package protection so that developers can work with the library in the REPL
and so that universities can use Spark as a platform for doing research.  However, in order
to keep pushing the public APIs forward, we have to reserve the right to change internals.

A compromise I'm totally willing to accept would be to add deprecated compatibility methods
for at least a couple of versions, provided the community is willing to put in the effort
to make PR adding them.

> Catalyst breaks backwards compatibility in branch-1.6
> -----------------------------------------------------
>                 Key: SPARK-11855
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.0
>            Reporter: Santiago M. Mola
>            Priority: Critical
> There's a number of APIs broken in catalyst 1.6.0. I'm trying to compile most cases:
> *UnresolvedRelation*'s constructor has been changed from taking a Seq to a TableIdentifier.
A deprecated constructor taking Seq would be needed to be backwards compatible.
> {code}
>  case class UnresolvedRelation(
> -    tableIdentifier: Seq[String],
> +    tableIdentifier: TableIdentifier,
>      alias: Option[String] = None) extends LeafNode {
> {code}
> It is similar with *UnresolvedStar*:
> {code}
> -case class UnresolvedStar(table: Option[String]) extends Star with Unevaluable {
> +case class UnresolvedStar(target: Option[Seq[String]]) extends Star with Unevaluable
> {code}
> *Catalog* did get a lot of signatures changed too (because of TableIdentifier). Providing
the older methods as deprecated also seems viable here.
> Spark 1.5 already broke backwards compatibility of part of catalyst API with respect
to 1.4. I understand there are good reasons for some cases, but we should try to minimize
backwards compatibility breakages for 1.x. Specially now that 2.x is on the horizon and there
will be a near opportunity to remove deprecated stuff.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message