spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: Slight API incompatibility caused by SPARK-4072
Date Wed, 15 Jul 2015 19:21:58 GMT
Or, alternatively, the bus could catch that error and ignore / log it,
instead of stopping the context...

On Wed, Jul 15, 2015 at 12:20 PM, Marcelo Vanzin <vanzin@cloudera.com>
wrote:

> Hmm, the Java listener was added in 1.3, so I think it will work for my
> needs.
>
> Might be worth it to make it clear in the SparkListener documentation that
> people should avoid using it directly. Or follow Reynold's suggestion.
>
> On Wed, Jul 15, 2015 at 12:14 PM, Patrick Wendell <pwendell@gmail.com>
> wrote:
>
>> One related note here is that we have a Java version of this that is
>> an abstract class - in the doc it says that it exists more or less to
>> allow for binary compatibility (it says it's for Java users, but
>> really Scala could use this also):
>>
>>
>> https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/JavaSparkListener.java#L23
>>
>> I think it might be reasonable that the Scala trait provides only
>> source compatibitly and the Java class provides binary compatibility.
>>
>> - Patrick
>>
>> On Wed, Jul 15, 2015 at 11:47 AM, Marcelo Vanzin <vanzin@cloudera.com>
>> wrote:
>> > Hey all,
>> >
>> > Just noticed this when some of our tests started to fail. SPARK-4072
>> added a
>> > new method to the "SparkListener" trait, and even though it has a
>> default
>> > implementation, it doesn't seem like that applies retroactively.
>> >
>> > Namely, if you have an existing, compiled app that has an
>> implementation of
>> > SparkListener, that app won't work on 1.5 without a recompile. You'll
>> get
>> > something like this:
>> >
>> > java.lang.AbstractMethodError
>> >       at
>> >
>> org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62)
>> >       at
>> >
>> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
>> >       at
>> >
>> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
>> >       at
>> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56)
>> >       at
>> >
>> org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
>> >       at
>> >
>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79)
>> >       at
>> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1235)
>> >       at
>> >
>> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
>> >
>> >
>> > Now I know that "SparkListener" is marked as @DeveloperApi, but is this
>> > something we should care about? Seems like adding methods to traits is
>> just
>> > as backwards-incompatible as adding new methods to Java interfaces.
>> >
>> >
>> > --
>> > Marcelo
>>
>
>
>
> --
> Marcelo
>



-- 
Marcelo

Mime
View raw message