spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingyu Kim <m...@palantir.com>
Subject Re: [SPARK-3050] Spark program running with 1.0.2 jar cannot run against a 1.0.1 cluster
Date Fri, 15 Aug 2014 18:13:36 GMT
Thanks for your response. I think I misinterpreted the
stability/compatibility guarantee with 1.0 release. It seems like the
compatibility is only at the API level.

This is interesting because it means any system/product that is built on top
of Spark and uses Spark with a long-running SparkContext connecting to the
cluster over network, will need to make sure it has the exact same version
of Spark jar as the cluster, even to the patch version. This would be
analogous to having to compile Spark against a very specific version of
Hadoop, as opposed to currently being able to use the Spark package with
CDH4 against most of the CDH4 Hadoop clusters.

Is it correct that Spark is focusing and prioritizing around the
spark-submit use cases than the aforementioned use cases? I just wanted to
better understand the future direction/prioritization of spark.

Thanks,
Mingyu

From:  Patrick Wendell <pwendell@gmail.com>
Date:  Thursday, August 14, 2014 at 6:32 PM
To:  Gary Malouf <malouf.gary@gmail.com>
Cc:  Mingyu Kim <mkim@palantir.com>, "dev@spark.apache.org"
<dev@spark.apache.org>
Subject:  Re: [SPARK-3050] Spark program running with 1.0.2 jar cannot run
against a 1.0.1 cluster

I commented on the bug. For driver mode, you'll need to get the
corresponding version of spark-submit for Spark 1.0.2.


On Thu, Aug 14, 2014 at 3:43 PM, Gary Malouf <malouf.gary@gmail.com> wrote:
> To be clear, is it 'compiled' against 1.0.2 or it packaged with it?
> 
> 
> On Thu, Aug 14, 2014 at 6:39 PM, Mingyu Kim <mkim@palantir.com> wrote:
> 
>> > I ran a really simple code that runs with Spark 1.0.2 jar and connects to
>> > a Spark 1.0.1 cluster, but it fails with java.io.InvalidClassException. I
>> > filed the bug at https://issues.apache.org/jira/browse/SPARK-3050
>> <https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/br
>> owse/SPARK-3050&k=fDZpZZQMmYwf27OU23GmAQ%3D%3D%0A&r=UKDOcu6qL3KsoZhpOohNBR1uc
>> PNmWnbd3eEJ9hVUdMk%3D%0A&m=qvQ59wZwD7EuezjTuLzmNTRUamDRDnI7%2F0%2BnULtXk4k%3D
>> %0A&s=b7abf7638a3e6fac2ddac9d8f0ca52f1a92945465abfb2e2d996a96d2301fec5> .
>> >
>> > I assumed the minor and patch releases shouldn┬╣t break compatibility. Is
>> > that correct?
>> >
>> > Thanks,
>> > Mingyu
>> >




Mime
View raw message