spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: [Spark SQL] SQLContext getOrCreate incorrect behaviour
Date Mon, 21 Dec 2015 16:32:38 GMT
In Jerry's example, the first SparkContext, sc, has been stopped.

So there would be only one SparkContext running at any given moment.

Cheers

On Mon, Dec 21, 2015 at 8:23 AM, Chester @work <chester@alpinenow.com>
wrote:

> Jerry
>     I thought you should not create more than one SparkContext within one
> Jvm, ...
> Chester
>
> Sent from my iPhone
>
> > On Dec 20, 2015, at 2:59 PM, Jerry Lam <chilinglam@gmail.com> wrote:
> >
> > Hi Spark developers,
> >
> > I found that SQLContext.getOrCreate(sc: SparkContext) does not behave
> correctly when a different spark context is provided.
> >
> > ```
> > val sc = new SparkContext
> > val sqlContext =SQLContext.getOrCreate(sc)
> > sc.stop
> > ...
> >
> > val sc2 = new SparkContext
> > val sqlContext2 = SQLContext.getOrCreate(sc2)
> > sc2.stop
> > ```
> >
> > The sqlContext2 will reference sc instead of sc2 and therefore, the
> program will not work because sc has been stopped.
> >
> > Best Regards,
> >
> > Jerry
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message