carbondata-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pallavi Singh <pallavi.si...@knoldus.in>
Subject Re: Spark 2.1.1 with CarbonData 1.1.0
Date Thu, 25 May 2017 05:00:04 GMT
Hi Bill,

CarbonContext is only for Spark 1.6, for Spark 2.1 you should use
CarbonSession. Please refer to CarbonSessionExample
<https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala>
for
more details.

On Thu, May 25, 2017 at 12:01 AM, Bill Speirs <wspeirs@datto.com> wrote:

> I'm trying to follow the directions for using Spark 2.1.1 with
> CarbonData 1.1.0 found here:
> http://carbondata.apache.org/quick-start-guide.html
>
> I compiled CarbonData using: mvn -DskipTests -Pspark-2.1
> -Dspark.version=2.1.1 clean package
>
> I ran Spark with: ./bin/spark-shell --jars
> ../carbondata-parent-1.1.0/assembly/target/scala-2.11/
> carbondata_2.11-1.1.0-shade-hadoop2.2.0.jar
>
> I imported the two classes (not sure if the second is a typo with the
> underline):
>
> scala> import org.apache.spark.sql.SparkSession
> import org.apache.spark.sql.SparkSession
>
> scala> import org.apache.spark.sql.CarbonSession._
> import org.apache.spark.sql.CarbonSession._
>
> Then tried to create the carbon variable:
>
> scala> val carbon =
> SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("
> hdfs://localhost:9000/carbondata")
> <console>:28: error: value getOrCreateCarbonContext is not a member of
> org.apache.spark.sql.SparkSession.Builder
>        val carbon =
> SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("
> hdfs://localhost:9000/carbondata")
>
> Thoughts on what I'm doing wrong here?
>
> Bill-
>



-- 
Regards | Pallavi Singh
Software Consultant
Knoldus Software LLP
+91-9911235949

Mime
View raw message