carbondata-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravindra Pesala <ravi.pes...@gmail.com>
Subject Re: Spark 2.1.1 with CarbonData 1.1.0
Date Thu, 25 May 2017 15:39:18 GMT
Hi Bill,

Actually CarbonData is not supported with spark 2.1.1, it is only supported
with spark 2.1.0.
And about your error it says that method is not present so better give
absolute path of CarbonData jar to spark-shell and also check the imports
are done properly or not. Please refer to the example CarbonSessionExample
inside examples folder for creating carbon session

Regards,
Ravindra
On Thu, 25 May 2017 at 10:30 AM, Pallavi Singh <pallavi.singh@knoldus.in>
wrote:

> Hi Bill,
>
> CarbonContext is only for Spark 1.6, for Spark 2.1 you should use
> CarbonSession. Please refer to CarbonSessionExample
> <https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala>
for
> more details.
>
> On Thu, May 25, 2017 at 12:01 AM, Bill Speirs <wspeirs@datto.com> wrote:
>
>> I'm trying to follow the directions for using Spark 2.1.1 with
>> CarbonData 1.1.0 found here:
>> http://carbondata.apache.org/quick-start-guide.html
>>
>> I compiled CarbonData using: mvn -DskipTests -Pspark-2.1
>> -Dspark.version=2.1.1 clean package
>>
>> I ran Spark with: ./bin/spark-shell --jars
>>
>> ../carbondata-parent-1.1.0/assembly/target/scala-2.11/carbondata_2.11-1.1.0-shade-hadoop2.2.0.jar
>>
>> I imported the two classes (not sure if the second is a typo with the
>> underline):
>>
>> scala> import org.apache.spark.sql.SparkSession
>> import org.apache.spark.sql.SparkSession
>>
>> scala> import org.apache.spark.sql.CarbonSession._
>> import org.apache.spark.sql.CarbonSession._
>>
>> Then tried to create the carbon variable:
>>
>> scala> val carbon =
>>
>> SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")
>> <console>:28: error: value getOrCreateCarbonContext is not a member of
>> org.apache.spark.sql.SparkSession.Builder
>>        val carbon =
>>
>> SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")
>>
>> Thoughts on what I'm doing wrong here?
>>
>> Bill-
>>
>
>
>
> --
> Regards | Pallavi Singh
> Software Consultant
> Knoldus Software LLP
> +91-9911235949
>

Mime
View raw message