carbondata-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arnaud G <greatpat...@gmail.com>
Subject Issue with quickstart introduction
Date Wed, 26 Jul 2017 15:59:07 GMT
Hi,



I have compiled the latest version of CarbonData which is compatible with
HDP2.6. I’m doing the following steps but the data are never copied to the
table.



Start Spark Shell:

/home/ubuntu/carbondata# spark-shell --jars /home/ubuntu/carbondata/
carbondata_2.11-1.2.0-SNAPSHOT-shade-hadoop2.7.2.jar



Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.2.6.0.3-8

      /_/



Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_121)

Type in expressions to have them evaluated.

Type :help for more information.



scala>  import org.apache.spark.sql.SparkSession

import org.apache.spark.sql.SparkSession



scala> import org.apache.spark.sql.CarbonSession._

import org.apache.spark.sql.CarbonSession._



scala> val carbon = SparkSession.builder().config(sc.getConf).
getOrCreateCarbonSession("/test/carbondata/","/test/carbondata/")

17/07/26 14:58:42 WARN SparkContext: Using an existing SparkContext; some
configuration may not take effect.

17/07/26 14:58:42 WARN CarbonProperties: main The enable unsafe sort value
"null" is invalid. Using the default value "false

17/07/26 14:58:42 WARN CarbonProperties: main The custom block distribution
value "null" is invalid. Using the default value "false

17/07/26 14:58:42 WARN CarbonProperties: main The enable vector reader
value "null" is invalid. Using the default value "true

17/07/26 14:58:42 WARN CarbonProperties: main The value "null" configured
for key carbon.lock.type" is invalid. Using the default value "HDFSLOCK

carbon: org.apache.spark.sql.SparkSession = org.apache.spark.sql.
CarbonSession@5f7bd970



scala> carbon.sql("CREATE TABLE IF NOT EXISTS test_carbon(id string, name
string, city string,age Int)  STORED BY 'carbondata'")

17/07/26 15:04:35 AUDIT CreateTable:
[gateway-dc1r04n01][hdfs][Thread-1]Creating
Table with Database name [default] and Table name [test_carbon]

17/07/26 15:04:36 WARN HiveExternalCatalog: Couldn't find corresponding
Hive SerDe for data source provider org.apache.spark.sql.CarbonSource.
Persisting data source table `default`.`test_carbon` into Hive metastore in
Spark SQL specific format, which is NOT compatible with Hive.

17/07/26 15:04:36 AUDIT CreateTable: [gateway-dc1][hdfs][Thread-1]Table
created with Database name [default] and Table name [test_carbon]

res7: org.apache.spark.sql.DataFrame = []



scala> carbon.sql("describe test_carbon").show()

+--------+---------+-------+

|col_name|data_type|comment|

+--------+---------+-------+

|      id|   string|   null|

|    name|   string|   null|

|    city|   string|   null|

|     age|      int|   null|

+--------+---------+-------+





scala> carbon.sql("INSERT INTO test_carbon VALUES(1,'x1','x2',34)")

17/07/26 15:07:25 AUDIT CarbonDataRDDFactory$:
[gateway-dc1][hdfs][Thread-1]Data
load request has been received for table default.test_carbon

17/07/26 15:07:25 WARN CarbonDataProcessorUtil: main sort scope is set to
LOCAL_SORT

17/07/26 15:07:25 WARN CarbonDataProcessorUtil: Executor task launch worker
for task 5 sort scope is set to LOCAL_SORT

17/07/26 15:07:25 WARN CarbonDataProcessorUtil: Executor task launch worker
for task 5 batch sort size is set to 0

17/07/26 15:07:25 WARN CarbonDataProcessorUtil: Executor task launch worker
for task 5 sort scope is set to LOCAL_SORT

17/07/26 15:07:25 WARN CarbonDataProcessorUtil: Executor task launch worker
for task 5 sort scope is set to LOCAL_SORT

17/07/26 15:07:25 AUDIT CarbonDataRDDFactory$:
[gateway-dc1r04n01][hdfs][Thread-1]Data
load is successful for default.test_carbon

res11: org.apache.spark.sql.DataFrame = []



scala> carbon.sql("LOAD DATA INPATH 'hdfs://xxxx/test/carbondata/sample.csv'
INTO TABLE test_carbon")

17/07/26 14:59:28 AUDIT CarbonDataRDDFactory$:
[gateway-dc1][hdfs][Thread-1]Data
load request has been received for table default.test_table

17/07/26 14:59:28 WARN CarbonDataProcessorUtil: main sort scope is set to
LOCAL_SORT

17/07/26 14:59:28 WARN CarbonDataProcessorUtil: [Executor task launch
worker for task
0][partitionID:default_test_table_8662d5ff-9392-4e23-b37e-9a4485f71f0e]
sort scope is set to LOCAL_SORT

17/07/26 14:59:28 WARN CarbonDataProcessorUtil: [Executor task launch
worker for task
0][partitionID:default_test_table_8662d5ff-9392-4e23-b37e-9a4485f71f0e]
batch sort size is set to 0

17/07/26 14:59:28 WARN CarbonDataProcessorUtil: [Executor task launch
worker for task
0][partitionID:default_test_table_8662d5ff-9392-4e23-b37e-9a4485f71f0e]
sort scope is set to LOCAL_SORT

17/07/26 14:59:28 WARN CarbonDataProcessorUtil: [Executor task launch
worker for task
0][partitionID:default_test_table_8662d5ff-9392-4e23-b37e-9a4485f71f0e]
sort scope is set to LOCAL_SORT

17/07/26 14:59:29 AUDIT CarbonDataRDDFactory$:
[gateway-dc1][hdfs][Thread-1]Data
load is successful for default.test_table

res1: org.apache.spark.sql.DataFrame = []





scala> carbon.sql("Select * from test_carbon").show()

java.io.FileNotFoundException: File
/test/carbondata/default/test_table/Fact/Part0/Segment_0
does not exist.

  at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(
DistributedFileSystem.java:1081)

  at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(
DistributedFileSystem.java:1059)

  at org.apache.hadoop.hdfs.DistributedFileSystem$23.
doCall(DistributedFileSystem.java:1004)

  at org.apache.hadoop.hdfs.DistributedFileSystem$23.
doCall(DistributedFileSystem.java:1000)

  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
FileSystemLinkResolver.java:81)

  at org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(
DistributedFileSystem.java:1000)

  at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1735)

  at org.apache.carbondata.hadoop.CarbonInputFormat.getFileStatusInternal(
CarbonInputFormat.java:862)

  at org.apache.carbondata.hadoop.CarbonInputFormat.getFileStatus(
CarbonInputFormat.java:845)

  at org.apache.carbondata.hadoop.CarbonInputFormat.listStatus(
CarbonInputFormat.java:802)

  at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.
getSplits(FileInputFormat.java:387)

  at org.apache.carbondata.hadoop.CarbonInputFormat.getSplitsInternal(
CarbonInputFormat.java:319)

  at org.apache.carbondata.hadoop.CarbonInputFormat.getTableBlockInfo(
CarbonInputFormat.java:523)

  at org.apache.carbondata.hadoop.CarbonInputFormat.
getSegmentAbstractIndexs(CarbonInputFormat.java:616)

  at org.apache.carbondata.hadoop.CarbonInputFormat.getDataBlocksOfSegment(
CarbonInputFormat.java:441)

  at org.apache.carbondata.hadoop.CarbonInputFormat.getSplits(
CarbonInputFormat.java:379)

  at org.apache.carbondata.hadoop.CarbonInputFormat.getSplits(
CarbonInputFormat.java:302)

  at org.apache.carbondata.spark.rdd.CarbonScanRDD.
getPartitions(CarbonScanRDD.scala:81)

  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)

  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)

  at scala.Option.getOrElse(Option.scala:121)

  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)

  at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(
MapPartitionsRDD.scala:35)

  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)

  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)

  at scala.Option.getOrElse(Option.scala:121)

  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)

  at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(
MapPartitionsRDD.scala:35)

  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)

  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)

  at scala.Option.getOrElse(Option.scala:121)

  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)

  at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:
311)

  at org.apache.spark.sql.execution.CollectLimitExec.
executeCollect(limit.scala:38)

  at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$
Dataset$$execute$1$1.apply(Dataset.scala:2378)

  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(
SQLExecution.scala:57)

  at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2780)

  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$
execute$1(Dataset.scala:2377)

  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$
collect(Dataset.scala:2384)

  at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2120)

  at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2119)

  at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2810)

  at org.apache.spark.sql.Dataset.head(Dataset.scala:2119)

  at org.apache.spark.sql.Dataset.take(Dataset.scala:2334)

  at org.apache.spark.sql.Dataset.showString(Dataset.scala:248)

 at org.apache.spark.sql.Dataset.show(Dataset.scala:638)

  at org.apache.spark.sql.Dataset.show(Dataset.scala:597)

  at org.apache.spark.sql.Dataset.show(Dataset.scala:606)

  ... 50 elided



I have check the folder on HDFS and there is a structure
/test/carbondata/default/test_carbon/ but the folder is empty.


I’m pretty sure that I’m missing silly, but I have not been able to find a
way to insert data in the table.



On another subject, I’m trying to also access this through presto, but here
the error is always: Query 20170726_145207_00005_ytsnk failed: line 1:1:
Schema 'default' does not exist



I’m also a little bit lost as from Spark it seems that the table are
created in the hive metastore, but the Presto plugin doesn’t seem to refer
to it.



Thanks for reading!


AG

Mime
View raw message