carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravindra Pesala (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CARBONDATA-1990) Null values shown when the basic word count example is tried on carbon streaming table
Date Sat, 03 Feb 2018 10:16:00 GMT

     [ https://issues.apache.org/jira/browse/CARBONDATA-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravindra Pesala updated CARBONDATA-1990:
----------------------------------------
    Fix Version/s:     (was: 1.3.0)
                   1.4.0

> Null values shown when the basic word count example is tried on carbon streaming table
> --------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-1990
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1990
>             Project: CarbonData
>          Issue Type: Bug
>          Components: spark-integration
>    Affects Versions: 1.3.0
>         Environment: spark-2.1
>            Reporter: anubhav tarar
>            Priority: Major
>             Fix For: 1.4.0
>
>
> import org.apache.spark.sql.SparkSession
> import org.apache.spark.sql.CarbonSession._
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.{AnalysisException, DataFrame, DataFrameWriter, Row, SaveMode}
> val carbon = SparkSession.builder().config(sc.getConf) .getOrCreateCarbonSession("hdfs://localhost:54311/newCarbonStore","/tmp")
>  carbon.sql("drop table if exists stream_table_csvSource")
>  carbon.sql("create table stream_table_csvSource(word string,count string) stored by
'carbondata' TBLPROPERTIES('streaming'='true')")
> import carbon.sqlContext.implicits._
> val userSchema = StructType(
> Array(StructField("name", StringType)))
> val lines = carbon.readStream
> .format("csv")
> .option("path", "file:///home/anubhav/Documents/csv")
> .option("sep",",")
> .schema(userSchema)
> .load()
> lines.printSchema
> val words = lines.as[String].flatMap(_.split("\n"))
> val wordCounts = words.groupBy("value").count()
> val qry = wordCounts.writeStream
>               .outputMode("complete")
>              .format("carbondata")
>              .option("checkpointLocation","file/home/anubhav/Downloads/checkpointlist155")
>             .option("dbName", "default")
>             .option("tableName", "stream_table_csvSource")
>             .start()
>           qry.awaitTermination()
> then i close my shell and select data from table there are null values shown  
> scala> carbon.sql("select * from stream_table_csvSource").show
> 18/01/05 14:14:34 ERROR CarbonProperties: Executor task launch worker-0 Configured value
for property carbon.number.of.cores.while.loading is wrong. Falling back to the default value
2
> +-------+-----+
> |   word|count|
> +-------+-----+
> |anubhav| null|
> |geetika| null|
> | prince| null|
> +-------+-----+



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message