carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Babulal (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CARBONDATA-406) Empty Folder is created when data load from dataframe
Date Thu, 10 Nov 2016 15:57:58 GMT
Babulal created CARBONDATA-406:
----------------------------------

             Summary: Empty Folder is created when data load from dataframe
                 Key: CARBONDATA-406
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-406
             Project: CarbonData
          Issue Type: Bug
          Components: data-load
    Affects Versions: 0.1.0-incubating
            Reporter: Babulal
            Priority: Trivial


Load the data from dataframe to carbon table with tempCSV=false option .
Load is success but emtyFolder is getting created  in HDFS 
Cluster size= 3 node .
Type:- Stanalone Spark 


Steps
 val customSchema = StructType(Array(    StructField("imei", StringType, true),    StructField("deviceInformationId",
IntegerType, true),    StructField("mac", StringType, true),    StructField("productdate",
TimestampType , true),    StructField("updatetime", TimestampType, true),    StructField("gamePointId",
DoubleType, true),    StructField("contractNumber", DoubleType, true)       ));


val df = cc.read.format("com.databricks.spark.csv").option("header", "false").schema(customSchema).load("/opt/data/xyz/100_default_date_11_header.csv");

Start data loading 
scala> df.write.format("carbondata").option("tableName","mycarbon2").save();


Check Logs

leges:{}, groupPrivileges:null, rolePrivileges:null))
INFO  10-11 23:52:44,005 - Creating directory if it doesn't exist: hdfs://10.18.102.236:54310/opt/Carbon/Spark/spark/bin/null/bin/carbonshellstore/hivemetadata/mycarbon4
AUDIT 10-11 23:52:44,037 - [BLR1000007781][root][Thread-1]Table created with Database name
[default] and Table name [mycarbon4]
INFO  10-11 23:52:44,040 - Successfully able to get the table metadata file lock


In the HDFS this Path is empty 
hdfs://10.18.102.236:54310/opt/Carbon/Spark/spark/bin/null/bin/carbonshellstore/hivemetadata/mycarbon4

Actual Store location is :- hdfs://10.18.102.236:54310/opt/Carbon/mystore

Expect :- Empty folder should not be created. . It seems that it is created in  SPARK_HOME/bin
 . 
SPARK_HOME is /opt/Carbon/Spark/spark/bin




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message