carbondata-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "lk_hadoop"<lk_had...@163.com>
Subject get error when load tpcds data catalog_returns
Date Thu, 24 Aug 2017 08:32:23 GMT
hi,all
       I want to test carbondata by using tpc-ds data , I try to load table : catalog_returns

       I got error : 
       org.apache.carbondata.processing.newflow.exception.CarbonDataLoadingException: There
is an unexpected error: unable to generate the mdkey
 at org.apache.carbondata.processing.newflow.steps.DataWriterProcessorStepImpl.execute(DataWriterProcessorStepImpl.java:125)
 at org.apache.carbondata.processing.newflow.DataLoadExecutor.execute(DataLoadExecutor.java:48)
 at org.apache.carbondata.spark.rdd.NewCarbonDataLoadRDD$$anon$1.<init>(NewCarbonDataLoadRDD.scala:243)
 at org.apache.carbondata.spark.rdd.NewCarbonDataLoadRDD.compute(NewCarbonDataLoadRDD.scala:220)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
 at org.apache.spark.scheduler.Task.run(Task.scala:99)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.carbondata.processing.newflow.exception.CarbonDataLoadingException:
unable to generate the mdkey
 at org.apache.carbondata.processing.newflow.steps.DataWriterProcessorStepImpl.processBatch(DataWriterProcessorStepImpl.java:181)
 at org.apache.carbondata.processing.newflow.steps.DataWriterProcessorStepImpl.execute(DataWriterProcessorStepImpl.java:111)
 ... 11 more
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@67098e0
rejected from java.util.concurrent.ThreadPoolExecutor@5b91b608[Shutting down, pool size =
1, active threads = 1, queued tasks = 0, completed tasks = 24]
 at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
 at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
 at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
 at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
 at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.addDataToStore(CarbonFactDataHandlerColumnar.java:466)
 at org.apache.carbondata.processing.newflow.steps.DataWriterProcessorStepImpl.processBatch(DataWriterProcessorStepImpl.java:178)
 ... 12 more

my table creat with sql :
create table if not exists tpcds_carbon.catalog_returns
(
    cr_returned_date_sk       int                       ,
    cr_returned_time_sk       int                       ,
    cr_item_sk                int               ,
    cr_refunded_customer_sk   int                       ,
    cr_refunded_cdemo_sk      int                       ,
    cr_refunded_hdemo_sk      int                       ,
    cr_refunded_addr_sk       int                       ,
    cr_returning_customer_sk  int                       ,
    cr_returning_cdemo_sk     int                       ,
    cr_returning_hdemo_sk     int                       ,
    cr_returning_addr_sk      int                       ,
    cr_call_center_sk         int                       ,
    cr_catalog_page_sk        int                       ,
    cr_ship_mode_sk           int                       ,
    cr_warehouse_sk           int                       ,
    cr_reason_sk              int                       ,
    cr_order_number           int               ,
    cr_return_quantity        int                       ,
    cr_return_amount          decimal(7,2)                  ,
    cr_return_tax             decimal(7,2)                  ,
    cr_return_amt_inc_tax     decimal(7,2)                  ,
    cr_fee                    decimal(7,2)                  ,
    cr_return_ship_cost       decimal(7,2)                  ,
    cr_refunded_cash          decimal(7,2)                  ,
    cr_reversed_charge        decimal(7,2)                  ,
    cr_store_credit           decimal(7,2)                  ,
    cr_net_loss               decimal(7,2)   
)STORED BY 'carbondata' 
TBLPROPERTIES ('DICTIONARY_INCLUDE'='cr_item_sk,cr_order_number')    //because these two cols
are the real PK

and I load data with :

carbon.sql("load data inpath 'hdfs://AAA:9000/tpcds/source/catalog_returns/catalog_returns_1_4.dat'
into table carbon_catalog_returns2 OPTIONS('DELIMITER'='|','fileheader'='cr_returned_date_sk,cr_returned_time_sk,cr_item_sk,cr_refunded_customer_sk,cr_refunded_cdemo_sk,cr_refunded_hdemo_sk,cr_refunded_addr_sk,cr_returning_customer_sk,cr_returning_cdemo_sk,cr_returning_hdemo_sk,cr_returning_addr_sk,cr_call_center_sk,cr_catalog_page_sk,cr_ship_mode_sk,cr_warehouse_sk,cr_reason_sk,cr_order_number,cr_return_quantity,cr_return_amount,cr_return_tax,cr_return_amt_inc_tax,cr_fee,cr_return_ship_cost,cr_refunded_cash,cr_reversed_charge,cr_store_credit,cr_net_loss')")

any one know what was wrong?




2017-08-24


lk_hadoop 
Mime
View raw message