carbondata-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravindra Pesala <ravi.pes...@gmail.com>
Subject Re: get error when load tpcds data catalog_returns
Date Thu, 24 Aug 2017 09:04:32 GMT
Hi,

Which version of carbon and spark are you using? How much data are you
loading and what is the machine configuration?

I have tried loading catlog_returns with 20 MB data in my local machine and
it is successful. I used the latest master branch and spark-2.1 version.

Also please send the complete log as the log information you provided does
not say the actual cause.

Regards,
Ravindra.

On 24 August 2017 at 14:02, lk_hadoop <lk_hadoop@163.com> wrote:

> hi,all
>        I want to test carbondata by using tpc-ds data , I try to load
> table : catalog_returns
>        I got error :
>        org.apache.carbondata.processing.newflow.exception.CarbonDataLoadingException:
> There is an unexpected error: unable to generate the mdkey
>  at org.apache.carbondata.processing.newflow.steps.
> DataWriterProcessorStepImpl.execute(DataWriterProcessorStepImpl.java:125)
>  at org.apache.carbondata.processing.newflow.DataLoadExecutor.execute(
> DataLoadExecutor.java:48)
>  at org.apache.carbondata.spark.rdd.NewCarbonDataLoadRDD$$anon$1.<init>(
> NewCarbonDataLoadRDD.scala:243)
>  at org.apache.carbondata.spark.rdd.NewCarbonDataLoadRDD.
> compute(NewCarbonDataLoadRDD.scala:220)
>  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
>  at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.carbondata.processing.newflow.exception.CarbonDataLoadingException:
> unable to generate the mdkey
>  at org.apache.carbondata.processing.newflow.steps.
> DataWriterProcessorStepImpl.processBatch(DataWriterProcessorStepImpl.
> java:181)
>  at org.apache.carbondata.processing.newflow.steps.
> DataWriterProcessorStepImpl.execute(DataWriterProcessorStepImpl.java:111)
>  ... 11 more
> Caused by: java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@67098e0 rejected from
> java.util.concurrent.ThreadPoolExecutor@5b91b608[Shutting down, pool size
> = 1, active threads = 1, queued tasks = 0, completed tasks = 24]
>  at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(
> ThreadPoolExecutor.java:2047)
>  at java.util.concurrent.ThreadPoolExecutor.reject(
> ThreadPoolExecutor.java:823)
>  at java.util.concurrent.ThreadPoolExecutor.execute(
> ThreadPoolExecutor.java:1369)
>  at java.util.concurrent.AbstractExecutorService.submit(
> AbstractExecutorService.java:134)
>  at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.
> addDataToStore(CarbonFactDataHandlerColumnar.java:466)
>  at org.apache.carbondata.processing.newflow.steps.
> DataWriterProcessorStepImpl.processBatch(DataWriterProcessorStepImpl.
> java:178)
>  ... 12 more
>
> my table creat with sql :
> create table if not exists tpcds_carbon.catalog_returns
> (
>     cr_returned_date_sk       int                       ,
>     cr_returned_time_sk       int                       ,
>     cr_item_sk                int               ,
>     cr_refunded_customer_sk   int                       ,
>     cr_refunded_cdemo_sk      int                       ,
>     cr_refunded_hdemo_sk      int                       ,
>     cr_refunded_addr_sk       int                       ,
>     cr_returning_customer_sk  int                       ,
>     cr_returning_cdemo_sk     int                       ,
>     cr_returning_hdemo_sk     int                       ,
>     cr_returning_addr_sk      int                       ,
>     cr_call_center_sk         int                       ,
>     cr_catalog_page_sk        int                       ,
>     cr_ship_mode_sk           int                       ,
>     cr_warehouse_sk           int                       ,
>     cr_reason_sk              int                       ,
>     cr_order_number           int               ,
>     cr_return_quantity        int                       ,
>     cr_return_amount          decimal(7,2)                  ,
>     cr_return_tax             decimal(7,2)                  ,
>     cr_return_amt_inc_tax     decimal(7,2)                  ,
>     cr_fee                    decimal(7,2)                  ,
>     cr_return_ship_cost       decimal(7,2)                  ,
>     cr_refunded_cash          decimal(7,2)                  ,
>     cr_reversed_charge        decimal(7,2)                  ,
>     cr_store_credit           decimal(7,2)                  ,
>     cr_net_loss               decimal(7,2)
> )STORED BY 'carbondata'
> TBLPROPERTIES ('DICTIONARY_INCLUDE'='cr_item_sk,cr_order_number')
> //because these two cols are the real PK
>
> and I load data with :
>
> carbon.sql("load data inpath 'hdfs://AAA:9000/tpcds/source/
> catalog_returns/catalog_returns_1_4.dat' into table
> carbon_catalog_returns2 OPTIONS('DELIMITER'='|','
> fileheader'='cr_returned_date_sk,cr_returned_time_sk,cr_
> item_sk,cr_refunded_customer_sk,cr_refunded_cdemo_sk,cr_
> refunded_hdemo_sk,cr_refunded_addr_sk,cr_returning_customer_
> sk,cr_returning_cdemo_sk,cr_returning_hdemo_sk,cr_
> returning_addr_sk,cr_call_center_sk,cr_catalog_page_sk,
> cr_ship_mode_sk,cr_warehouse_sk,cr_reason_sk,cr_order_
> number,cr_return_quantity,cr_return_amount,cr_return_tax,
> cr_return_amt_inc_tax,cr_fee,cr_return_ship_cost,cr_
> refunded_cash,cr_reversed_charge,cr_store_credit,cr_net_loss')")
>
> any one know what was wrong?
>
>
>
>
> 2017-08-24
> ------------------------------
> lk_hadoop
>



-- 
Thanks & Regards,
Ravi

Mime
View raw message