carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kingsley Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CARBONDATA-1168) Driver Delete data operation is failed due to failure in creating delete delta file for segment
Date Mon, 19 Jun 2017 00:59:00 GMT

    [ https://issues.apache.org/jira/browse/CARBONDATA-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053380#comment-16053380
] 

Kingsley Chen commented on CARBONDATA-1168:
-------------------------------------------

I tested it in the version of spark1.6+CarbonData-1.0.0-incubating, it can process 100,000+
in one batch. So is that a bug of version 1.1.0?? Is there anyone can help me???

> Driver Delete data operation is failed due to failure in creating delete delta file for
segment
> -----------------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-1168
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1168
>             Project: CarbonData
>          Issue Type: Bug
>          Components: sql
>    Affects Versions: 1.1.0
>         Environment: spark1.6+carbon1.1.0
> we have a 20 nodes cluster and 32GB ram each
>            Reporter: Kingsley Chen
>              Labels: features
>             Fix For: NONE
>
>
> We use spark code to delete data from table as below
> ------------------spark code----------------------
> val deleteSql = s"DELETE FROM $tableName WHERE $rowkeyName IN (${rowKeyVals.mkString(",")})"
>     cc.sql(deleteSql).show()
> ------------------spark code----------------------
> when the array size of rowKeyVals is larger than 200, the delete operation will be failed,
and print the log:
> Delete data request has been received for default.item
> Delete data operation is failed for default.item
> Driver Delete data operation is failed due to failure in creating delete delta file for
segment : null block : null
> ++
> ||
> ++
> ++
> That is to say, it only delete success maximun at 200 a batch,and took about 1min which
is
> too slow. So my question is how to tuning the performance to make the batch larger 
> and delete faster



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message