carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "xuchuanyin (JIRA)" <>
Subject [jira] [Created] (CARBONDATA-1690) Query failed after swap table by renaming
Date Thu, 09 Nov 2017 12:49:01 GMT
xuchuanyin created CARBONDATA-1690:

             Summary: Query failed after swap table by renaming
                 Key: CARBONDATA-1690
             Project: CarbonData
          Issue Type: Bug
          Components: spark-integration
    Affects Versions: 1.3.0
            Reporter: xuchuanyin
            Assignee: xuchuanyin
             Fix For: 1.3.0


I encountered query error after swap table by renaming table. Steps to reproduce this bug
are listed as below.

These steps work fine:

2. LOAD DATA TO `t1`;
4. LOAD DATA TO `t2`;
5. RENAME `t1` TO `t3`;
6. RENAME `t2` TO `t1`;
7. QUERY `t1`;

These steps work wrong:

2. LOAD DATA TO `t1`;
4. LOAD DATA TO `t2`;
**5. QUERY `t1`;**   --- Added this step
6. RENAME `t1` TO `t3`;
7. RENAME `t2` TO `t1`;
8. QUERY `t1`;   --- This step will cause failure

The above two scenario differs from that the second one add Step5 and the error will be thrown
in Step8. The error message in sparksql shell looks like
Error: File hdfs://slave1:9000/carbonstore/default/test_table/Fact/Part0/Segment_0/part-0-0_batchno0-0-1510144676427.carbondata
does not exist. (state=,code=0)

# Analyze

Renaming table name in carbondata actually is done through renaming the corresponding data
folder name. In addition, carbondata also refresh the metadata and its cache.

Having seen from the error message above, we find that the file name is exactly the one before
rename operation. We guess the problems may lies in data map.

In the second scenario, before renaming, when we query `t1 ` (Step5), the corresponding data
map will be loaded and cached. Since data map is table name based, when we query `t1` again
(Step8) after renaming, the previous data map will be used, which is outdated and incorrect,
thus will cause the `FileNotFoundException` error.

In the first scenario, when we query `t1` (Step7), it is the first time to load the data map,
so the correct data will be readed, that's why it acts OK.

# Resolve

There are two ways to fix this bug:

1. Change the index key of Data Map. Use `table_name + table_schema_last_update_time` in replace
of `table_name`.

2. Clear corresponding Data Map when doing renaming operation.

I prefer the second one since it is easy to implement —— just one line of code.

This message was sent by Atlassian JIRA

View raw message