carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravindra Pesala (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CARBONDATA-650) Columns switching error in performing the string functions
Date Tue, 17 Jan 2017 13:23:26 GMT

     [ https://issues.apache.org/jira/browse/CARBONDATA-650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravindra Pesala updated CARBONDATA-650:
---------------------------------------
    Assignee: QiangCai

> Columns switching error in performing the string functions
> ----------------------------------------------------------
>
>                 Key: CARBONDATA-650
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-650
>             Project: CarbonData
>          Issue Type: Bug
>          Components: data-load
>    Affects Versions: 1.1.0
>         Environment: Spark 2.1 Cluster mode
>            Reporter: Harsh Sharma
>            Assignee: QiangCai
>            Priority: Minor
>              Labels: bug
>         Attachments: Driver Logs
>
>
> We tried to execute the string function ASCII(COLUMN_NAME) on the Carbon Table in insert
query with the following scenario,
> - Create carbon tables as :
> CREATE TABLE target_uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string,
DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1
decimal(30,10), DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES ("TABLE_BLOCKSIZE"= "256 MB");
> CREATE TABLE source_uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string,
DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1
decimal(30,10), DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES ("TABLE_BLOCKSIZE"= "256 MB");
>  - Load records into source carbon table as :
> LOAD DATA inpath 'hdfs://hadoop-master:54311/data/uniqdata/2000_UniqData.csv' INTO table
source_uniqdata options('DELIMITER'=',', 'FILEHEADER'='CUST_ID, CUST_NAME, ACTIVE_EMUI_VERSION,
DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2, DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1,
Double_COLUMN2, INTEGER_COLUMN1');
>  - Insert the records of source table into target table and apply the string function
> insert into table target_uniqdata select CUST_ID, ASCII(CUST_NAME), ACTIVE_EMUI_VERSION,
DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2, DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1,
Double_COLUMN2, INTEGER_COLUMN1 from source_uniqdata;
> Below is the order of columns in the source_uniqdata table,
> --+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+
> |    CUST_NAME     |    ACTIVE_EMUI_VERSION     |          DOB           |          DOJ
          | CUST_ID  | BIGINT_COLUMN1  | BIGINT_COLUMN2  |     DECIMAL_COLUMN1     |     DECIMAL_COLUMN2
    |    Double_COLUMN1    |    Double_COLUMN2     | INTEGER_COLUMN1  |
> +------------------+----------------------------+------------------------+------------------------+----------+-----------------+-----------------+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+
> | CUST_NAME_00987  | ACTIVE_EMUI_VERSION_00987  | 1972-09-14 01:00:03.0  | 1972-09-14
02:00:03.0  | 9987     | 123372037841    | -223372035867   | 12345679888.1234000000  | 22345679888.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 988              |
> | CUST_NAME_00988  | ACTIVE_EMUI_VERSION_00988  | 1972-09-15 01:00:03.0  | 1972-09-15
02:00:03.0  | 9988     | 123372037842    | -223372035866   | 12345679889.1234000000  | 22345679889.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 989              |
> | CUST_NAME_00989  | ACTIVE_EMUI_VERSION_00989  | 1972-09-16 01:00:03.0  | 1972-09-16
02:00:03.0  | 9989     | 123372037843    | -223372035865   | 12345679890.1234000000  | 22345679890.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 990              |
> | CUST_NAME_00990  | ACTIVE_EMUI_VERSION_00990  | 1972-09-17 01:00:03.0  | 1972-09-17
02:00:03.0  | 9990     | 123372037844    | -223372035864   | 12345679891.1234000000  | 22345679891.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 991              |
> | CUST_NAME_00991  | ACTIVE_EMUI_VERSION_00991  | 1972-09-18 01:00:03.0  | 1972-09-18
02:00:03.0  | 9991     | 123372037845    | -223372035863   | 12345679892.1234000000  | 22345679892.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 992              |
> | CUST_NAME_00992  | ACTIVE_EMUI_VERSION_00992  | 1972-09-19 01:00:03.0  | 1972-09-19
02:00:03.0  | 9992     | 123372037846    | -223372035862   | 12345679893.1234000000  | 22345679893.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 993              |
> | CUST_NAME_00993  | ACTIVE_EMUI_VERSION_00993  | 1972-09-20 01:00:03.0  | 1972-09-20
02:00:03.0  | 9993     | 123372037847    | -223372035861   | 12345679894.1234000000  | 22345679894.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 994              |
> | CUST_NAME_00994  | ACTIVE_EMUI_VERSION_00994  | 1972-09-21 01:00:03.0  | 1972-09-21
02:00:03.0  | 9994     | 123372037848    | -223372035860   | 12345679895.1234000000  | 22345679895.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 995              |
> | CUST_NAME_00995  | ACTIVE_EMUI_VERSION_00995  | 1972-09-22 01:00:03.0  | 1972-09-22
02:00:03.0  | 9995     | 123372037849    | -223372035859   | 12345679896.1234000000  | 22345679896.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 996              |
> | CUST_NAME_00996  | ACTIVE_EMUI_VERSION_00996  | 1972-09-23 01:00:03.0  | 1972-09-23
02:00:03.0  | 9996     | 123372037850    | -223372035858   | 12345679897.1234000000  | 22345679897.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 997              |
> | CUST_NAME_00997  | ACTIVE_EMUI_VERSION_00997  | 1972-09-24 01:00:03.0  | 1972-09-24
02:00:03.0  | 9997     | 123372037851    | -223372035857   | 12345679898.1234000000  | 22345679898.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 998              |
> | CUST_NAME_00998  | ACTIVE_EMUI_VERSION_00998  | 1972-09-25 01:00:03.0  | 1972-09-25
02:00:03.0  | 9998     | 123372037852    | -223372035856   | 12345679899.1234000000  | 22345679899.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 999              |
> | CUST_NAME_00999  | ACTIVE_EMUI_VERSION_00999  | 1972-09-26 01:00:03.0  | 1972-09-26
02:00:03.0  | 9999     | 123372037853    | -223372035855   | 12345679900.1234000000  | 22345679900.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1000             |
> ::::: AFTER ASCII OPERATION ::::SPARK 2.1::::::
> -+-------------------------+----------------------+-----------------------+------------------+--+
> | CUST_NAME  | ACTIVE_EMUI_VERSION  |  DOB  |          DOJ           |  CUST_ID   | BIGINT_COLUMN1
 | BIGINT_COLUMN2  |     DECIMAL_COLUMN1     |     DECIMAL_COLUMN2     |    Double_COLUMN1
   |    Double_COLUMN2     | INTEGER_COLUMN1  |
> +------------+----------------------+-------+------------------------+------------+-----------------+-----------------+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+
> | 9987       | 67                   | NULL  | 1972-09-14 01:00:03.0  | 85264203   | 123372037841
   | -223372035867   | 12345679888.1234000000  | 22345679888.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 988              |
> | 9988       | 67                   | NULL  | 1972-09-15 01:00:03.0  | 85350603   | 123372037842
   | -223372035866   | 12345679889.1234000000  | 22345679889.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 989              |
> | 9989       | 67                   | NULL  | 1972-09-16 01:00:03.0  | 85437003   | 123372037843
   | -223372035865   | 12345679890.1234000000  | 22345679890.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 990              |
> | 9990       | 67                   | NULL  | 1972-09-17 01:00:03.0  | 85523403   | 123372037844
   | -223372035864   | 12345679891.1234000000  | 22345679891.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 991              |
> | 9991       | 67                   | NULL  | 1972-09-18 01:00:03.0  | 85609803   | 123372037845
   | -223372035863   | 12345679892.1234000000  | 22345679892.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 992              |
> | 9992       | 67                   | NULL  | 1972-09-19 01:00:03.0  | 85696203   | 123372037846
   | -223372035862   | 12345679893.1234000000  | 22345679893.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 993              |
> | 9993       | 67                   | NULL  | 1972-09-20 01:00:03.0  | 85782603   | 123372037847
   | -223372035861   | 12345679894.1234000000  | 22345679894.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 994              |
> | 9994       | 67                   | NULL  | 1972-09-21 01:00:03.0  | 85869003   | 123372037848
   | -223372035860   | 12345679895.1234000000  | 22345679895.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 995              |
> | 9995       | 67                   | NULL  | 1972-09-22 01:00:03.0  | 85955403   | 123372037849
   | -223372035859   | 12345679896.1234000000  | 22345679896.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 996              |
> | 9996       | 67                   | NULL  | 1972-09-23 01:00:03.0  | 86041803   | 123372037850
   | -223372035858   | 12345679897.1234000000  | 22345679897.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 997              |
> | 9997       | 67                   | NULL  | 1972-09-24 01:00:03.0  | 86128203   | 123372037851
   | -223372035857   | 12345679898.1234000000  | 22345679898.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 998              |
> | 9998       | 67                   | NULL  | 1972-09-25 01:00:03.0  | 86214603   | 123372037852
   | -223372035856   | 12345679899.1234000000  | 22345679899.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 999              |
> | 9999       | 67                   | NULL  | 1972-09-26 01:00:03.0  | 86301003   | 123372037853
   | -223372035855   | 12345679900.1234000000  | 22345679900.1234000000  | 1.12345674897976E10
 | -1.12345674897976E10  | 1000             |
> +------------+----------------------+-------+------------------------+------------+-----------------+-----------------+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+
> As you can see the values of the columns have been switched.
> ::::: AFTER ASCII OPERATION ::::SPARK 1.6::::::
> +----------+------------+----------------------------+------------------------+------------------------+-----------------+-----------------+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+
> | cust_id  | cust_name  |    active_emui_version     |          dob           |     
    doj           | bigint_column1  | bigint_column2  |     decimal_column1     |     decimal_column2
    |    double_column1    |    double_column2     | integer_column1  |
> +----------+------------+----------------------------+------------------------+------------------------+-----------------+-----------------+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+
> | 10801    | 67         | ACTIVE_EMUI_VERSION_01801  | 1974-12-07 01:00:03.0  | 1974-12-07
02:00:03.0  | 123372038655    | -223372035053   | 12345680702.1234000000  | 22345680702.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1802             |
> | 10802    | 67         | ACTIVE_EMUI_VERSION_01802  | 1974-12-08 01:00:03.0  | 1974-12-08
02:00:03.0  | 123372038656    | -223372035052   | 12345680703.1234000000  | 22345680703.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1803             |
> | 10803    | 67         | ACTIVE_EMUI_VERSION_01803  | 1974-12-09 01:00:03.0  | 1974-12-09
02:00:03.0  | 123372038657    | -223372035051   | 12345680704.1234000000  | 22345680704.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1804             |
> | 10804    | 67         | ACTIVE_EMUI_VERSION_01804  | 1974-12-10 01:00:03.0  | 1974-12-10
02:00:03.0  | 123372038658    | -223372035050   | 12345680705.1234000000  | 22345680705.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1805             |
> | 10805    | 67         | ACTIVE_EMUI_VERSION_01805  | 1974-12-11 01:00:03.0  | 1974-12-11
02:00:03.0  | 123372038659    | -223372035049   | 12345680706.1234000000  | 22345680706.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1806             |
> | 10806    | 67         | ACTIVE_EMUI_VERSION_01806  | 1974-12-12 01:00:03.0  | 1974-12-12
02:00:03.0  | 123372038660    | -223372035048   | 12345680707.1234000000  | 22345680707.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1807             |
> | 10807    | 67         | ACTIVE_EMUI_VERSION_01807  | 1974-12-13 01:00:03.0  | 1974-12-13
02:00:03.0  | 123372038661    | -223372035047   | 12345680708.1234000000  | 22345680708.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1808             |
> | 10808    | 67         | ACTIVE_EMUI_VERSION_01808  | 1974-12-14 01:00:03.0  | 1974-12-14
02:00:03.0  | 123372038662    | -223372035046   | 12345680709.1234000000  | 22345680709.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1809             |
> | 10809    | 67         | ACTIVE_EMUI_VERSION_01809  | 1974-12-15 01:00:03.0  | 1974-12-15
02:00:03.0  | 123372038663    | -223372035045   | 12345680710.1234000000  | 22345680710.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1810             |
> | 10810    | 67         | ACTIVE_EMUI_VERSION_01810  | 1974-12-16 01:00:03.0  | 1974-12-16
02:00:03.0  | 123372038664    | -223372035044   | 12345680711.1234000000  | 22345680711.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1811             |
> | 10811    | 67         | ACTIVE_EMUI_VERSION_01811  | 1974-12-17 01:00:03.0  | 1974-12-17
02:00:03.0  | 123372038665    | -223372035043   | 12345680712.1234000000  | 22345680712.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1812             |
> | 10812    | 67         | ACTIVE_EMUI_VERSION_01812  | 1974-12-18 01:00:03.0  | 1974-12-18
02:00:03.0  | 123372038666    | -223372035042   | 12345680713.1234000000  | 22345680713.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1813             |
> | 10813    | 67         | ACTIVE_EMUI_VERSION_01813  | 1974-12-19 01:00:03.0  | 1974-12-19
02:00:03.0  | 123372038667    | -223372035041   | 12345680714.1234000000  | 22345680714.1234000000
 | 1.12345674897976E10  | -1.12345674897976E10  | 1814             |
> Here, its working in Spark 1.6



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message