Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 65F0A200BFF for ; Tue, 17 Jan 2017 14:23:34 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 646D7160B43; Tue, 17 Jan 2017 13:23:34 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 61AC7160B46 for ; Tue, 17 Jan 2017 14:23:33 +0100 (CET) Received: (qmail 6149 invoked by uid 500); 17 Jan 2017 13:23:32 -0000 Mailing-List: contact issues-help@carbondata.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@carbondata.incubator.apache.org Delivered-To: mailing list issues@carbondata.incubator.apache.org Received: (qmail 6133 invoked by uid 99); 17 Jan 2017 13:23:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Jan 2017 13:23:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 21B08180BB7 for ; Tue, 17 Jan 2017 13:23:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.199 X-Spam-Level: X-Spam-Status: No, score=-1.199 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id EK7SKXazvq0g for ; Tue, 17 Jan 2017 13:23:29 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id AE03F5F4AC for ; Tue, 17 Jan 2017 13:23:28 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id C287EE20CD for ; Tue, 17 Jan 2017 13:23:27 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 8DA2725286 for ; Tue, 17 Jan 2017 13:23:26 +0000 (UTC) Date: Tue, 17 Jan 2017 13:23:26 +0000 (UTC) From: "Ravindra Pesala (JIRA)" To: issues@carbondata.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CARBONDATA-650) Columns switching error in performing the string functions MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 17 Jan 2017 13:23:34 -0000 [ https://issues.apache.org/jira/browse/CARBONDATA-650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala updated CARBONDATA-650: --------------------------------------- Assignee: QiangCai > Columns switching error in performing the string functions > ---------------------------------------------------------- > > Key: CARBONDATA-650 > URL: https://issues.apache.org/jira/browse/CARBONDATA-650 > Project: CarbonData > Issue Type: Bug > Components: data-load > Affects Versions: 1.1.0 > Environment: Spark 2.1 Cluster mode > Reporter: Harsh Sharma > Assignee: QiangCai > Priority: Minor > Labels: bug > Attachments: Driver Logs > > > We tried to execute the string function ASCII(COLUMN_NAME) on the Carbon Table in insert query with the following scenario, > - Create carbon tables as : > CREATE TABLE target_uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES ("TABLE_BLOCKSIZE"= "256 MB"); > CREATE TABLE source_uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES ("TABLE_BLOCKSIZE"= "256 MB"); > - Load records into source carbon table as : > LOAD DATA inpath 'hdfs://hadoop-master:54311/data/uniqdata/2000_UniqData.csv' INTO table source_uniqdata options('DELIMITER'=',', 'FILEHEADER'='CUST_ID, CUST_NAME, ACTIVE_EMUI_VERSION, DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2, DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1, Double_COLUMN2, INTEGER_COLUMN1'); > - Insert the records of source table into target table and apply the string function > insert into table target_uniqdata select CUST_ID, ASCII(CUST_NAME), ACTIVE_EMUI_VERSION, DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2, DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1, Double_COLUMN2, INTEGER_COLUMN1 from source_uniqdata; > Below is the order of columns in the source_uniqdata table, > --+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+ > | CUST_NAME | ACTIVE_EMUI_VERSION | DOB | DOJ | CUST_ID | BIGINT_COLUMN1 | BIGINT_COLUMN2 | DECIMAL_COLUMN1 | DECIMAL_COLUMN2 | Double_COLUMN1 | Double_COLUMN2 | INTEGER_COLUMN1 | > +------------------+----------------------------+------------------------+------------------------+----------+-----------------+-----------------+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+ > | CUST_NAME_00987 | ACTIVE_EMUI_VERSION_00987 | 1972-09-14 01:00:03.0 | 1972-09-14 02:00:03.0 | 9987 | 123372037841 | -223372035867 | 12345679888.1234000000 | 22345679888.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 988 | > | CUST_NAME_00988 | ACTIVE_EMUI_VERSION_00988 | 1972-09-15 01:00:03.0 | 1972-09-15 02:00:03.0 | 9988 | 123372037842 | -223372035866 | 12345679889.1234000000 | 22345679889.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 989 | > | CUST_NAME_00989 | ACTIVE_EMUI_VERSION_00989 | 1972-09-16 01:00:03.0 | 1972-09-16 02:00:03.0 | 9989 | 123372037843 | -223372035865 | 12345679890.1234000000 | 22345679890.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 990 | > | CUST_NAME_00990 | ACTIVE_EMUI_VERSION_00990 | 1972-09-17 01:00:03.0 | 1972-09-17 02:00:03.0 | 9990 | 123372037844 | -223372035864 | 12345679891.1234000000 | 22345679891.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 991 | > | CUST_NAME_00991 | ACTIVE_EMUI_VERSION_00991 | 1972-09-18 01:00:03.0 | 1972-09-18 02:00:03.0 | 9991 | 123372037845 | -223372035863 | 12345679892.1234000000 | 22345679892.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 992 | > | CUST_NAME_00992 | ACTIVE_EMUI_VERSION_00992 | 1972-09-19 01:00:03.0 | 1972-09-19 02:00:03.0 | 9992 | 123372037846 | -223372035862 | 12345679893.1234000000 | 22345679893.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 993 | > | CUST_NAME_00993 | ACTIVE_EMUI_VERSION_00993 | 1972-09-20 01:00:03.0 | 1972-09-20 02:00:03.0 | 9993 | 123372037847 | -223372035861 | 12345679894.1234000000 | 22345679894.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 994 | > | CUST_NAME_00994 | ACTIVE_EMUI_VERSION_00994 | 1972-09-21 01:00:03.0 | 1972-09-21 02:00:03.0 | 9994 | 123372037848 | -223372035860 | 12345679895.1234000000 | 22345679895.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 995 | > | CUST_NAME_00995 | ACTIVE_EMUI_VERSION_00995 | 1972-09-22 01:00:03.0 | 1972-09-22 02:00:03.0 | 9995 | 123372037849 | -223372035859 | 12345679896.1234000000 | 22345679896.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 996 | > | CUST_NAME_00996 | ACTIVE_EMUI_VERSION_00996 | 1972-09-23 01:00:03.0 | 1972-09-23 02:00:03.0 | 9996 | 123372037850 | -223372035858 | 12345679897.1234000000 | 22345679897.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 997 | > | CUST_NAME_00997 | ACTIVE_EMUI_VERSION_00997 | 1972-09-24 01:00:03.0 | 1972-09-24 02:00:03.0 | 9997 | 123372037851 | -223372035857 | 12345679898.1234000000 | 22345679898.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 998 | > | CUST_NAME_00998 | ACTIVE_EMUI_VERSION_00998 | 1972-09-25 01:00:03.0 | 1972-09-25 02:00:03.0 | 9998 | 123372037852 | -223372035856 | 12345679899.1234000000 | 22345679899.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 999 | > | CUST_NAME_00999 | ACTIVE_EMUI_VERSION_00999 | 1972-09-26 01:00:03.0 | 1972-09-26 02:00:03.0 | 9999 | 123372037853 | -223372035855 | 12345679900.1234000000 | 22345679900.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1000 | > ::::: AFTER ASCII OPERATION ::::SPARK 2.1:::::: > -+-------------------------+----------------------+-----------------------+------------------+--+ > | CUST_NAME | ACTIVE_EMUI_VERSION | DOB | DOJ | CUST_ID | BIGINT_COLUMN1 | BIGINT_COLUMN2 | DECIMAL_COLUMN1 | DECIMAL_COLUMN2 | Double_COLUMN1 | Double_COLUMN2 | INTEGER_COLUMN1 | > +------------+----------------------+-------+------------------------+------------+-----------------+-----------------+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+ > | 9987 | 67 | NULL | 1972-09-14 01:00:03.0 | 85264203 | 123372037841 | -223372035867 | 12345679888.1234000000 | 22345679888.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 988 | > | 9988 | 67 | NULL | 1972-09-15 01:00:03.0 | 85350603 | 123372037842 | -223372035866 | 12345679889.1234000000 | 22345679889.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 989 | > | 9989 | 67 | NULL | 1972-09-16 01:00:03.0 | 85437003 | 123372037843 | -223372035865 | 12345679890.1234000000 | 22345679890.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 990 | > | 9990 | 67 | NULL | 1972-09-17 01:00:03.0 | 85523403 | 123372037844 | -223372035864 | 12345679891.1234000000 | 22345679891.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 991 | > | 9991 | 67 | NULL | 1972-09-18 01:00:03.0 | 85609803 | 123372037845 | -223372035863 | 12345679892.1234000000 | 22345679892.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 992 | > | 9992 | 67 | NULL | 1972-09-19 01:00:03.0 | 85696203 | 123372037846 | -223372035862 | 12345679893.1234000000 | 22345679893.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 993 | > | 9993 | 67 | NULL | 1972-09-20 01:00:03.0 | 85782603 | 123372037847 | -223372035861 | 12345679894.1234000000 | 22345679894.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 994 | > | 9994 | 67 | NULL | 1972-09-21 01:00:03.0 | 85869003 | 123372037848 | -223372035860 | 12345679895.1234000000 | 22345679895.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 995 | > | 9995 | 67 | NULL | 1972-09-22 01:00:03.0 | 85955403 | 123372037849 | -223372035859 | 12345679896.1234000000 | 22345679896.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 996 | > | 9996 | 67 | NULL | 1972-09-23 01:00:03.0 | 86041803 | 123372037850 | -223372035858 | 12345679897.1234000000 | 22345679897.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 997 | > | 9997 | 67 | NULL | 1972-09-24 01:00:03.0 | 86128203 | 123372037851 | -223372035857 | 12345679898.1234000000 | 22345679898.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 998 | > | 9998 | 67 | NULL | 1972-09-25 01:00:03.0 | 86214603 | 123372037852 | -223372035856 | 12345679899.1234000000 | 22345679899.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 999 | > | 9999 | 67 | NULL | 1972-09-26 01:00:03.0 | 86301003 | 123372037853 | -223372035855 | 12345679900.1234000000 | 22345679900.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1000 | > +------------+----------------------+-------+------------------------+------------+-----------------+-----------------+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+ > As you can see the values of the columns have been switched. > ::::: AFTER ASCII OPERATION ::::SPARK 1.6:::::: > +----------+------------+----------------------------+------------------------+------------------------+-----------------+-----------------+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+ > | cust_id | cust_name | active_emui_version | dob | doj | bigint_column1 | bigint_column2 | decimal_column1 | decimal_column2 | double_column1 | double_column2 | integer_column1 | > +----------+------------+----------------------------+------------------------+------------------------+-----------------+-----------------+-------------------------+-------------------------+----------------------+-----------------------+------------------+--+ > | 10801 | 67 | ACTIVE_EMUI_VERSION_01801 | 1974-12-07 01:00:03.0 | 1974-12-07 02:00:03.0 | 123372038655 | -223372035053 | 12345680702.1234000000 | 22345680702.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1802 | > | 10802 | 67 | ACTIVE_EMUI_VERSION_01802 | 1974-12-08 01:00:03.0 | 1974-12-08 02:00:03.0 | 123372038656 | -223372035052 | 12345680703.1234000000 | 22345680703.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1803 | > | 10803 | 67 | ACTIVE_EMUI_VERSION_01803 | 1974-12-09 01:00:03.0 | 1974-12-09 02:00:03.0 | 123372038657 | -223372035051 | 12345680704.1234000000 | 22345680704.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1804 | > | 10804 | 67 | ACTIVE_EMUI_VERSION_01804 | 1974-12-10 01:00:03.0 | 1974-12-10 02:00:03.0 | 123372038658 | -223372035050 | 12345680705.1234000000 | 22345680705.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1805 | > | 10805 | 67 | ACTIVE_EMUI_VERSION_01805 | 1974-12-11 01:00:03.0 | 1974-12-11 02:00:03.0 | 123372038659 | -223372035049 | 12345680706.1234000000 | 22345680706.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1806 | > | 10806 | 67 | ACTIVE_EMUI_VERSION_01806 | 1974-12-12 01:00:03.0 | 1974-12-12 02:00:03.0 | 123372038660 | -223372035048 | 12345680707.1234000000 | 22345680707.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1807 | > | 10807 | 67 | ACTIVE_EMUI_VERSION_01807 | 1974-12-13 01:00:03.0 | 1974-12-13 02:00:03.0 | 123372038661 | -223372035047 | 12345680708.1234000000 | 22345680708.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1808 | > | 10808 | 67 | ACTIVE_EMUI_VERSION_01808 | 1974-12-14 01:00:03.0 | 1974-12-14 02:00:03.0 | 123372038662 | -223372035046 | 12345680709.1234000000 | 22345680709.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1809 | > | 10809 | 67 | ACTIVE_EMUI_VERSION_01809 | 1974-12-15 01:00:03.0 | 1974-12-15 02:00:03.0 | 123372038663 | -223372035045 | 12345680710.1234000000 | 22345680710.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1810 | > | 10810 | 67 | ACTIVE_EMUI_VERSION_01810 | 1974-12-16 01:00:03.0 | 1974-12-16 02:00:03.0 | 123372038664 | -223372035044 | 12345680711.1234000000 | 22345680711.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1811 | > | 10811 | 67 | ACTIVE_EMUI_VERSION_01811 | 1974-12-17 01:00:03.0 | 1974-12-17 02:00:03.0 | 123372038665 | -223372035043 | 12345680712.1234000000 | 22345680712.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1812 | > | 10812 | 67 | ACTIVE_EMUI_VERSION_01812 | 1974-12-18 01:00:03.0 | 1974-12-18 02:00:03.0 | 123372038666 | -223372035042 | 12345680713.1234000000 | 22345680713.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1813 | > | 10813 | 67 | ACTIVE_EMUI_VERSION_01813 | 1974-12-19 01:00:03.0 | 1974-12-19 02:00:03.0 | 123372038667 | -223372035041 | 12345680714.1234000000 | 22345680714.1234000000 | 1.12345674897976E10 | -1.12345674897976E10 | 1814 | > Here, its working in Spark 1.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)