From issues-return-108106-archive-asf-public=cust-asf.ponee.io@hive.apache.org Wed Mar 7 19:25:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 42EAF180656 for ; Wed, 7 Mar 2018 19:25:04 +0100 (CET) Received: (qmail 68158 invoked by uid 500); 7 Mar 2018 18:25:03 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 68149 invoked by uid 99); 7 Mar 2018 18:25:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Mar 2018 18:25:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id C57C0C146D for ; Wed, 7 Mar 2018 18:25:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id o38Pk8etUrq9 for ; Wed, 7 Mar 2018 18:25:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id A7C545F3F0 for ; Wed, 7 Mar 2018 18:25:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id D7FA9E0311 for ; Wed, 7 Mar 2018 18:25:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 2558225401 for ; Wed, 7 Mar 2018 18:25:00 +0000 (UTC) Date: Wed, 7 Mar 2018 18:25:00 +0000 (UTC) From: "Vihang Karajgaonkar (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389925#comment-16389925 ] Vihang Karajgaonkar commented on HIVE-15995: -------------------------------------------- Hi [~szita] thanks for the updated patch. I was looking at the other queries which take in {{cascade}} option and I noticed that they have a partition spec in the query syntax. Eg: add/rename column is under alterTblPartitionStatementSuffix in HiveParser.g I think it has to do with the option of adding partition spec to the command. Do you think it makes sense to add a optional partition spec as well similar to add/replace columns query? I think column descriptors are also at partition level. Eg. you can do a {{describe table foo partition (p1=10);}} and hive will show the column information for that partition. So users can potentially update the columns at a partition level as well. The current patch is either just the table no partitions or table with all partitions. There is no way to update columns for a subset of partitions. Also, in the qtest can you add a test for a partitioned table? Include describe table .. partition .. syntax to confirm that partition columns are also getting updated. > Syncing metastore table with serde schema > ----------------------------------------- > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore > Affects Versions: 1.2.1, 2.1.0, 3.0.0 > Reporter: Michal Ferlinski > Assignee: Adam Szita > Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can alter the 'avro.schema.url' property to update table schema to the next version. Updating properties however doesn't affect column list stored in metastore DB so the table is not in the newest version when returned from metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)