From issues-return-106293-archive-asf-public=cust-asf.ponee.io@hive.apache.org Thu Feb 22 18:16:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4728818067E for ; Thu, 22 Feb 2018 18:16:04 +0100 (CET) Received: (qmail 26273 invoked by uid 500); 22 Feb 2018 17:16:03 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 26264 invoked by uid 99); 22 Feb 2018 17:16:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Feb 2018 17:16:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id D5B9BC0145 for ; Thu, 22 Feb 2018 17:16:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.311 X-Spam-Level: X-Spam-Status: No, score=-110.311 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id AIzDcptG9ABc for ; Thu, 22 Feb 2018 17:16:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 563EF5F188 for ; Thu, 22 Feb 2018 17:16:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 8D44AE018A for ; Thu, 22 Feb 2018 17:16:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 474CF26852 for ; Thu, 22 Feb 2018 17:16:00 +0000 (UTC) Date: Thu, 22 Feb 2018 17:16:00 +0000 (UTC) From: "Adam Szita (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-15995: ------------------------------ Status: In Progress (was: Patch Available) > Syncing metastore table with serde schema > ----------------------------------------- > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore > Affects Versions: 2.1.0, 1.2.1, 3.0.0 > Reporter: Michal Ferlinski > Assignee: Adam Szita > Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, HIVE-15995.3.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can alter the 'avro.schema.url' property to update table schema to the next version. Updating properties however doesn't affect column list stored in metastore DB so the table is not in the newest version when returned from metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)