Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A0BF2105B7 for ; Thu, 28 Nov 2013 05:03:39 +0000 (UTC) Received: (qmail 88510 invoked by uid 500); 28 Nov 2013 05:03:38 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 88373 invoked by uid 500); 28 Nov 2013 05:03:36 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 88362 invoked by uid 500); 28 Nov 2013 05:03:36 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 88355 invoked by uid 99); 28 Nov 2013 05:03:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Nov 2013 05:03:35 +0000 Date: Thu, 28 Nov 2013 05:03:35 +0000 (UTC) From: "Xuefu Zhang (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-5907) after adding new column to hive partition table,the new columns'value will be null be for reload all partitions MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-5907: ------------------------------ Description: when add new columns on external partition table ,query new column will get NULL value {code} 0: jdbc:hive2://localhost:10000> create external table test_alex1_replacereloadpart (id string,myname string) partitioned by(dt string); No rows affected (0.395 seconds) 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart add partition (dt='2013-11-28') location '2013-11-28'; No rows affected (0.339 seconds) 0: jdbc:hive2://localhost:10000> insert overwrite table test_alex1_replacereloadpart partition(dt='2013-11-28') select id,myname from test_alex1; +-----+---------+ | id | myname | +-----+---------+ +-----+---------+ No rows selected (24.073 seconds) 0: jdbc:hive2://localhost:10000> select * from test_alex1_replacereloadpart; +-------------------------------------------------------------------+---------+-------------+ | id | myname | dt | +-------------------------------------------------------------------+---------+-------------+ | 20501B55B0C64273A5316C5078442569E425CC318ADC454CA163D72503489436 | 1 | 2013-11-28 | | 20501B55B0C64273A5316C5078442569E3D87D24CA4E42D9A200E391417A7994 | 1 | 2013-11-28 | | 20501B55B0C64273A5316C5078442569814086424B2942919C97DF1D2D2C1F55 | 1 | 2013-11-28 | | 205018A818FB43A38CC29FF701E47D8B205018A818FB43A38CC29FF701E47D8B | 1 | 2013-11-28 | | 205016D04EDA435CB578D059F931221EB9F06E42815243A486B611B96366DD01 | 1 | 2013-11-28 | | 205016D04EDA435CB578D059F931221E205016D04EDA435CB578D059F931221E | 1 | 2013-11-28 | | 20500D24895641DBBBDBE71640E57887EA06A0AEB4324117B8125FC815297B36 | 1 | 2013-11-28 | | 205006E8A3EB4CA1997D947D89C5FD1B205006E8A3EB4CA1997D947D89C5FD1B | 1 | 2013-11-28 | | 2050054E19B44C4D992D97C1661A26C32050054E19B44C4D992D97C1661A26C3 | 1 | 2013-11-28 | | 204fa32a43ef4aefac3b391562c5a25b7149D57B47E74C6F8C22CC30292FADF1 | 1 | 2013-11-28 | +-------------------------------------------------------------------+---------+-------------+ 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart replace columns (id string,myname string,bee string); No rows affected (0.202 seconds) {code} i delete old data from hive partition dir ,and add new data into the same dir {code} 0: jdbc:hive2://localhost:10000> select * from test_alex1_replacereloadpart; +-------------------------------------------------------------------+---------+------+-------------+ | id | myname | bee | dt | +-------------------------------------------------------------------+---------+------+-------------+ | 20501B55B0C64273A5316C5078442569E425CC318ADC454CA163D72503489436 | 1 | | 2013-11-28 | | 20501B55B0C64273A5316C5078442569E3D87D24CA4E42D9A200E391417A7994 | 1 | | 2013-11-28 | | 20501B55B0C64273A5316C5078442569814086424B2942919C97DF1D2D2C1F55 | 1 | | 2013-11-28 | | 205018A818FB43A38CC29FF701E47D8B205018A818FB43A38CC29FF701E47D8B | 1 | | 2013-11-28 | | 205016D04EDA435CB578D059F931221EB9F06E42815243A486B611B96366DD01 | 1 | | 2013-11-28 | | 205016D04EDA435CB578D059F931221E205016D04EDA435CB578D059F931221E | 1 | | 2013-11-28 | | 20500D24895641DBBBDBE71640E57887EA06A0AEB4324117B8125FC815297B36 | 1 | | 2013-11-28 | | 205006E8A3EB4CA1997D947D89C5FD1B205006E8A3EB4CA1997D947D89C5FD1B | 1 | | 2013-11-28 | | 2050054E19B44C4D992D97C1661A26C32050054E19B44C4D992D97C1661A26C3 | 1 | | 2013-11-28 | | 204fa32a43ef4aefac3b391562c5a25b7149D57B47E74C6F8C22CC30292FADF1 | 1 | | 2013-11-28 | +-------------------------------------------------------------------+---------+------+-------------+ {code} after reload the partition ,the new data appear {code} 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart drop partition (dt='2013-11-28'); No rows affected (0.286 seconds) 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart add partition (dt='2013-11-28') location '2013-11-28'; No rows affected (0.551 seconds) 0: jdbc:hive2://localhost:10000> select * from test_alex1_replacereloadpart; +-------------------------------------------------------------------+---------+------+-------------+ | id | myname | bee | dt | +-------------------------------------------------------------------+---------+------+-------------+ | 20501B55B0C64273A5316C5078442569E425CC318ADC454CA163D72503489436 | 1 | 10 | 2013-11-28 | | 20501B55B0C64273A5316C5078442569E3D87D24CA4E42D9A200E391417A7994 | 1 | 9 | 2013-11-28 | | 20501B55B0C64273A5316C5078442569814086424B2942919C97DF1D2D2C1F55 | 1 | 8 | 2013-11-28 | | 205018A818FB43A38CC29FF701E47D8B205018A818FB43A38CC29FF701E47D8B | 1 | 7 | 2013-11-28 | | 205016D04EDA435CB578D059F931221EB9F06E42815243A486B611B96366DD01 | 1 | 6 | 2013-11-28 | | 205016D04EDA435CB578D059F931221E205016D04EDA435CB578D059F931221E | 1 | 5 | 2013-11-28 | | 20500D24895641DBBBDBE71640E57887EA06A0AEB4324117B8125FC815297B36 | 1 | 4 | 2013-11-28 | | 205006E8A3EB4CA1997D947D89C5FD1B205006E8A3EB4CA1997D947D89C5FD1B | 1 | 3 | 2013-11-28 | | 2050054E19B44C4D992D97C1661A26C32050054E19B44C4D992D97C1661A26C3 | 1 | 2 | 2013-11-28 | | 204fa32a43ef4aefac3b391562c5a25b7149D57B47E74C6F8C22CC30292FADF1 | 1 | 1 | 2013-11-28 | +-------------------------------------------------------------------+---------+------+-------------+ {code} was: when add new columns on external partition table ,query new column will get NULL value {code} 0: jdbc:hive2://localhost:10000> create external table test_alex1_replacereloadpart (id string,myname string) partitioned by(dt string); No rows affected (0.395 seconds) 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart add partition (dt='2013-11-28') location '2013-11-28'; No rows affected (0.339 seconds) 0: jdbc:hive2://localhost:10000> insert overwrite table test_alex1_replacereloadpart partition(dt='2013-11-28') select id,myname from test_alex1; +-----+---------+ | id | myname | +-----+---------+ +-----+---------+ No rows selected (24.073 seconds) 0: jdbc:hive2://localhost:10000> select * from test_alex1_replacereloadpart; +-------------------------------------------------------------------+---------+-------------+ | id | myname | dt | +-------------------------------------------------------------------+---------+-------------+ | 20501B55B0C64273A5316C5078442569E425CC318ADC454CA163D72503489436 | 1 | 2013-11-28 | | 20501B55B0C64273A5316C5078442569E3D87D24CA4E42D9A200E391417A7994 | 1 | 2013-11-28 | | 20501B55B0C64273A5316C5078442569814086424B2942919C97DF1D2D2C1F55 | 1 | 2013-11-28 | | 205018A818FB43A38CC29FF701E47D8B205018A818FB43A38CC29FF701E47D8B | 1 | 2013-11-28 | | 205016D04EDA435CB578D059F931221EB9F06E42815243A486B611B96366DD01 | 1 | 2013-11-28 | | 205016D04EDA435CB578D059F931221E205016D04EDA435CB578D059F931221E | 1 | 2013-11-28 | | 20500D24895641DBBBDBE71640E57887EA06A0AEB4324117B8125FC815297B36 | 1 | 2013-11-28 | | 205006E8A3EB4CA1997D947D89C5FD1B205006E8A3EB4CA1997D947D89C5FD1B | 1 | 2013-11-28 | | 2050054E19B44C4D992D97C1661A26C32050054E19B44C4D992D97C1661A26C3 | 1 | 2013-11-28 | | 204fa32a43ef4aefac3b391562c5a25b7149D57B47E74C6F8C22CC30292FADF1 | 1 | 2013-11-28 | +-------------------------------------------------------------------+---------+-------------+ 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart replace columns (id string,myname string,bee string); No rows affected (0.202 seconds) i delete old data from hive partition dir ,and add new data into the same dir 0: jdbc:hive2://localhost:10000> select * from test_alex1_replacereloadpart; +-------------------------------------------------------------------+---------+------+-------------+ | id | myname | bee | dt | +-------------------------------------------------------------------+---------+------+-------------+ | 20501B55B0C64273A5316C5078442569E425CC318ADC454CA163D72503489436 | 1 | | 2013-11-28 | | 20501B55B0C64273A5316C5078442569E3D87D24CA4E42D9A200E391417A7994 | 1 | | 2013-11-28 | | 20501B55B0C64273A5316C5078442569814086424B2942919C97DF1D2D2C1F55 | 1 | | 2013-11-28 | | 205018A818FB43A38CC29FF701E47D8B205018A818FB43A38CC29FF701E47D8B | 1 | | 2013-11-28 | | 205016D04EDA435CB578D059F931221EB9F06E42815243A486B611B96366DD01 | 1 | | 2013-11-28 | | 205016D04EDA435CB578D059F931221E205016D04EDA435CB578D059F931221E | 1 | | 2013-11-28 | | 20500D24895641DBBBDBE71640E57887EA06A0AEB4324117B8125FC815297B36 | 1 | | 2013-11-28 | | 205006E8A3EB4CA1997D947D89C5FD1B205006E8A3EB4CA1997D947D89C5FD1B | 1 | | 2013-11-28 | | 2050054E19B44C4D992D97C1661A26C32050054E19B44C4D992D97C1661A26C3 | 1 | | 2013-11-28 | | 204fa32a43ef4aefac3b391562c5a25b7149D57B47E74C6F8C22CC30292FADF1 | 1 | | 2013-11-28 | +-------------------------------------------------------------------+---------+------+-------------+ after reload the partition ,the new data appear 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart drop partition (dt='2013-11-28'); No rows affected (0.286 seconds) 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart add partition (dt='2013-11-28') location '2013-11-28'; No rows affected (0.551 seconds) 0: jdbc:hive2://localhost:10000> select * from test_alex1_replacereloadpart; +-------------------------------------------------------------------+---------+------+-------------+ | id | myname | bee | dt | +-------------------------------------------------------------------+---------+------+-------------+ | 20501B55B0C64273A5316C5078442569E425CC318ADC454CA163D72503489436 | 1 | 10 | 2013-11-28 | | 20501B55B0C64273A5316C5078442569E3D87D24CA4E42D9A200E391417A7994 | 1 | 9 | 2013-11-28 | | 20501B55B0C64273A5316C5078442569814086424B2942919C97DF1D2D2C1F55 | 1 | 8 | 2013-11-28 | | 205018A818FB43A38CC29FF701E47D8B205018A818FB43A38CC29FF701E47D8B | 1 | 7 | 2013-11-28 | | 205016D04EDA435CB578D059F931221EB9F06E42815243A486B611B96366DD01 | 1 | 6 | 2013-11-28 | | 205016D04EDA435CB578D059F931221E205016D04EDA435CB578D059F931221E | 1 | 5 | 2013-11-28 | | 20500D24895641DBBBDBE71640E57887EA06A0AEB4324117B8125FC815297B36 | 1 | 4 | 2013-11-28 | | 205006E8A3EB4CA1997D947D89C5FD1B205006E8A3EB4CA1997D947D89C5FD1B | 1 | 3 | 2013-11-28 | | 2050054E19B44C4D992D97C1661A26C32050054E19B44C4D992D97C1661A26C3 | 1 | 2 | 2013-11-28 | | 204fa32a43ef4aefac3b391562c5a25b7149D57B47E74C6F8C22CC30292FADF1 | 1 | 1 | 2013-11-28 | +-------------------------------------------------------------------+---------+------+-------------+ > after adding new column to hive partition table,the new columns'value will be null be for reload all partitions > --------------------------------------------------------------------------------------------------------------- > > Key: HIVE-5907 > URL: https://issues.apache.org/jira/browse/HIVE-5907 > Project: Hive > Issue Type: Bug > Reporter: alex kim > Priority: Critical > > when add new columns on external partition table ,query new column will get NULL value > {code} > 0: jdbc:hive2://localhost:10000> create external table test_alex1_replacereloadpart (id string,myname string) partitioned by(dt string); > No rows affected (0.395 seconds) > 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart add partition (dt='2013-11-28') location '2013-11-28'; > No rows affected (0.339 seconds) > 0: jdbc:hive2://localhost:10000> insert overwrite table test_alex1_replacereloadpart partition(dt='2013-11-28') select id,myname from test_alex1; > +-----+---------+ > | id | myname | > +-----+---------+ > +-----+---------+ > No rows selected (24.073 seconds) > 0: jdbc:hive2://localhost:10000> select * from test_alex1_replacereloadpart; > +-------------------------------------------------------------------+---------+-------------+ > | id | myname | dt | > +-------------------------------------------------------------------+---------+-------------+ > | 20501B55B0C64273A5316C5078442569E425CC318ADC454CA163D72503489436 | 1 | 2013-11-28 | > | 20501B55B0C64273A5316C5078442569E3D87D24CA4E42D9A200E391417A7994 | 1 | 2013-11-28 | > | 20501B55B0C64273A5316C5078442569814086424B2942919C97DF1D2D2C1F55 | 1 | 2013-11-28 | > | 205018A818FB43A38CC29FF701E47D8B205018A818FB43A38CC29FF701E47D8B | 1 | 2013-11-28 | > | 205016D04EDA435CB578D059F931221EB9F06E42815243A486B611B96366DD01 | 1 | 2013-11-28 | > | 205016D04EDA435CB578D059F931221E205016D04EDA435CB578D059F931221E | 1 | 2013-11-28 | > | 20500D24895641DBBBDBE71640E57887EA06A0AEB4324117B8125FC815297B36 | 1 | 2013-11-28 | > | 205006E8A3EB4CA1997D947D89C5FD1B205006E8A3EB4CA1997D947D89C5FD1B | 1 | 2013-11-28 | > | 2050054E19B44C4D992D97C1661A26C32050054E19B44C4D992D97C1661A26C3 | 1 | 2013-11-28 | > | 204fa32a43ef4aefac3b391562c5a25b7149D57B47E74C6F8C22CC30292FADF1 | 1 | 2013-11-28 | > +-------------------------------------------------------------------+---------+-------------+ > 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart replace columns (id string,myname string,bee string); > No rows affected (0.202 seconds) > {code} > i delete old data from hive partition dir ,and add new data into the same dir > {code} > 0: jdbc:hive2://localhost:10000> select * from test_alex1_replacereloadpart; > +-------------------------------------------------------------------+---------+------+-------------+ > | id | myname | bee | dt | > +-------------------------------------------------------------------+---------+------+-------------+ > | 20501B55B0C64273A5316C5078442569E425CC318ADC454CA163D72503489436 | 1 | | 2013-11-28 | > | 20501B55B0C64273A5316C5078442569E3D87D24CA4E42D9A200E391417A7994 | 1 | | 2013-11-28 | > | 20501B55B0C64273A5316C5078442569814086424B2942919C97DF1D2D2C1F55 | 1 | | 2013-11-28 | > | 205018A818FB43A38CC29FF701E47D8B205018A818FB43A38CC29FF701E47D8B | 1 | | 2013-11-28 | > | 205016D04EDA435CB578D059F931221EB9F06E42815243A486B611B96366DD01 | 1 | | 2013-11-28 | > | 205016D04EDA435CB578D059F931221E205016D04EDA435CB578D059F931221E | 1 | | 2013-11-28 | > | 20500D24895641DBBBDBE71640E57887EA06A0AEB4324117B8125FC815297B36 | 1 | | 2013-11-28 | > | 205006E8A3EB4CA1997D947D89C5FD1B205006E8A3EB4CA1997D947D89C5FD1B | 1 | | 2013-11-28 | > | 2050054E19B44C4D992D97C1661A26C32050054E19B44C4D992D97C1661A26C3 | 1 | | 2013-11-28 | > | 204fa32a43ef4aefac3b391562c5a25b7149D57B47E74C6F8C22CC30292FADF1 | 1 | | 2013-11-28 | > +-------------------------------------------------------------------+---------+------+-------------+ > {code} > after reload the partition ,the new data appear > {code} > 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart drop partition (dt='2013-11-28'); > No rows affected (0.286 seconds) > 0: jdbc:hive2://localhost:10000> alter table test_alex1_replacereloadpart add partition (dt='2013-11-28') location '2013-11-28'; > No rows affected (0.551 seconds) > 0: jdbc:hive2://localhost:10000> select * from test_alex1_replacereloadpart; > +-------------------------------------------------------------------+---------+------+-------------+ > | id | myname | bee | dt | > +-------------------------------------------------------------------+---------+------+-------------+ > | 20501B55B0C64273A5316C5078442569E425CC318ADC454CA163D72503489436 | 1 | 10 | 2013-11-28 | > | 20501B55B0C64273A5316C5078442569E3D87D24CA4E42D9A200E391417A7994 | 1 | 9 | 2013-11-28 | > | 20501B55B0C64273A5316C5078442569814086424B2942919C97DF1D2D2C1F55 | 1 | 8 | 2013-11-28 | > | 205018A818FB43A38CC29FF701E47D8B205018A818FB43A38CC29FF701E47D8B | 1 | 7 | 2013-11-28 | > | 205016D04EDA435CB578D059F931221EB9F06E42815243A486B611B96366DD01 | 1 | 6 | 2013-11-28 | > | 205016D04EDA435CB578D059F931221E205016D04EDA435CB578D059F931221E | 1 | 5 | 2013-11-28 | > | 20500D24895641DBBBDBE71640E57887EA06A0AEB4324117B8125FC815297B36 | 1 | 4 | 2013-11-28 | > | 205006E8A3EB4CA1997D947D89C5FD1B205006E8A3EB4CA1997D947D89C5FD1B | 1 | 3 | 2013-11-28 | > | 2050054E19B44C4D992D97C1661A26C32050054E19B44C4D992D97C1661A26C3 | 1 | 2 | 2013-11-28 | > | 204fa32a43ef4aefac3b391562c5a25b7149D57B47E74C6F8C22CC30292FADF1 | 1 | 1 | 2013-11-28 | > +-------------------------------------------------------------------+---------+------+-------------+ > {code} -- This message was sent by Atlassian JIRA (v6.1#6144)