Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CA46111AC1 for ; Thu, 24 Apr 2014 19:24:01 +0000 (UTC) Received: (qmail 86201 invoked by uid 500); 24 Apr 2014 19:24:00 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 86148 invoked by uid 500); 24 Apr 2014 19:24:00 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 86120 invoked by uid 99); 24 Apr 2014 19:24:00 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Apr 2014 19:24:00 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id CCEE81D70CF; Thu, 24 Apr 2014 19:23:54 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============2889672782741823355==" MIME-Version: 1.0 Subject: Re: Review Request 20096: HIVE-6835: Reading of partitioned Avro data fails if partition schema does not match table schema From: "Xuefu Zhang" To: "Anthony Hsu" , "Xuefu Zhang" , "hive" Date: Thu, 24 Apr 2014 19:23:54 -0000 Message-ID: <20140424192354.2045.27045@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Xuefu Zhang" X-ReviewGroup: hive X-ReviewRequest-URL: https://reviews.apache.org/r/20096/ X-Sender: "Xuefu Zhang" References: <20140424174300.2045.51871@reviews.apache.org> In-Reply-To: <20140424174300.2045.51871@reviews.apache.org> Reply-To: "Xuefu Zhang" X-ReviewRequest-Repository: hive-git --===============2889672782741823355== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20096/#review41375 ----------------------------------------------------------- Ship it! Ship It! - Xuefu Zhang On April 24, 2014, 5:42 p.m., Anthony Hsu wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/20096/ > ----------------------------------------------------------- > > (Updated April 24, 2014, 5:42 p.m.) > > > Review request for hive. > > > Repository: hive-git > > > Description > ------- > > The problem occurs when you store the "avro.schema.(literal|url)" in the SERDEPROPERTIES instead of the TBLPROPERTIES, add a partition, change the table's schema, and then try reading from the old partition. > > I fixed this problem by adding a new initialize() method to AbstractSerDe that takes both table properties and partition properties. The default implementation of this new method uses partition properties if its not null and table properties otherwise. I then overrode the new initalize() method in the AvroSerDe, and had the AvroSerDe always use the table properties. I also added a helper method that takes a Deserializer and calls the new initialize() method whenever the Deserializer is an instanceof AbstractSerDe. I then had to change all calls to Deserializer.initialize() to use my helper method instead. > > > Diffs > ----- > > contrib/src/java/org/apache/hadoop/hive/contrib/serde2/s3/S3LogDeserializer.java 69b618b > contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java 394ce3f > hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 089a31a > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InternalUtil.java fb650dd > hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestHCatRecordSerDe.java e84b789 > hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java c1d170a > hcatalog/core/src/test/java/org/apache/hive/hcatalog/rcfile/TestRCFileMapReduceInputFormat.java 9dde771 > hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java 7ba6bb8 > hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/StrictJsonWriter.java 9b26550 > jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java 3215178 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1bbe02e > ql/src/java/org/apache/hadoop/hive/ql/exec/DefaultFetchFormatter.java 25385ba > ql/src/java/org/apache/hadoop/hive/ql/exec/DemuxOperator.java b0b0925 > ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 6daf199 > ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableDummyOperator.java e00b7d3 > ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java c8003f5 > ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 80ccf5a > ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 055d13e > ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 2416948 > ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 1354b36 > ql/src/java/org/apache/hadoop/hive/ql/exec/SkewJoinHandler.java 3bf58f6 > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java c52a093 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecReducer.java 2ef79d4 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 0e4bdff > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java 49b8da1 > ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java f339651 > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 77305ff > ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java 3a258e4 > ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java 43cef5c > ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 6144303 > ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783 > ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestPTFRowContainer.java cea3529 > ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/Utilities.java 4fc613e > ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizedRowBatchCtx.java 7f3cb15 > ql/src/test/org/apache/hadoop/hive/ql/io/TestRCFile.java 5edd265 > ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 5664f3f > ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetSerDe.java be518b9 > ql/src/test/queries/clientpositive/avro_partitioned.q 6fe5117 > ql/src/test/results/clientpositive/avro_partitioned.q.out 644716d > serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 1ab15a8 > serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java d226d21 > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 55bfa2e > serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 9aa3c45 > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerde.java a5d494f > serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java e512f42 > serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java e8639ff > serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyArrayMapStruct.java 714045b > serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 28eb868 > serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 69c891d > serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestCrossMapEqualComparer.java a69fcb7 > serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestSimpleMapEqualComparer.java dd9610e > service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 2a113d5 > > Diff: https://reviews.apache.org/r/20096/diff/ > > > Testing > ------- > > Added test cases > > > Thanks, > > Anthony Hsu > > --===============2889672782741823355==--