Return-Path: X-Original-To: apmail-hive-issues-archive@minotaur.apache.org Delivered-To: apmail-hive-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C860D18B41 for ; Mon, 24 Aug 2015 10:29:46 +0000 (UTC) Received: (qmail 26295 invoked by uid 500); 24 Aug 2015 10:29:46 -0000 Delivered-To: apmail-hive-issues-archive@hive.apache.org Received: (qmail 26269 invoked by uid 500); 24 Aug 2015 10:29:46 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 26259 invoked by uid 99); 24 Aug 2015 10:29:46 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2015 10:29:46 +0000 Date: Mon, 24 Aug 2015 10:29:46 +0000 (UTC) From: "Cheng Lian (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-11625) Map instances with null keys are not properly handled for Parquet tables MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-11625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709076#comment-14709076 ] Cheng Lian commented on HIVE-11625: ----------------------------------- Sorry, according to the following statements in parquet-format spec {quote} The {{key}} field encodes the map's key type. This field must have repetition {{required}} and must always be present. {quote} Map keys written to Parquet must not be null. Then I think the problem here is that, whether should we silently ignore null keys when writing a map to a Parquet table like what Hive 0.14.0 does, or throw an exception (probably a more descriptive one instead of the one mentioned in the ticket description) like Hive 1.2.1. > Map instances with null keys are not properly handled for Parquet tables > ------------------------------------------------------------------------ > > Key: HIVE-11625 > URL: https://issues.apache.org/jira/browse/HIVE-11625 > Project: Hive > Issue Type: Sub-task > Affects Versions: 0.14.0, 0.13.1, 1.0.1, 1.1.1, 1.2.1 > Reporter: Cheng Lian > > Hive allows maps with null keys: > {code:sql} > hive> select map(null, 'foo', 1, 'bar', null, 'baz'); > {null:"baz",1:"bar"} > {code} > However, when written into Parquet tables, map entries with null as keys are either dropped or cause exceptions. Below is the result of Hive 0.14.0 and 0.13.1: > {code:sql} > hive> CREATE TABLE map_test STORED AS PARQUET > > AS SELECT MAP(null, 'foo', 1, 'bar', null, 'baz'); > ... > hive> SELECT * from map_test; > {1:"bar"} > {code} > And Hive 1.2.1 throws exception: > {noformat} > java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable (null) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable (null) > at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:516) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163) > ... 8 more > Caused by: java.lang.RuntimeException: Parquet record is malformed: empty fields are illegal, the field should be ommited completely instead > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:64) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31) > at parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121) > at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123) > at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42) > at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111) > at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124) > at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:753) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) > at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) > at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97) > at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162) > at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508) > ... 9 more > Caused by: parquet.io.ParquetEncodingException: empty fields are illegal, the field should be ommited completely instead > at parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:244) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeMap(DataWritableWriter.java:228) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeValue(DataWritableWriter.java:116) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeGroupFields(DataWritableWriter.java:89) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:60) > ... 23 more > java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable (null) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable (null) > at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:516) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163) > ... 8 more > Caused by: java.lang.RuntimeException: Parquet record is malformed: empty fields are illegal, the field should be ommited completely instead > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:64) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31) > at parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121) > at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123) > at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42) > at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111) > at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124) > at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:753) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) > at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) > at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97) > at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162) > at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508) > ... 9 more > Caused by: parquet.io.ParquetEncodingException: empty fields are illegal, the field should be ommited completely instead > at parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:244) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeMap(DataWritableWriter.java:228) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeValue(DataWritableWriter.java:116) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeGroupFields(DataWritableWriter.java:89) > at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:60) > ... 23 more > {noformat} > The problematic method is [{{DataWritableWriter.writeMap()}}|https://github.com/apache/hive/blob/release-1.2.1/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java#L223-L237]. Although the key value entry is not null, either key or value can be null. And null keys are not properly handled. > Changing [these 3 lines|https://github.com/apache/hive/blob/release-1.2.1/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java#L226-L228] to > {code:java} > if (keyElement != null) { > recordConsumer.startField(keyName, 0); > writeValue(keyElement, keyInspector, keyType); > recordConsumer.endField(keyName, 0); > } > {code} > should fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)