hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Peña (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HIVE-9033) Fix ordering differences due to Java8 (part 2)
Date Mon, 08 Dec 2014 20:25:12 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238422#comment-14238422
] 

Sergio Peña commented on HIVE-9033:
-----------------------------------

There is one test that still fails on Java8:
ql/src/test/queries/clientpositive/parquet_map_null.q

Java8 expects the following output:
{noformat}
< {"key1":null,"key2":"val2"}
< {"key3":"val3","key4":null}
< {"key3":"val3","key4":null}
< {"key3":"val3","key4":null}
< {"key3":"val3","key4":null}
---
> {"key2":"val2","key1":null}
> {"key4":null,"key3":"val3"}
> {"key4":null,"key3":"val3"}
> {"key4":null,"key3":"val3"}
> {"key4":null,"key3":"val3"}
{noformat}

I found the problem is because the test is creating a parque table from an avro table; and
avro is getting the map data as HashMap objects. This HashMap is returned by the Avro API.

The line where this is happening is:
{noformat}
File: AvroGenericRecordReader.java

public boolean next(NullWritable nullWritable, AvroGenericRecordWritable record) throws IOException
{
   ...
   GenericData.Record r = (GenericData.Record)reader.next();
   ...
}
{noformat}

GenericData.Record is a list of HashMaps.

> Fix ordering differences due to Java8 (part 2)
> ----------------------------------------------
>
>                 Key: HIVE-9033
>                 URL: https://issues.apache.org/jira/browse/HIVE-9033
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergio Peña
>            Assignee: Sergio Peña
>         Attachments: HIVE-9033.2.patch
>
>
> Java 8 uses a different hash function for HashMap, which is leading to iteration order
differences in several cases. (See Java8 vs Java7)
> This is a second part of ordering fixes to help divide the work with other contributors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message