hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth J (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6711) ORC maps uses getMapSize() from MapOI which is unreliable
Date Sun, 23 Mar 2014 22:19:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944609#comment-13944609
] 

Prasanth J commented on HIVE-6711:
----------------------------------

Test failure is not related.

> ORC maps uses getMapSize() from MapOI which is unreliable
> ---------------------------------------------------------
>
>                 Key: HIVE-6711
>                 URL: https://issues.apache.org/jira/browse/HIVE-6711
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.11.0, 0.12.0, 0.13.0
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>              Labels: orcfile
>             Fix For: 0.13.0, 0.14.0
>
>         Attachments: HIVE-6711.1.patch
>
>
> HIVE-6707 had issues with map size. getMapSize() of LazyMap and LazyBinaryMap does not
deserialize the keys and count the number of unique keys. Since getMapSize() may return non-distinct
count of keys, the length of maps stored using ORC's map tree writer will not be in sync with
actual map size. As a result of this RLE reader will try to read beyond the disk range expecting
more map entries and will throw exception.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message