hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-12450) OrcFileMergeOperator does not use correct compression buffer size
Date Wed, 18 Nov 2015 22:15:11 GMT

     [ https://issues.apache.org/jira/browse/HIVE-12450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prasanth Jayachandran updated HIVE-12450:
-----------------------------------------
    Attachment: zlib-hang.png
                HIVE-12450.3.patch

Added test case that covers ZlibCodec infinite loop issue (refer zlib-hang.png for stacktrace).


> OrcFileMergeOperator does not use correct compression buffer size
> -----------------------------------------------------------------
>
>                 Key: HIVE-12450
>                 URL: https://issues.apache.org/jira/browse/HIVE-12450
>             Project: Hive
>          Issue Type: Bug
>          Components: ORC
>    Affects Versions: 1.2.0, 1.3.0, 1.2.1, 2.0.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Critical
>         Attachments: HIVE-12450.1.patch, HIVE-12450.2.patch, HIVE-12450.3.patch, zlib-hang.png
>
>
> OrcFileMergeOperator checks for compatibility before merging orc files. This compatibility
check include checking compression buffer size. But the output file that is created does not
honor the compression buffer size and always defaults to 256KB. This will not be a problem
when reading the orc file but can create unwanted memory pressure because of wasted space
within compression buffer.
> This issue also can make the merged file unreadable under certain cases. For example,
if the original compression buffer size is 8KB and if  hive.exec.orc.default.buffer.size is
set to 4KB. The merge file operator will use 4KB instead of actual 8KB which can result in
hanging of ORC reader (more specifically ZlibCodec will wait for more compression buffers).




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message