hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <>
Subject [jira] [Commented] (HIVE-13232) Aggressively drop compression buffers in ORC OutStreams
Date Fri, 22 Apr 2016 23:02:12 GMT


Prasanth Jayachandran commented on HIVE-13232:

Backported to branch-1 and branch-2.0

> Aggressively drop compression buffers in ORC OutStreams
> -------------------------------------------------------
>                 Key: HIVE-13232
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: ORC
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 1.3.0, 2.1.0, 2.0.1
>         Attachments: HIVE-13232-branch-1.patch, HIVE-13232.patch, HIVE-13232.patch, HIVE-13232.patch
> In Hive 0.11, when ORC's OutStream's were flushed they dropped all of the their buffers.
In the patch for HIVE-4324, we inadvertently changed that behavior so that one of the buffers
is held on to. For queries with a lot of writers and thus under significant memory pressure
this can have a significant impact on the memory usage. 
> Note that "hive.optimize.sort.dynamic.partition" avoids this problem by sorting on the
dynamic partition key and thus only a single ORC writer is open at once. This will use memory
more effectively and avoid creating ORC files with very small stripes, which will produce
better downstream performance.

This message was sent by Atlassian JIRA

View raw message