hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Wilfong (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-4157) ORC runs out of heap when writing
Date Tue, 12 Mar 2013 23:11:13 GMT
Kevin Wilfong created HIVE-4157:
-----------------------------------

             Summary: ORC runs out of heap when writing
                 Key: HIVE-4157
                 URL: https://issues.apache.org/jira/browse/HIVE-4157
             Project: Hive
          Issue Type: Improvement
          Components: Serializers/Deserializers
    Affects Versions: 0.11.0
            Reporter: Kevin Wilfong
            Assignee: Kevin Wilfong


The OutStream class used by the ORC file format seems to aggressively allocate memory for
ByteBuffers and doesn't seem too eager to give it back.

This causes issues with heap space, particularly when a wide tables/dynamic partitions are
involved.

As a first step to resolving this problem, the OutStream class can be modified to lazily allocate
memory, and more actively make it available for garbage collection.

Follow ups could include checking the amount of free memory as part of determining if a spill
is needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message