hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-4248) Implement a memory manager for ORC
Date Fri, 05 Apr 2013 23:15:16 GMT

     [ https://issues.apache.org/jira/browse/HIVE-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Phabricator updated HIVE-4248:
------------------------------

    Attachment: HIVE-4248.D9993.1.patch

omalley requested code review of "HIVE-4248 [jira] Implement a memory manager for ORC".

Reviewers: JIRA

HIVE-4248 add memory manager for ORC

With the large default stripe size (256MB) and dynamic partitions, it is quite easy for users
to run out of memory when writing ORC files. We probably need a solution that keeps track
of the total number of concurrent ORC writers and divides the available heap space between
them.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D9993

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/MemoryManager.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestFileDump.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestMemoryManager.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/23895/

To: JIRA, omalley

                
> Implement a memory manager for ORC
> ----------------------------------
>
>                 Key: HIVE-4248
>                 URL: https://issues.apache.org/jira/browse/HIVE-4248
>             Project: Hive
>          Issue Type: New Feature
>          Components: Serializers/Deserializers
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: HIVE-4248.D9993.1.patch
>
>
> With the large default stripe size (256MB) and dynamic partitions, it is quite easy for
users to run out of memory when writing ORC files. We probably need a solution that keeps
track of the total number of concurrent ORC writers and divides the available heap space between
them. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message