orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ORC-409) Changes for extending MemoryManagerImpl
Date Fri, 28 Sep 2018 00:50:00 GMT
Prasanth Jayachandran created ORC-409:

             Summary: Changes for extending MemoryManagerImpl
                 Key: ORC-409
                 URL: https://issues.apache.org/jira/browse/ORC-409
             Project: ORC
          Issue Type: Bug
    Affects Versions: 1.5.4, 1.6.0
            Reporter: Prasanth Jayachandran
            Assignee: Prasanth Jayachandran

Orc memory manager uses MemoryMX bean to get max heap usage and assumes 50% (configurable)
is available for orc writers. This works well if container model where container size is close
to Xmx. In LLAP for example, a single container runs multiple executors and a single task
take up 50% of heap which is typically much more that the task executors own memory limit. 

Example: 128Gb container + 32 executors (assuming no cache) means there is 4GB available per
executor. But a single task that opens multiple orc writers in the default case will assume
64GB memory is available for orc when in fact it should only use 2GB from memory per executor.


WriterOption provides a mechanism to plug-in a custom memory manager for such scenarios which
is greate. But if the custom memory manager want to extend MemoryManagerImpl class, MemoryManagerImpl
should use getTotalMemoryPool() everywhere so that custom memory manager can just Override
getTotalMemoryPool() method and rest of the code from MemoryManagerImpl. 

This message was sent by Atlassian JIRA

View raw message