orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ORC-408) hard limit on memory use by ORC writers per tag
Date Wed, 26 Sep 2018 19:43:00 GMT
Sergey Shelukhin created ORC-408:

             Summary: hard limit on memory use by ORC writers per tag
                 Key: ORC-408
                 URL: https://issues.apache.org/jira/browse/ORC-408
             Project: ORC
          Issue Type: Bug
            Reporter: Sergey Shelukhin

Scenario: we want to hard-limit (within the constraints imposed by using Java) the memory
used by a particular Hive task dedicated to ORC writing, to protect other tasks from misbehaving
queries. This is similar to how we e.g. limit the memory used for hash join - when the hash
table goes over the limit, the task fails.

However, we currently cannot even hard-limit this for a single writer, much less for several
writers combined, when they are writing.
I wonder if it's possible to add two features to MemoryManager:
1) Grouping writers. A tag can be supplied externally (e.g. when creating the writer).
2) Hard-limiting the memory by tag - if the group exceeds the memory allowance, all the corresponding
writers should be made to fail on next operation, via the callback.

This message was sent by Atlassian JIRA

View raw message