orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ORC-408) hard limit on memory use by ORC writers per tag
Date Wed, 26 Sep 2018 19:43:00 GMT
Sergey Shelukhin created ORC-408:
------------------------------------

             Summary: hard limit on memory use by ORC writers per tag
                 Key: ORC-408
                 URL: https://issues.apache.org/jira/browse/ORC-408
             Project: ORC
          Issue Type: Bug
            Reporter: Sergey Shelukhin


Scenario: we want to hard-limit (within the constraints imposed by using Java) the memory
used by a particular Hive task dedicated to ORC writing, to protect other tasks from misbehaving
queries. This is similar to how we e.g. limit the memory used for hash join - when the hash
table goes over the limit, the task fails.

However, we currently cannot even hard-limit this for a single writer, much less for several
writers combined, when they are writing.
I wonder if it's possible to add two features to MemoryManager:
1) Grouping writers. A tag can be supplied externally (e.g. when creating the writer).
2) Hard-limiting the memory by tag - if the group exceeds the memory allowance, all the corresponding
writers should be made to fail on next operation, via the callback.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message