accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ACCUMULO-665) large values, complex iterator stacks, and RFile readers can consume a surprising amount of memory
Date Fri, 29 Jun 2012 21:19:42 GMT
Eric Newton created ACCUMULO-665:
------------------------------------

             Summary: large values, complex iterator stacks, and RFile readers can consume
a surprising amount of memory
                 Key: ACCUMULO-665
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-665
             Project: Accumulo
          Issue Type: Bug
          Components: tserver
    Affects Versions: 1.5.0, 1.4.0
         Environment: large cluster
            Reporter: Eric Newton
            Assignee: Eric Newton
            Priority: Minor


On a production cluster, with a complex iterator tree, a large value (~350M) was causing a
4G tserver to fail with out-of-memory.

There were several factors contributing to the problem:
# a bug: the query should not have been looking to the big data
# complex iterator tree, causing many copies of the data to be held at the same time
# RFile doubles the buffer it uses to load values, and continues to use that large buffer
for future values

This ticket is for the last point.  If we know we're not even going to look at the value,
we can read past it without storing it in memory.  It is surprising that skipping past a large
value would cause the server to run out of memory, especially since it should fit into memory
enough times to be returned to the caller.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message