accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Medinets <david.medin...@gmail.com>
Subject Re: [jira] [Created] (ACCUMULO-665) large values, complex iterator stacks, and RFile readers can consume a surprising amount of memory
Date Sat, 30 Jun 2012 23:40:42 GMT
How would you define complex iterator stack? Can you outline the elements?
On Jun 29, 2012 5:19 PM, "Eric Newton (JIRA)" <jira@apache.org> wrote:

> Eric Newton created ACCUMULO-665:
> ------------------------------------
>
>              Summary: large values, complex iterator stacks, and RFile
> readers can consume a surprising amount of memory
>                  Key: ACCUMULO-665
>                  URL: https://issues.apache.org/jira/browse/ACCUMULO-665
>              Project: Accumulo
>           Issue Type: Bug
>           Components: tserver
>     Affects Versions: 1.5.0, 1.4.0
>          Environment: large cluster
>             Reporter: Eric Newton
>             Assignee: Eric Newton
>             Priority: Minor
>
>
> On a production cluster, with a complex iterator tree, a large value
> (~350M) was causing a 4G tserver to fail with out-of-memory.
>
> There were several factors contributing to the problem:
> # a bug: the query should not have been looking to the big data
> # complex iterator tree, causing many copies of the data to be held at the
> same time
> # RFile doubles the buffer it uses to load values, and continues to use
> that large buffer for future values
>
> This ticket is for the last point.  If we know we're not even going to
> look at the value, we can read past it without storing it in memory.  It is
> surprising that skipping past a large value would cause the server to run
> out of memory, especially since it should fit into memory enough times to
> be returned to the caller.
>
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message