Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
Received-SPF: softfail (athena.apache.org: transitioning domain of
 wilhelm.von.cloud@accumulo.net does not designate 209.85.214.169 as permitted
 sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOiJXP5LmumyvFeUVA1QwdPsXnyRd6RKDCQgfeFnON+tAh_tsw@mail.gmail.com>
References: <1609498285.73408.1341004784298.JavaMail.jiratomcat@issues-vm>
	<CAOiJXP5LmumyvFeUVA1QwdPsXnyRd6RKDCQgfeFnON+tAh_tsw@mail.gmail.com>
Date: Sat, 30 Jun 2012 20:00:37 -0400
Message-ID: 
 <CAMz+Duv0TAxjGc_w_s7qsbmJJ-ow1pKUmVojiLgc0_=3PzDhcQ@mail.gmail.com>
Subject: Re: [jira] [Created] (ACCUMULO-665) large values, complex iterator
 stacks, and RFile readers can consume a surprising amount of memory
From: William Slacum <wilhelm.von.cloud@accumulo.net>
To: dev@accumulo.apache.org
Content-Type: multipart/alternative; boundary=e89a8f839e8f0c059f04c3b95fa0

--e89a8f839e8f0c059f04c3b95fa0
Content-Type: text/plain; charset=ISO-8859-1

He's referring to something like the BooleanLogic iterator stack in the
Wikipedia example. It's a tree of user iterators that are merging streams
of key-value pairs together, so you end up getting many open readers and
possibly many RFile blocks spread out among many HDFS blocks concurrently.

On Sat, Jun 30, 2012 at 7:40 PM, David Medinets <david.medinets@gmail.com>wrote:

> How would you define complex iterator stack? Can you outline the elements?
> On Jun 29, 2012 5:19 PM, "Eric Newton (JIRA)" <jira@apache.org> wrote:
>
> > Eric Newton created ACCUMULO-665:
> > ------------------------------------
> >
> >              Summary: large values, complex iterator stacks, and RFile
> > readers can consume a surprising amount of memory
> >                  Key: ACCUMULO-665
> >                  URL: https://issues.apache.org/jira/browse/ACCUMULO-665
> >              Project: Accumulo
> >           Issue Type: Bug
> >           Components: tserver
> >     Affects Versions: 1.5.0, 1.4.0
> >          Environment: large cluster
> >             Reporter: Eric Newton
> >             Assignee: Eric Newton
> >             Priority: Minor
> >
> >
> > On a production cluster, with a complex iterator tree, a large value
> > (~350M) was causing a 4G tserver to fail with out-of-memory.
> >
> > There were several factors contributing to the problem:
> > # a bug: the query should not have been looking to the big data
> > # complex iterator tree, causing many copies of the data to be held at
> the
> > same time
> > # RFile doubles the buffer it uses to load values, and continues to use
> > that large buffer for future values
> >
> > This ticket is for the last point.  If we know we're not even going to
> > look at the value, we can read past it without storing it in memory.  It
> is
> > surprising that skipping past a large value would cause the server to run
> > out of memory, especially since it should fit into memory enough times to
> > be returned to the caller.
> >
> >
> > --
> > This message is automatically generated by JIRA.
> > If you think it was sent incorrectly, please contact your JIRA
> > administrators:
> > https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
> >
> >
>

--e89a8f839e8f0c059f04c3b95fa0--