Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Mon, 24 Oct 2016 18:57:58 +0000 (UTC)
From: "Ivan Bella (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.13014810.1477331291000.70298.1477335478370@Atlassian.JIRA>
In-Reply-To: <JIRA.13014810.1477331291000@Atlassian.JIRA>
References: <JIRA.13014810.1477331291000@Atlassian.JIRA> <JIRA.13014810.1477331291103@arcas>
Subject: [jira] [Comment Edited] (ACCUMULO-4502) Called next when there is
 no top
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 24 Oct 2016 18:58:00 -0000


    [ https://issues.apache.org/jira/browse/ACCUMULO-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602717#comment-15602717 ] 

Ivan Bella edited comment on ACCUMULO-4502 at 10/24/16 6:57 PM:
----------------------------------------------------------------

The condition in the HeapIterator is that the {{topIdx == null}} which implies that hasTop should have returned false.

In PartialMutationSkippingIterator.consume it is simply doing the following:

{code}
protected void consume() throws IOException
  while (getSource().hasTop() && ((MemKey) getSource().getTopKey()).kvCount > kvCount)
    getSource().next();
}
{code}

So obviously it is calling hasTop() before calling next().  The rub is that there is a SourceSwitchingIterator in between the PartialMutationSkippingIterator and the HeapIterator at the bottom.  Most likely a minor compaction caused the SourceSwitchingIterator to switch via the switchNow method.  When that happens the following code in {{_switchNow()}} is invoked:

{code}
if (switchSource()) {
  if (key != null) {
    iter.seek(...)
  }
}
{code}

So if {{getSource().getTop()}} was invoked in the PartialMutationSkippingIterator, then switchSource is called but BEFORE the nested {{ if (key != null) \{ iter.seek(...) \} }} is invoked, the {{getSource().next()}} is invoked by the PartialMutationSkippingIterator: this scenario would leave the topIdx to be null in the HeapIterator (most likely the RFile.Reader at this point) and subsequently cause this exception.

(elserj: some wiki formatting for readability)


was (Author: ivan.bella):
The condition in the HeapIterator is that the {{topIdx == null}} which implies that hasTop should have returned false.

In PartialMutationSkippingIterator.consume it is simply doing the following:

{code}
protected void consume() throws IOException
  while (getSource().hasTop() && ((MemKey) getSource().getTopKey()).kvCount > kvCount)
    getSource().next();
}
{code}

So obviously it is calling hasTop() before calling next().  The rub is that there is a SourceSwitchingIterator in between the PartialMutationSkippingIterator and the HeapIterator at the bottom.  Most likely a minor compaction caused the SourceSwitchingIterator to switch via the switchNow method.  When that happens the following code in {{_switchNow()}} is invoked:

{code}
if (switchSource()) {
  if (key != null) {
    iter.seek(...)
  }
}
{code}

So if {{getSource().getTop()}} was invoked in the PartialMutationSkippingIterator, then switchSource is called but BEFORE the nested {{ if (key != null) \{ iter.seek(...) } }} is invoked, the {{getSource().next()}} is invoked by the PartialMutationSkippingIterator: this scenario would leave the topIdx to be null in the HeapIterator (most likely the RFile.Reader at this point) and subsequently cause this exception.

(elserj: some wiki formatting for readability)

> Called next when there is no top
> --------------------------------
>
>                 Key: ACCUMULO-4502
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4502
>             Project: Accumulo
>          Issue Type: Bug
>          Components: core, tserver
>    Affects Versions: 1.6.6
>            Reporter: Ivan Bella
>
> This happens very rarely but we have seen the following exception (pulled from a server running 1.6.4).  Looking at the code I believe this condition can still happen in 1.8.0:
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Called next() when there is no top
> ...
> Caused by: java.lang.IllegalStateException: Called next() when there is no top
> HeapIterator.next(HeapIterator.java: 77)
> WrappingIterator.next(WrappingIterator.java: 96)
> MemKeyConversionIterator.next(InMemoryMap.java:162)
> SourceSwitchingIterator.readNext(SourceSwitchingIterator.java: 139)
> SourceSwitchingIterator.next(SourceSwitchingIterator.java: 123)
> PartialMutationSkippingIterator.consume(InMemoryMap.java:108)
> SkippingIterator.seek(SkippingIterator.java:43)


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)