accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-4483) NegativeArraySizeException from scan thread right after minor compaction
Date Tue, 04 Oct 2016 18:54:20 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-4483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546296#comment-15546296
] 

Josh Elser commented on ACCUMULO-4483:
--------------------------------------

bq. Not using sampling. Didn't know it existed until now.

A shiny, new feature in 1.8. It's actually pretty slick :)

bq. Further analysis reveals that the underlying iterator is returning the same K,V twice
and that MemValue.decode is modifying the source Value (which it should not). So, MemValue.decode()
succeeds the first time and fails the second.

bq. The underlying iterator in this case is the RFile.LocalityGroupReader. The seek is called
several times in sequence. The second time it manages to pass through and leave the RelativeKey
rf and the Value val with the same value. This is OK, 

Loving the teamwork here :). Curious that we didn't uncover this in any testing for 1.8.0.
This would be good to understand why (apparently?) our testing was insufficient to catch this.

bq. however the MemValue.decode method should not be calling Value.set on the original value.
It should be fairly easy to reproduce this in a test case.

The {{Value.set()}} doesn't appear to have been modified in Keith's commit though. So, is
it the interactions of {{LocalityGroupReader}} with what {{MemValue.decode(..)}} is doing
that is causing the problem?

Feel free to ping for a review when you get a fix put together.

> NegativeArraySizeException from scan thread right after minor compaction
> ------------------------------------------------------------------------
>
>                 Key: ACCUMULO-4483
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4483
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.8.0
>         Environment: Accumulo 1.8.0
> Java 1.8.0_72
>            Reporter: Dave Marion
>            Priority: Blocker
>             Fix For: 1.8.1
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> I have been getting NegativeArraySizeExceptions after upgrading to Accumulo 1.8.0. I
have tracked it down to 2 or more concurrent scans on a tablet that has just undergone minor
compaction. The minor compaction thread writes the in-memory map to a local temporary rfile
and tries to switch the current iterators to use it instead of the native map. The iterator
code in the scan thread may also switch itself to use the local temporary rfile it if notices
it before the minor compaction threads performs the switch. This all works up to this point.
Shortly after the switch one of the iterator threads will get a NegativeArraySizeException
from:
> SourceSwitchingIterator.seek() -> SourceSwitchingIterator.readNext() -> MemKeyConversionIterator.seek()
-> MemKeyConversionIterator.getTopKeyValue() ->MemValue.decode(). I added a bunch of
logging to find that the length of the byte array passed to MemValue.decode is zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message