accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-3067) scan performance degrades after compaction
Date Thu, 21 Aug 2014 18:41:11 GMT


Josh Elser commented on ACCUMULO-3067:

Did a little bit of digging (by no means do I understand this fully :D) and found,
specifically this quote from the post:

Zombie methods are methods whose code has been made invalid by class loading. Generally the
server compiler makes aggressive inlining decisions of non-final methods. As long as the inlined
method is never overridden the code is correct. When a subclass is loaded and the method overridden,
the compiled code is broken for all future calls to it. The code gets declared "not entrant"
(no future callers to the broken code), but sometimes existing callers can keep using the
code. In the case of inlining, that's not good enough; existing callers' stack frames are
"deoptimized" when they return to the code from nested calls (or just if they are running
in the code). When no more stack frames hold PC's into the broken code it's declared a "zombie"
- ready for removal once the GC gets around to it.

This makes it seem like we add in some extra iterators near the "top" of the stack which override
some earlier classes which invalidates the ones we had previously compiled. Not sure if I'm
entirely interpreting this correctly, but it seems sort of relevant to what you're seeing.

> scan performance degrades after compaction
> ------------------------------------------
>                 Key: ACCUMULO-3067
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>         Environment: Macbook Pro 2.6 GHz Intel Core i7, 16GB RAM, SSD, OSX 10.9.4, single
tablet server process, single client process
>            Reporter: Adam Fuchs
>         Attachments: Screen Shot 2014-08-19 at 4.19.37 PM.png, accumulo_query_perf_test.tar.gz,
> I've been running some scan performance tests on 1.6.0, and I'm running into an interesting
situation in which query performance starts at a certain level and then degrades by ~15% after
an event. The test follows roughly the following scenario:
>  # Single tabletserver instance
>  # Load 100M small (~10byte) key/values into a tablet and let it finish major compacting
>  # Disable the garbage collector (this makes the time to _the event_ longer)
>  # Restart the tabletserver
>  # Repeatedly scan from the beginning to the end of the table in a loop
>  # Something happens on the tablet server, like one of {idle compaction of metadata table,
forced flush of metadata table, forced compaction of metadata table, forced flush of trace
>  # Observe that scan rates dropped by 15-20%
>  # Observe that restarting the scan will not improve performance back to original level.
Performance only gets better upon restarting the tablet server.
> I've been able to get this not to happen by removing iterators from the iterator tree.
It doesn't seem to matter which iterators, but removing a certain number both improves performance
(significantly) and eliminates the degradation problem. The default iterator tree includes:

>  * SourceSwitchingIterator
>  ** VersioningIterator
>  *** SynchronizedIterator
>  **** VisibilityFilter
>  ***** ColumnQualifierFilter
>  ****** ColumnFamilySkippingIterator
>  ******* DeletingIterator
>  ******** StatsIterator
>  ********* MultiIterator
>  ********** MemoryIterator
>  ********** ProblemReportingIterator
>  *********** HeapIterator
>  ************ RFile.LocalityGroupReader
> We can eliminate the weird condition by narrowing the set of iterators to:
>  * SourceSwitchingIterator
>  ** VisibilityFilter
>  *** ColumnFamilySkippingIterator
>  **** DeletingIterator
>  ***** StatsIterator
>  ****** MultiIterator
>  ******* MemoryIterator
>  ******* ProblemReportingIterator
>  ******** HeapIterator
>  ********* RFile.LocalityGroupReader
> There are other combinations that also perform much better than the default. I haven't
been able to isolate this problem to a single iterator, despite removing each iterator one
at a time.
> Anybody know what might be happening here? Best theory so far: the JVM learns that iterators
can be used in a different way after a compaction, and some JVM optimization like JIT compilation,
branch prediction, or automatic inlining stops happening.

This message was sent by Atlassian JIRA

View raw message