accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Russ Weeks (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-3931) Repeated invocations of BatchScanner iterator().hasNext() causes MAC to become unresponsive
Date Wed, 29 Jul 2015 17:00:08 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646412#comment-14646412
] 

Russ Weeks commented on ACCUMULO-3931:
--------------------------------------

bq. What do you think?

I think I need to clear up a couple of my assumptions:
# a BatchScanner is expensive to create (relative to a small scan)
# a BatchScanner can't be re-used after it's been closed.

This is part of my work on ACCUMULO-638 to implement a [KeyColumnValueStore|http://thinkaurelius.github.io/titan/javadoc/current/com/thinkaurelius/titan/diskstorage/keycolumnvalue/KeyColumnValueStore.html]
for Accumulo. The approach I'm taking is that an AccumuloKCVS has an internal BatchWriter
and a BatchScanner, each of which are long lived. Titan calls the [getSlice|http://thinkaurelius.github.io/titan/javadoc/current/com/thinkaurelius/titan/diskstorage/keycolumnvalue/KeyColumnValueStore.html#getSlice-com.thinkaurelius.titan.diskstorage.keycolumnvalue.KeySliceQuery-com.thinkaurelius.titan.diskstorage.keycolumnvalue.StoreTransaction-]
method on the AccumuloKCVS, I turn it into one or more ranges and call setRanges on the BatchScanner.
I read some but possibly not all of the KV pairs that come back.

I'm worried that repeated calls to getSlice are going to exhaust the thread pool because I'm
not closing the scanner. The alternative is to set up and tear down the scanner on every call
to getSlice, but that seems like it could be a lot of overhead.

> Repeated invocations of BatchScanner iterator().hasNext() causes MAC to become unresponsive
> -------------------------------------------------------------------------------------------
>
>                 Key: ACCUMULO-3931
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3931
>             Project: Accumulo
>          Issue Type: Bug
>          Components: mini
>    Affects Versions: 1.6.3, 1.7.0
>            Reporter: Russ Weeks
>            Assignee: Eric Newton
>         Attachments: tablet_server_0.txt, tablet_server_1.txt, test_runner_stack.txt
>
>
> Steps to reproduce:
> # Instantiate a MiniAccumuloCluster with at least two tablet servers
> # Create a table; add splits to the table.
> # Add some mutations to the table distributed across the splits; flush the mutations.
> # Create a BatchScanner across the full range of the table.
> # Assert that the batch scanner has at least one KV pair by calling {{scanner.iterator().hasNext()}}
> # Repeat.
> It doesn't seem to matter if you close the scanner and create a new one in between calls
to hasNext, or if you re-seek the same scanner, or if the scanner is created in a static context
and re-used by multiple tests or created by each test. Eventually you will see that the {{TabletServerBatchReaderIterator}}
gets stuck polling its resultsQueue, preventing further tests from running.
> This happens roughyl 20% of the time in 1.6 when I run {{mvn clean test -Dtest=org.apache.accumulo.minicluster.MultipleHasNextTest
--projects minicluster}}, maybe less often in 1.7, and 100% of the time when I try to use
the MAC in my company's product build environment, which uses gradle.
> (I'll update with a link to a failing unit test as soon as I get an issue ID)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message