accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-3931) Repeated invocations of BatchScanner iterator().hasNext() causes MAC to become unresponsive
Date Wed, 29 Jul 2015 03:29:05 GMT


Josh Elser commented on ACCUMULO-3931:

bq. Is there some way to release those threads without consuming the whole iterator? eg. if
I call scanner.setRanges, will that do it? or do I have to call scanner.close()?

I think {{close()}} is the only exposed management of the underlying threadpool.

bq. The use case is, my code is retrieving the first n kv pairs for a given set of ranges
(yeah I get that the order of the returned keys is not guaranteed with a batch scanner) and
I'm hoping that I can reuse a single scanner, not close it and re-open it for every invocation.

So, ultimately, you'd like to create a BatchScanner once, but each time you'd {{setRanges()}}
and {{iterator()}}, you want it to use distinct resources (e.g. a new threadpool)? If we make
some sort of BatchScannerOptions (akin to BatchWriterOptions) and return a CloseableIterator
instead of a normal Iterator which should make these changes easier to manage. At the same
time, this would be easy to do outside the API with a factory method to make the BatchScanner
(assuming creating a new BatchScanner all the time isn't a problem in itself). What do you

> Repeated invocations of BatchScanner iterator().hasNext() causes MAC to become unresponsive
> -------------------------------------------------------------------------------------------
>                 Key: ACCUMULO-3931
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: mini
>    Affects Versions: 1.6.3, 1.7.0
>            Reporter: Russ Weeks
>            Assignee: Eric Newton
>         Attachments: tablet_server_0.txt, tablet_server_1.txt, test_runner_stack.txt
> Steps to reproduce:
> # Instantiate a MiniAccumuloCluster with at least two tablet servers
> # Create a table; add splits to the table.
> # Add some mutations to the table distributed across the splits; flush the mutations.
> # Create a BatchScanner across the full range of the table.
> # Assert that the batch scanner has at least one KV pair by calling {{scanner.iterator().hasNext()}}
> # Repeat.
> It doesn't seem to matter if you close the scanner and create a new one in between calls
to hasNext, or if you re-seek the same scanner, or if the scanner is created in a static context
and re-used by multiple tests or created by each test. Eventually you will see that the {{TabletServerBatchReaderIterator}}
gets stuck polling its resultsQueue, preventing further tests from running.
> This happens roughyl 20% of the time in 1.6 when I run {{mvn clean test -Dtest=org.apache.accumulo.minicluster.MultipleHasNextTest
--projects minicluster}}, maybe less often in 1.7, and 100% of the time when I try to use
the MAC in my company's product build environment, which uses gradle.
> (I'll update with a link to a failing unit test as soon as I get an issue ID)

This message was sent by Atlassian JIRA

View raw message