accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: Scanner / Batch Scanner Reuse
Date Fri, 25 Jan 2019 22:08:38 GMT
On Fri, Jan 25, 2019 at 4:22 PM Andrew Hulbert <andrew.hulbert@ccri.com> wrote:
>
> Keith,
>
> I was thinking about that a year or so ago...do you think there'd be any
> problems with modifying the batch scanners to share a thread pool per
> JVM? Or be able to pass in a thread pool. I think the reason was we were
> trying to limit the client scans threads globally in the JVM. But it
> hasn't been a huge issue.

I think passing in an ExecutorService makes sense.  It would be good
to do this before 2.0.0 for the new AccumuloClient API.   The
following method could be removed ....

https://github.com/apache/accumulo/blob/7e2e6bd2d1afff7fde0223463f84b0c70f9cab7f/core/src/main/java/org/apache/accumulo/core/client/AccumuloClient.java#L80

... and replaced with something like

createBatchScanner(String tableName, Authorizations authorizations,
BatchScannerConfig config);

On BatchScannerConfig and executor could be set.   Also we could allow
executors to be set for the batch writer and conditional writers.

>
> Andrew
>
> On 1/16/19 10:27 AM, Keith Turner wrote:
> > On Mon, Jan 14, 2019 at 2:05 PM gtotsline <greg.totsline@lmco.com> wrote:
> >> Hi -
> >>
> >> Can Scanners / Batch Scanners be reused?  Is there any downside to reusing a
> >> scanner (e.g. poorer performance)?  I assume creating a scanner takes time,
> >> so asking this question to see if I can avoid needlessly recreating a
> >> BatchScanner every time I do a query.
> > The main difference I can think of is that each batch scanner instance
> > creates a thread pool.  Other resources like tablet server connection
> > pools and tablet location caches are shared between batch scanner
> > instances.
> >
> >> Thanks!
> >>
> >>
> >>
> >> --
> >> Sent from: http://apache-accumulo.1065345.n5.nabble.com/Developers-f3.html

Mime
View raw message