accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Accumulo Seek performance
Date Thu, 25 Aug 2016 14:53:09 GMT
Sven,

Strange results. BatchScanners most definitely can be processed in 
parallel by the tabletservers.

There is a dynamically resizing threadpool in the TabletServers that 
respond to load on the system. As the pool remains full, it will grow. 
As it remains empty, it will shrink.

A few more questions: how many TabletServers do you have and did you run 
this benchmark multiple times in succession to see if the results 
changed? Also, have you tried increasing the number of threads per 
batchscanner to see if that makes a difference?

I might have to try to run a similar later today. I am curious :)

Sven Hodapp wrote:
> Hi,
>
> I've changed the code a little bit, so that it uses a thread pool (via the Future):
>
>      val ranges500 = ranges.asScala.grouped(500)  // this means 6 BatchScanners will
be created
>
>      for (ranges<- ranges500) {
>        val bscan = instance.createBatchScanner(ARTIFACTS, auths, 2)
>        bscan.setRanges(ranges.asJava)
>        Future {
>          time("mult-scanner") {
>            bscan.asScala.toList  // toList forces the iteration of the iterator
>          }
>        }
>      }
>
> Here are the results:
>
>      background log: info: mult-scanner time: 4807.289358 ms
>      background log: info: mult-scanner time: 4930.996522 ms
>      background log: info: mult-scanner time: 9510.010808 ms
>      background log: info: mult-scanner time: 11394.152391 ms
>      background log: info: mult-scanner time: 13297.247295 ms
>      background log: info: mult-scanner time: 14032.704837 ms
>
>      background log: info: single-scanner time: 15322.624393 ms
>
> Every Future completes independent, but in return every batch scanner iterator needs
more time to complete. :(
> This means the batch scanners aren't really processed in parallel on the server side?
> Should I reconfigure something? Maybe the tablet servers haven't/can't allocate enough
threads or memory? (Every of the two nodes has 8 cores and 64GB memory and a storage with
~300MB/s...)
>
> Regards,
> Sven
>

Mime
View raw message