accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dylan Hutchison <dhutc...@cs.washington.edu>
Subject Re: Profile a (batch) scan
Date Sat, 27 Aug 2016 00:21:28 GMT
If Zipkin isn't providing fine-grained enough information, one approach you
could take is to add an iterator to your scan that tracks performance.  Log
the start times, seek ranges, and count throughput.  Each tablet will do
this independently in separate threads.

You might base this approach on Accumulo's StopWatch
<https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/util/StopWatch.java>
utility class.  I adapted this class when I wanted to time particular code
segments in the Graphulo Watch
<https://github.com/Accla/graphulo/blob/tracing/src/main/java/edu/mit/ll/graphulo/skvi/Watch.java>
class.  You can see the Watch in action in the commented segments of this
old branch
<https://github.com/Accla/graphulo/blob/tracing/src/main/java/edu/mit/ll/graphulo/skvi/RemoteWriteIterator.java#L396>.
It captures the number of times a code segment is called, the average
runtime in that section, and the max and min runtime in that section.  It's
a kind of brute force approach, one that would have been better to
integrate with HTrace, but it worked well enough for my purpose.

In your iterator, you would count and measure the time statistics for the
next and seek methods.

On Fri, Aug 26, 2016 at 3:31 PM, Jonathan Lasko <jlasko@bbn.com> wrote:

> I haven't had time to dig into it yet but am hoping the Zipkin will help
> with some of these insights. (Unless that is the distributed trace you were
> referring to?)
>
> -Jonathan
>
>
> On 08/26/2016 04:54 PM, Mario Pastorelli wrote:
>
> I would like to understand the performance of a batch scan and I would
> like to have some hints on how to proceed. I have enabled the distributed
> trace, and it tells me that some batch scanner threads take much more time
> than others to complete but this is not helpful enough because it's not
> telling me why some threads take more. My gut feeling is that one batch
> thread is scanning more data than the others, which means that the data is
> not well distributed for a query, but I use a random shard byte as prefix
> of the keys which should guarantee that data of the same range is almost
> equally distributed among the tservers. I enabled JMX on the tservers and
> attached jvisualvm to get an idea of the state of each tserver but I
> couldn't find anything meaningful. I would like to know if there is a way
> to profile what's going on on a single tserver for a single scan thread and
> by this I mean:
>
>    1. where are the tablets required by a scan? Which tablet server?
>    2. how fast was the lookups on the index for that scan?
>    3. how many bytes/records were read for that scan without the iterators
>    4. how many seeks are done by the scan and possibly why
>
> The main Accumulo UI is fine to get an overview of Accumulo but don't
> really give you any information about the performance of a single query and
> it seems to me that they are heavily affected by what iterators do.
> Profiling a single scan is much more interesting. Is there a way to profile
> a single (batch) scan in Accumulo such that I have a complete overview of
> the entire process of reading and sending back records to the driver?
>
> Thanks,
> Mario
>
> --
> Mario Pastorelli | TERALYTICS
>
> *software engineer*
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: <mario.pastorelli@teralytics.ch>mario.pastorelli@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the
> sole attention and use of the intended recipient. Please notify us at once
> if you think that it may not be intended for you and delete it immediately.
>
>
>

Mime
View raw message