cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: Collecting Latency Metrics
Date Wed, 29 May 2019 17:02:25 GMT
Hello,

This metric is available indeed:

Most of the metrics available are documented here:
http://cassandra.apache.org/doc/latest/operating/metrics.html

For client requests (coordinator perspective latency):
http://cassandra.apache.org/doc/latest/operating/metrics.html#client-request-metrics
For local requests (per table/host latency, locally, no network
communication included):
http://cassandra.apache.org/doc/latest/operating/metrics.html#table-metrics

LatencySpecial type that tracks latency (in microseconds) with a Timer plus
> a Counter that tracks the total latency accrued since starting. The
> former is useful if you track the change in total latency since the last
> check. Each metric name of this type will have ‘Latency’ and ‘TotalLatency’
> appended to it.


You need 'Latency', not 'TotalLatency'. I would guess that's the issue
because latencies are available for as far as I remember (including C*2.0,
1.2 for sure :)).

Also, be aware that quite a few things changed in the metric structure
between C* 2.1 and C*2.2 (and C*3.0 is similar to C*2.2).

Examples of changes:
- ColumnFamily --> Table
- 99percentile --> p99
- 1MinuteRate -->  m1_rate
- metric name before KS and Table names and some other changes of this kind.
- ^ aggregations / aliases and indexes changed because of this ^ - breaking
most of the charts (in my case at least).
- ‘.value’ is not appended to the metric name anymore for gauges, nothing
instead.

For example (Grafana / Graphite):
From
```aliasByNode(averageSeriesWithWildcards(cassandra.$env.$dc.$host.org.apache.cassandra.metrics.ColumnFamily.$ks.$table.ReadLatency.95percentile,
2, 3), 1, 7, 8, 9)```
to
```aliasByNode(averageSeriesWithWildcards(cassandra.$env.$dc.$host.org.apache.cassandra.metrics.Table.ReadLatency.$ks.$table.p95,
2, 3), 1, 8, 9, 10)```


Another tip, is to use ccm locally (https://github.com/riptano/ccm) for
example and 'jconsole $cassandra_pid'. I use this -->jconsole $(ccm node1
show | grep pid | awk -F= '{print $2}')
Once you're in, you can explore available mbeans and find the metrics
available in 'org.apache.cassandra.[...]'. It's not ideal as you search
'manually' but it allowed me to find some metrics in the past or fix issues
from the doc above.

Out of curiosity, may I ask what backend you used for your monitoring?

C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



Le mer. 29 mai 2019 à 15:32, shalom sagges <shalomsagges@gmail.com> a
écrit :

> Hi All,
>
> I'm creating a dashboard that should collect read/write latency metrics on
> C* 3.x.
> In older versions (e.g. 2.0) I used to divide the total read latency in
> microseconds with the read count.
>
> Is there a metric attribute that shows read/write latency without the need
> to do the math, such as in nodetool tablestats "Local read latency" output?
> I saw there's a Mean attribute in org.apache.cassandra.metrics.ReadLatency
> but I'm not sure this is the right one.
>
> I'd really appreciate your help on this one.
> Thanks!
>
>
>

Mime
View raw message