cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: Metrics matrix: migrate 2.1.x metrics to 2.2.x+
Date Tue, 02 Oct 2018 10:20:52 GMT
Hello Carl,

I guess we can use bean_regex to do specific targetted metrics for the
> important tables anyway.
>

Yes, this would work, but 350 is very limited for Cassandra dashboards. We
have a LOT of metrics available.

Datadog 350 metric limit is a PITA for tables once you get over 10 tables
>

I noticed this while I was working on providing default dashboards for
Cassandra-Datadog integration. I was told by Datadog team it would not be
an issue for users, that I should not care about it. As you pointed out,
per table metrics quickly increase the total number of metrics we need to
collect.

I believe you can set the following option: *"max_returned_metrics: 1000"* -
it can be used if metrics are missing to increase the limit of the number
of collected metrics. Be aware of CPU utilization that this might imply
(greatly improved in dd-agent version 6+ I believe -thanks Datadog teams
for that- making this fully usable for Cassandra). This option should go in
the *cassandra.yaml* file for Cassandra integrations, off the top of my
head.

Also, do not hesitate to reach to Datadog directly for this kind of
questions, I have always been very happy with their support so far, I am
sure they would guide you through this as well, probably better than we can
do :). It also provides them with feedback on what people are struggling
with I imagine.

I am interested to know if you still have issues getting more metrics
(option above not working / CPU under too much load) as this would make the
dashboards we built mostly unusable for clusters with more tables. We might
then need to review the design.

As a side note, I believe metrics are handled the same way cross version,
they got the same name/label for C*2.1, 2.2 and 3+ on Datadog. There is an
abstraction layer that removes this complexity (if I remember well, we
built those dashboards a while ago).

C*heers
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Le lun. 1 oct. 2018 à 19:38, Carl Mueller
<carl.mueller@smartthings.com.invalid> a écrit :

> That's great too, thank you.
>
> Datadog 350 metric limit is a PITA for tables once you get over 10 tables,
> but I guess we can use bean_regex to do specific targetted metrics for the
> important tables anyway.
>
> On Mon, Oct 1, 2018 at 4:21 AM Alain RODRIGUEZ <arodrime@gmail.com> wrote:
>
>> Hello Carl,
>>
>> Here is a message I sent to my team a few months ago. I hope this will be
>> helpful to you and more people around :). It might not be exhaustive and we
>> were moving from C*2.1 to C*3+ in this case, thus skipping C*2.2, but C*2.2
>> is similar to C*3.0 if I remember correctly in terms of metrics. Here it is
>> for what it's worth:
>>
>> Quite a few things changed between metric reporter in C* 2.1 and C*3.0.
>> - ColumnFamily --> Table
>> - XXpercentile --> pXX
>> - 1MinuteRate -->  m1_rate
>> - metric name before KS and Table names and some other changes of this
>> kind.
>> - ^ aggregations / aliases indexes changed because of this (using
>> graphite for example) ^
>> - ‘.value’ is not appended in the metric name anymore for gauges, nothing
>> instead.
>>
>> For example (graphite):
>>
>> From
>> aliasByNode(averageSeriesWithWildcards(cassandra.$env.$dc.$host.org.apache.cassandra.metrics.ColumnFamily.$ks.$table.ReadLatency.95percentile,
>> 2, 3), 1, 7, 8, 9)
>>
>> to
>> aliasByNode(averageSeriesWithWildcards(cassandra.$env.$dc.$host.org.apache.cassandra.metrics.Table.ReadLatency.$ks.$table.p95,
>> 2, 3), 1, 8, 9, 10)
>>
>> C*heers,
>> -----------------------
>> Alain Rodriguez - @arodream - alain@thelastpickle.com
>> France / Spain
>>
>> The Last Pickle - Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>> Le ven. 28 sept. 2018 à 20:38, Carl Mueller
>> <carl.mueller@smartthings.com.invalid> a écrit :
>>
>>> VERY NICE! Thank you very much
>>>
>>> On Fri, Sep 28, 2018 at 1:32 PM Lyuben Todorov <
>>> lyuben.todorov@instaclustr.com> wrote:
>>>
>>>> Nothing as fancy as a matrix but a list of what JMX term can see.
>>>> Link to the online diff here: https://www.diffchecker.com/G9FE9swS
>>>>
>>>> /lyubent
>>>>
>>>> On Fri, 28 Sep 2018 at 19:04, Carl Mueller
>>>> <carl.mueller@smartthings.com.invalid> wrote:
>>>>
>>>>> It's my understanding that metrics got heavily re-namespaced in JMX
>>>>> for 2.2 from 2.1
>>>>>
>>>>> Did anyone ever make a migration matrix/guide for conversion of old
>>>>> metrics to new metrics?
>>>>>
>>>>>
>>>>>

Mime
View raw message