cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: weird jvm metrics
Date Thu, 05 Jan 2017 18:59:01 GMT
On Thu, Jan 5, 2017 at 1:53 PM, Alain Rastoul <alf.mmm.cat@gmail.com> wrote:

> On 01/04/2017 11:12 PM, Edward Capriolo wrote:
>
>> The metric-reporter is actually leveraged from another project.
>>
>> https://github.com/addthis/metrics-reporter-config
>>
>> Check the version of metric-reporter (in cassandra/lib) and see if it
>> has changed from your old version to your new version.
>>
>>
>>
>>
>> On Wed, Jan 4, 2017 at 12:02 PM, Mike Torra <mtorra@demandware.com
>> <mailto:mtorra@demandware.com>> wrote:
>>
>>     Just bumping - has anyone seen this before?
>>
>>     http://stackoverflow.com/questions/41446352/cassandra-3-9-
>> jvm-metrics-have-bad-name
>>     <http://stackoverflow.com/questions/41446352/cassandra-3-9-
>> jvm-metrics-have-bad-name>
>>
>>     From: Mike Torra <mtorra@demandware.com <mailto:mtorra@demandware.com
>> >>
>>     Reply-To: "user@cassandra.apache.org
>>     <mailto:user@cassandra.apache.org>" <user@cassandra.apache.org
>>     <mailto:user@cassandra.apache.org>>
>>     Date: Wednesday, December 28, 2016 at 4:49 PM
>>     To: "user@cassandra.apache.org <mailto:user@cassandra.apache.org>"
>>     <user@cassandra.apache.org <mailto:user@cassandra.apache.org>>
>>     Subject: weird jvm metrics
>>
>>     Hi There -
>>
>>     I recently upgraded from cassandra 3.5 to 3.9 (DDC), and I noticed
>>     that the "new" jvm metrics are reporting with an extra '.' character
>>     in them. Here is a snippet of what I see from one of my nodes:
>>
>>     ubuntu@ip-10-0-2-163:~$ sudo tcpdump -i eth0 -v dst port 2003 -A |
>>     grep 'jvm'
>>
>>     tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture
>>     size 65535 bytes
>>
>>     .Je..l>.pi.cassandra.us-east-1.cassy-node1.jvm.buffers..dire
>> ct.capacity
>>     762371494 1482960946
>>
>>     pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.count 3054
>>     1482960946
>>
>>     pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.used
>>     762371496 1482960946
>>
>>     pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.capacity
>>     515226631134 1482960946
>>
>>     pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.count 45572
>>     1482960946
>>
>>     pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.used
>>     515319762610 1482960946
>>
>>     pi.cassandra.us-east-1.cassy-node1.jvm.fd.usage 0.00 1482960946
>>
>>
>>     My metrics.yaml looks like this:
>>
>>     graphite:
>>        -
>>          period: 60
>>          timeunit: 'SECONDS'
>>          prefix: 'pi.cassandra.us-east-1.cassy-node1'
>>          hosts:
>>           - host: '#RELAY_HOST#'
>>             port: 2003
>>          predicate:
>>            color: "white"
>>            useQualifiedName: true
>>            patterns:
>>              - "^org.+"
>>              - "^jvm.+"
>>              - "^java.lang.+"
>>
>>     All the org.* metrics come through fine, and the jvm.fd.usage metric
>>     strangely comes through fine, too. The rest of the jvm.* metrics
>>     have this extra '.' character that causes them to not show up in
>>     graphite.
>>
>>     Am I missing something silly here? Appreciate any help or suggestions.
>>
>>     - Mike
>>
>>
>> Hi,
>
> I also noticed this problem recently.
> Some jvm metrics have a double dot in name like:
> jvm.memory..total.max , jvm.memory..total.init (etc).
> Investigating a bit, it seems that an extra dot is added at the end of the
> name in CassandraDaemon.java, around line 367 (in 3.0.10):
> ...
>                 // enable metrics provided by metrics-jvm.jar
>
> CassandraMetricsRegistry.Metrics.register("jvm.buffers.", new
> BufferPoolMetricSet(ManagementFactory.getPlatformMBeanServer()));
>                 CassandraMetricsRegistry.Metrics.register("jvm.gc.", new
> GarbageCollectorMetricSet());
>
> CassandraMetricsRegistry.Metrics.register("jvm.memory.", new
> MemoryUsageGaugeSet());
>
> and also added in append method of MetricRegistry.
> Call stack is:
> MetricRegistry>>registerAll(String prefix, MetricSet metrics)
> MetricRegistry>>static String name(String name, String... names)
> MetricRegistry>>static void append(StringBuilder builder, String part)
>
> in append the dot is also added:
> ...
>             if(builder.length() > 0) {
>                 builder.append('.');
>             }
>             builder.append(part);
> ...
>
> The codahale MetricRegistry class seems to have no recent modification of
> name or append methods, so it looks like a small (but annoying) bug.
> May be the fix could be to simply remove the final dot ?
>
> Is it worth opening an issue in JIRA ?
>
>
> --
> best,
> Alain
>
>
Good troubleshooting. I would open a Jira. It seems like a good solution
would be to replace '..' with '.' somehow. It seems like no one would every
want ..

Mime
View raw message