cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Evans (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-9625) GraphiteReporter not reporting
Date Mon, 29 Jun 2015 20:59:06 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606384#comment-14606384
] 

Eric Evans edited comment on CASSANDRA-9625 at 6/29/15 8:58 PM:
----------------------------------------------------------------

I'm not surprised you're unable to replicate, I can't replicate this in my staging environment
(same software, fewer machines, less data, less traffic, etc).

{quote}
...do you have any logs or can you tell me how many column families you have?
{quote}

The logs are completely normal, only the usual startup messages, nothing else.  I locally
patched the reporter on one machine and added copious logging statements, the last one to
execute was at the beginning of this method: https://github.com/dropwizard/metrics/blob/v2.2.0/metrics-graphite/src/main/java/com/yammer/metrics/reporting/GraphiteReporter.java#L240
(it never makes it to the statement at the end).

I have about 150 column families.

{quote}
Also are you verifying that graphite didn't just poop out?
{quote}

Yeah, we use Graphite quite a bit, there'd be no missing an outage there.


was (Author: urandom):
I'm not surprised you're unable to replicate, I can't replicate this in my staging environment
(same software, fewer machines, less data, less traffic, etc).

{quote}
...do you have any logs or can you tell me how many column families you have?
{quote}

The logs are completely normal, only the usual startup messages, nothing else.  I locally
patched the reporter on one machine and added a copious logging statements, the last one to
execute was at the beginning of this method: https://github.com/dropwizard/metrics/blob/v2.2.0/metrics-graphite/src/main/java/com/yammer/metrics/reporting/GraphiteReporter.java#L240
(it never makes it to the statement at the end).

I have about 150 column families.

{quote}
Also are you verifying that graphite didn't just poop out?
{quote}

Yeah, we use Graphite quite a bit, there'd be no missing an outage there.

> GraphiteReporter not reporting
> ------------------------------
>
>                 Key: CASSANDRA-9625
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>            Reporter: Eric Evans
>            Assignee: T Jake Luciani
>             Fix For: 2.1.x
>
>         Attachments: metrics.yaml, thread-dump.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops working.  The
usual startup is logged, and one batch of samples is sent, but the reporting interval comes
and goes, and no other samples are ever sent.  The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment on 2.1.6;
We are able to reproduce this on all 6 of production nodes, but not on a 3 node (otherwise
identical) staging cluster (maybe it takes a certain level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message