Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Mon, 29 Jun 2015 20:59:06 +0000 (UTC)
From: "Eric Evans (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12839206.1434739251000.49493.1435611546840@Atlassian.JIRA>
In-Reply-To: <JIRA.12839206.1434739251000@Atlassian.JIRA>
References: <JIRA.12839206.1434739251000@Atlassian.JIRA>
 <JIRA.12839206.1434739251924@arcas>
Subject: [jira] [Comment Edited] (CASSANDRA-9625) GraphiteReporter not
 reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606384#comment-14606384 ] 

Eric Evans edited comment on CASSANDRA-9625 at 6/29/15 8:58 PM:
----------------------------------------------------------------

I'm not surprised you're unable to replicate, I can't replicate this in my staging environment (same software, fewer machines, less data, less traffic, etc).

{quote}
...do you have any logs or can you tell me how many column families you have?
{quote}

The logs are completely normal, only the usual startup messages, nothing else.  I locally patched the reporter on one machine and added copious logging statements, the last one to execute was at the beginning of this method: https://github.com/dropwizard/metrics/blob/v2.2.0/metrics-graphite/src/main/java/com/yammer/metrics/reporting/GraphiteReporter.java#L240 (it never makes it to the statement at the end).

I have about 150 column families.

{quote}
Also are you verifying that graphite didn't just poop out?
{quote}

Yeah, we use Graphite quite a bit, there'd be no missing an outage there.


was (Author: urandom):
I'm not surprised you're unable to replicate, I can't replicate this in my staging environment (same software, fewer machines, less data, less traffic, etc).

{quote}
...do you have any logs or can you tell me how many column families you have?
{quote}

The logs are completely normal, only the usual startup messages, nothing else.  I locally patched the reporter on one machine and added a copious logging statements, the last one to execute was at the beginning of this method: https://github.com/dropwizard/metrics/blob/v2.2.0/metrics-graphite/src/main/java/com/yammer/metrics/reporting/GraphiteReporter.java#L240 (it never makes it to the statement at the end).

I have about 150 column families.

{quote}
Also are you verifying that graphite didn't just poop out?
{quote}

Yeah, we use Graphite quite a bit, there'd be no missing an outage there.

> GraphiteReporter not reporting
> ------------------------------
>
>                 Key: CASSANDRA-9625
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>            Reporter: Eric Evans
>            Assignee: T Jake Luciani
>             Fix For: 2.1.x
>
>         Attachments: metrics.yaml, thread-dump.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops working.  The usual startup is logged, and one batch of samples is sent, but the reporting interval comes and goes, and no other samples are ever sent.  The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not on a 3 node (otherwise identical) staging cluster (maybe it takes a certain level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)