cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Kovyrin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9625) GraphiteReporter not reporting
Date Thu, 13 Aug 2015 16:39:46 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695532#comment-14695532
] 

Alexey Kovyrin commented on CASSANDRA-9625:
-------------------------------------------

In our case the issues have started when the cluster went over some large number of segments
in it and we've started seeing a lot of merging all the time. Then our nagios checks using
jolokia started timing out because the pages those checks were pulling from cassandra nodes
were huge (full of merge history items) and around the same time our graphite metrics stopped
being reported. A node restart helps with the issue for a few hours and then the server stops
sending anything again.

> GraphiteReporter not reporting
> ------------------------------
>
>                 Key: CASSANDRA-9625
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>            Reporter: Eric Evans
>            Assignee: T Jake Luciani
>         Attachments: metrics.yaml, thread-dump.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops working.  The
usual startup is logged, and one batch of samples is sent, but the reporting interval comes
and goes, and no other samples are ever sent.  The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment on 2.1.6;
We are able to reproduce this on all 6 of production nodes, but not on a 3 node (otherwise
identical) staging cluster (maybe it takes a certain level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message