Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Thu, 5 Jan 2017 09:59:58 +0000 (UTC)
From: "Stefan Podkowinski (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.13032021.1483549446000.648617.1483610398454@Atlassian.JIRA>
In-Reply-To: <JIRA.13032021.1483549446000@Atlassian.JIRA>
References: <JIRA.13032021.1483549446000@Atlassian.JIRA> <JIRA.13032021.1483549446312@arcas>
Subject: [jira] [Commented] (CASSANDRA-13096) Snapshots slow down jmx
 scrapping
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 05 Jan 2017 10:00:05 -0000


    [ https://issues.apache.org/jira/browse/CASSANDRA-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15800953#comment-15800953 ] 

Stefan Podkowinski commented on CASSANDRA-13096:
------------------------------------------------

I think you should at least limit scraping to metric mbeans by having a whitelist as in [this|https://github.com/prometheus/jmx_exporter#configuration] example. Constantly polling every attribute of all mbeans sounds like trouble to me, as even reading non-metric mbeans may have side effects.

> Snapshots slow down jmx scrapping
> ---------------------------------
>
>                 Key: CASSANDRA-13096
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13096
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Maxime Fouilleul
>         Attachments: CPU Load.png, Clear Snapshots.png, JMX Scrape Duration.png
>
>
> Hello,
> We are scraping the jmx metrics through a prometheus exporter and we noticed that some nodes became really long to answer (more than 20 seconds). After some investigations we do not find any hardware problem or overload issues on there "slow" nodes. It happens on different clusters, some with only few giga bytes of dataset and it does not seams to be related to a specific version neither as it happens on 2.1, 2.2 and 3.0 nodes. 
> After some unsuccessful actions, one of our ideas was to clean the snapshots staying on one problematic node:
> {code}
> nodetool clearsnapshot
> {code}
> And the magic happens... as you can see in the attached diagrams, the second we cleared the snapshots, the CPU activity dropped immediatly and the duration to scrape the jmx metrics goes from +20 secs to instantaneous...
> Can you enlighten us on this issue? Once again, it appears on our three 2.1, 2.2 and 3.0 versions, on different volumetry and it is not systematically linked to the snapshots as we have some nodes with the same snapshots volume which are going pretty well.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)