Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Tue, 29 Jul 2014 21:53:39 +0000 (UTC)
From: "Ryan McGuire (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12727994.1405619151526.70101.1406670819428@arcas>
In-Reply-To: <JIRA.12727994.1405619151526@arcas>
References: <JIRA.12727994.1405619151526@arcas>
Subject: [jira] [Commented] (CASSANDRA-7567) when the commit_log disk for a
 single node is overwhelmed the entire cluster slows down
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-7567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078451#comment-14078451 ] 

Ryan McGuire commented on CASSANDRA-7567:
-----------------------------------------

I'll rerun on both 2.1 and 2.0 and capture both a trace and matching logs. The logs were from a different cluster, matching to results in [this comment|https://issues.apache.org/jira/browse/CASSANDRA-7567?focusedCommentId=14072373&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14072373] but I started a new cluster (EC2, so not same IP) for the comment with the trace.

> when the commit_log disk for a single node is overwhelmed the entire cluster slows down
> ---------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7567
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7567
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: debian 7.5, bare metal, 14 nodes, 64CPUs, 64GB RAM, commit_log disk sata, data disk SSD, vnodes, leveled compaction strategy
>            Reporter: David O'Dell
>            Assignee: Brandon Williams
>         Attachments: 7567.logs.bz2, write_request_latency.png
>
>
> We've run into a situation where a single node out of 14 is experiencing high disk io. This can happen when a node is being decommissioned or after it joins the ring and runs into the bug cassandra-6621.
> When this occurs the write latency for the entire cluster spikes.
> From 0.3ms to 170ms.
> To simulate this simply run dd on the commit_log disk (dd if=/dev/zero of=/tmp/foo bs=1024) and you will see that instantly all nodes in the cluster have slowed down.
> BTW overwhelming the data disk does not have this same effect.
> Also I've tried this where the overwhelmed node isn't being connected directly from the client and it still has the same effect.


--
This message was sent by Atlassian JIRA
(v6.2#6252)