cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan McGuire (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7567) when the commit_log disk for a single node is overwhelmed the entire cluster slows down
Date Fri, 25 Jul 2014 19:56:39 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074829#comment-14074829
] 

Ryan McGuire commented on CASSANDRA-7567:
-----------------------------------------

{code}
cqlsh> select activity, source, source_elapsed, thread from system_traces.events where
session_id = f0c14390-1434-11e4-bbc2-1511583f4f50;

 activity                                 | source         | source_elapsed | thread
------------------------------------------+----------------+----------------+-----------------------
    Message received from /10.109.137.124 | 10.111.173.249 |             50 |            
Thread-11
        Determining replicas for mutation | 10.109.137.124 |            158 |            Thrift:116
       Sending message to /10.111.173.249 | 10.109.137.124 |            303 | WRITE-/10.111.173.249
        Sending message to /10.180.41.228 | 10.109.137.124 |            303 |  WRITE-/10.180.41.228
    Message received from /10.109.137.124 |  10.180.41.228 |             36 |            
Thread-28
                   Appending to commitlog | 10.111.173.249 |          94484 |  SharedPool-Worker-29
             Adding to Standard1 memtable | 10.111.173.249 |          94569 |  SharedPool-Worker-29
    Enqueuing response to /10.109.137.124 | 10.111.173.249 |          94636 |  SharedPool-Worker-29
       Sending message to /10.109.137.124 | 10.111.173.249 |          94752 | WRITE-/10.109.137.124
    Message received from /10.111.173.249 | 10.109.137.124 |          95898 |            
 Thread-4
 Processing response from /10.111.173.249 | 10.109.137.124 |          96001 |  SharedPool-Worker-11

(11 rows)
{code}

> when the commit_log disk for a single node is overwhelmed the entire cluster slows down
> ---------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7567
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7567
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: debian 7.5, bare metal, 14 nodes, 64CPUs, 64GB RAM, commit_log disk
sata, data disk SSD, vnodes, leveled compaction strategy
>            Reporter: David O'Dell
>            Assignee: Brandon Williams
>         Attachments: 7567.logs.bz2, write_request_latency.png
>
>
> We've run into a situation where a single node out of 14 is experiencing high disk io.
This can happen when a node is being decommissioned or after it joins the ring and runs into
the bug cassandra-6621.
> When this occurs the write latency for the entire cluster spikes.
> From 0.3ms to 170ms.
> To simulate this simply run dd on the commit_log disk (dd if=/dev/zero of=/tmp/foo bs=1024)
and you will see that instantly all nodes in the cluster have slowed down.
> BTW overwhelming the data disk does not have this same effect.
> Also I've tried this where the overwhelmed node isn't being connected directly from the
client and it still has the same effect.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message