cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Govindaraj (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-10439) Cassandra Read Request Latency
Date Fri, 02 Oct 2015 18:41:26 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-10439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Govindaraj updated CASSANDRA-10439:
-----------------------------------
    Attachment: Screen Shot 2015-10-02 at 2.39.57 PM.png

> Cassandra Read Request Latency
> ------------------------------
>
>                 Key: CASSANDRA-10439
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10439
>             Project: Cassandra
>          Issue Type: Task
>         Environment: PROD
>            Reporter: Govindaraj
>         Attachments: Screen Shot 2015-10-02 at 1.40.31 PM.png, Screen Shot 2015-10-02
at 12.08.06 PM.png, Screen Shot 2015-10-02 at 2.28.31 PM.png, Screen Shot 2015-10-02 at 2.28.39
PM.png, Screen Shot 2015-10-02 at 2.28.48 PM.png, Screen Shot 2015-10-02 at 2.28.54 PM.png,
Screen Shot 2015-10-02 at 2.39.57 PM.png
>
>
> Hi Team,
> Our PROD environment has two Data Centers configured as below.
> DC1 - Has 3 Cassandra nodes (dseĀ 4.7.0)
> DC2 - Has 3 Cassandra nodes (dseĀ 4.7.0)
> We are seeing the below issues repeatedly
> 1. Repeated alerts for Cassandra Read Request Latency. ReadStage has a lot of Pending
queue.
> #nodetool tpstats
> Pool Name                    Active   Pending      Completed   Blocked  All time blocked
> MutationStage                     0         0        8493347         0              
  0
> ReadStage                        32      3699           5835         0              
  0
> Also during the same time we see a high CPU load on the cassandra nodes
>  top
> top - 14:41:00 up 6 days, 21:58,  2 users,  load average: 33.75, 27.80, 21.41
> Tasks: 226 total,   1 running, 225 sleeping,   0 stopped,   0 zombie
> Cpu(s): 96.8%us,  1.8%sy,  1.1%ni,  0.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Mem:  16331716k total, 16016864k used,   314852k free,   189380k buffers
> Swap:        0k total,        0k used,        0k free,  5786164k cached
> 2. We stopped the COMPACTION on the nodes were CPU load is high. Immediately the load
came down and the read stage queue was cleared. All the pending tasks were processed in the
Read Stage queue.
> Can you please suggest what might be causing this issue? Also how can we troubleshoot
and fix it?
> Thanks,
> Venky



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message