cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Łukasz Mrożkiewicz (JIRA) <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9798) Cassandra seems to have deadlocks during flush operations
Date Mon, 20 Jul 2015 12:16:04 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633494#comment-14633494
] 

Łukasz Mrożkiewicz commented on CASSANDRA-9798:
-----------------------------------------------

Hi Benedict, 
I've updated C* to 2.1.8. run test with 50Mb/s (the same result like 200Mb/s no difference)


I thought also that this problem is caused by hardware utilisation but I can't find any metric
that show problems with resources:
CPU: 92-98% idle
JVM - Memory: problem occured when 3GB from 8GB of heap is used too.
iostat shows that there is no queue to write on disks (it is usual view of iostat):
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await
 svctm  %util
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00
  0.00   0.00
sda               0.00     5.00    0.00    1.00     0.00     0.02    48.00     0.00    3.00
  3.00   0.30
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00
  0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00
  0.00   0.00
sde               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00
  0.00   0.00
sdf               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00
  0.00   0.00
sdg               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00
  0.00   0.00
sdh               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00
  0.00   0.00
sdi               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00
  0.00   0.00

nodetool tpstats says that MemtablePostFlush pending tasks is increasing and completed is
const
MemtableFlushWriter               6        29             37         0                 0
MemtablePostFlush                 1        33             47         0                 0
MemtableReclaimMemory             0         0             53         0                 0

I noticed one new thing:
on one node there is a dropped mutation stage:~200k (and it is stable from one hour)
but pending mutation stage: is continous increasing: now is 850k 
nothing appeared in logs:
INFO  11:21:15 Enqueuing flush of hints: 111288602 (5%) on-heap, 0 (0%) off-heap
WARN  11:40:40 Batch of prepared statements for [] is of size 6102, exceeding specified threshold
of 5120 by 982.
WARN  11:47:16 Batch of prepared statements for [] is of size 5640, exceeding specified threshold
of 5120 by 520.
WARN  11:50:22 Batch of prepared statements for [] is of size 5481, exceeding specified threshold
of 5120 by 361.
now 12:12 in logs - only 4 lines in 50 minutes. all resources are idle.



> Cassandra seems to have deadlocks during flush operations
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-9798
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9798
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: 4x HP Gen9 dl 360 servers
> 2x8 cpu each (Intel(R) Xeon E5-2667 v3 @ 3.20GHz)
> 6x900GB 10kRPM disk for data
> 1x900GB 10kRPM disk for commitlog
> 64GB ram
> ETH: 10Gb/s
> Red Hat Enterprise Linux Server release 6.6 (Santiago) 2.6.32-504.el6.x86_64
> java build 1.8.0_45-b14 (openjdk) (tested on oracle java 8 too)
>            Reporter: Łukasz Mrożkiewicz
>             Fix For: 2.1.x
>
>         Attachments: cassandra.2.1.8.log, cassandra.log, cassandra.yaml, gc.log.0.current
>
>
> Hi,
> We noticed some problem with dropped mutationstages. Usually on one random node there
is a situation that:
> MutationStage "active" is full, "pending" is increasing  "completed" is stalled.
> MemtableFlushWriter "active" 6, pending: 25 completed: stalled 
> MemtablePostFlush "active" is 1, pending 29 completed: stalled
> after a some time (30s-10min) pending mutations are dropped and everything is working.
> When it happened:
> 1. Cpu idle is ~95%
> 2. no gc long pauses or more activity.
> 3. memory usage 3.5GB form 8GB
> 4. only writes is processed by cassandra
> 5. when LOAD > 400GB/node problems appeared 
> 6. cassandra 2.1.6
> There is gap in logs:
> {code}
> INFO  08:47:01 Timed out replaying hints to /192.168.100.83; aborting (0 delivered)
> INFO  08:47:01 Enqueuing flush of hints: 7870567 (0%) on-heap, 0 (0%) off-heap
> INFO  08:47:30 Enqueuing flush of table1: 95301807 (4%) on-heap, 0 (0%) off-heap
> INFO  08:47:31 Enqueuing flush of table1: 60462632 (3%) on-heap, 0 (0%) off-heap
> INFO  08:47:31 Enqueuing flush of table2: 76973746 (4%) on-heap, 0 (0%) off-heap
> INFO  08:47:31 Enqueuing flush of table1: 84290135 (4%) on-heap, 0 (0%) off-heap
> INFO  08:47:32 Enqueuing flush of table3: 56926652 (3%) on-heap, 0 (0%) off-heap
> INFO  08:47:32 Enqueuing flush of table1: 85124218 (4%) on-heap, 0 (0%) off-heap
> INFO  08:47:33 Enqueuing flush of table2: 95663415 (4%) on-heap, 0 (0%) off-heap
> INFO  08:47:58 CompactionManager                 2        39
> INFO  08:47:58 Writing Memtable-table2@1767938721(13843064 serialized bytes, 162359 ops,
4%/0% of on/off-heap l
> imit)
> INFO  08:47:58 Writing Memtable-hints@1433125911(478703 serialized bytes, 424 ops, 0%/0%
of on/off-heap limit)
> INFO  08:47:58 Writing Memtable-table2@1318583275(11783615 serialized bytes, 137378 ops,
4%/0% of on/off-heap l
> imit)
> INFO  08:47:58 Enqueuing flush of compactions_in_progress: 969 (0%) on-heap, 0 (0%) off-heap
> INFO  08:47:58 Writing Memtable-table1@541175113(17221327 serialized bytes, 180792 ops,
4%/0% of on/off-heap
>  limit)
> INFO  08:47:58 Writing Memtable-table1@1361154669(27138519 serialized bytes, 273472 ops,
6%/0% of on/off-hea
> p limit)
> INFO  08:48:03 2176 MUTATION messages dropped in last 5000ms
> {code}
> use case:
> 100% write - 100Mb/s, couples of CF ~10column each. max cell size 100B
> CMS and G1GC tested - no difference



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message