cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6149) OOM in Cassandra 2.0.1
Date Sun, 06 Oct 2013 17:43:48 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13787701#comment-13787701
] 

Kai Wang commented on CASSANDRA-6149:
-------------------------------------

I got a bunch of "less than" estimation. Here is the part right before OOM:

...
DEBUG [ReadStage:32] 2013-10-06 13:33:36,610 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
DEBUG [ReadStage:22] 2013-10-06 13:33:36,614 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464191488
DEBUG [ReadStage:22] 2013-10-06 13:33:36,614 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
 INFO [ScheduledTasks:1] 2013-10-06 13:33:36,913 GCInspector.java (line 116) GC for ConcurrentMarkSweep:
962 ms for 4 collections, 1036218016 used; max is 1037959168
DEBUG [ReadStage:24] 2013-10-06 13:33:36,916 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464257024
DEBUG [ReadStage:24] 2013-10-06 13:33:36,916 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\Keyword\Keyspace1-Keyword-jb-1-Data.db
DEBUG [ReadStage:25] 2013-10-06 13:33:36,918 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464322560
DEBUG [ReadStage:25] 2013-10-06 13:33:36,919 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
DEBUG [ReadStage:19] 2013-10-06 13:33:36,920 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464388096
DEBUG [ReadStage:19] 2013-10-06 13:33:36,920 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
DEBUG [ReadStage:10] 2013-10-06 13:33:36,939 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464453632
DEBUG [ReadStage:10] 2013-10-06 13:33:36,940 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
DEBUG [ReadStage:10] 2013-10-06 13:33:36,941 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464519168
DEBUG [ReadStage:10] 2013-10-06 13:33:36,941 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
DEBUG [ReadStage:30] 2013-10-06 13:33:37,171 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464584704
DEBUG [ReadStage:30] 2013-10-06 13:33:37,171 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
DEBUG [ReadStage:7] 2013-10-06 13:33:37,171 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464584704
DEBUG [ReadStage:8] 2013-10-06 13:33:37,171 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464584704
DEBUG [ReadStage:8] 2013-10-06 13:33:37,171 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
DEBUG [ReadStage:11] 2013-10-06 13:33:37,171 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464584704
DEBUG [ReadStage:11] 2013-10-06 13:33:37,171 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
DEBUG [ReadStage:7] 2013-10-06 13:33:37,171 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
DEBUG [ReadStage:17] 2013-10-06 13:33:39,618 FileCacheService.java (line 127) Estimated memory
usage is 327680 compared to actual usage 464846848
DEBUG [ReadStage:17] 2013-10-06 13:33:39,848 FileCacheService.java (line 137) Estimated memory
usage 327680 is less than threshold 258998272; caching reader for D:\var\lib\cassandra\data\Keyspace1\MetaData\Keyspace1-MetaData-jb-1-Data.db
ERROR [ReadStage:23] 2013-10-06 13:33:39,848 CassandraDaemon.java (line 185) Exception in
thread Thread[ReadStage:23,5,main]
java.lang.OutOfMemoryError: Java heap space
...

It does look like an estimation error.

> OOM in Cassandra 2.0.1
> ----------------------
>
>                 Key: CASSANDRA-6149
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6149
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Windows 7 64 bit. Java 64-bit 1.7.0_25. Cassandra 2.0.1
>            Reporter: Kai Wang
>              Labels: newbie
>         Attachments: 6149-debug.txt
>
>
> I have a program to stress test Cassandra. What it does is remove/insert rows with a
small set of row keys as fast as possible. Two CFs are involved. When I test against C* 1.2.3
with default configurations, it ran for 24 hours and C* doesn't having any issue. However
after I upgraded to C* 2.0.1, C* crashes on OOM within 1-2 minutes. I can consistently reproduce
this.
> I built C* from the source and found out the last good changeset is cfa097cdd5e28d7fe8204248e246a1fae226d2c0.
As soon as I include the next changeset 1e0d9513b748fae4ec0737283da71c65e9272102, C* starts
to crash. What's interesting is although it seems the change was reverted by fc1a7206fe15882fd64e7ba8eb68ba9dc320275f.
C* built from fc1a7206fe15882fd64e7ba8eb68ba9dc320275f has the same problem - OOM within minutes.
> I didn't test against the official 2.0.0. But the C* built from 03045ca22b11b0e5fc85c4fabd83ce6121b5709b
seems OK. I assume that's what 2.0.0 is.
> I use default configurations in all cases. I didn't tune anything.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message