cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Evans (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-1177) OutOfMemory on heavy inserts
Date Thu, 17 Jun 2010 23:00:24 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879992#action_12879992
] 

Eric Evans commented on CASSANDRA-1177:
---------------------------------------

bq. After getting even further with our import, because we lowered the thresholds for the
Memtable flushing, we seeing one node not answering over JMX and start to print more and more
GC messages to the log. However we looked into the data and commitlog directory and find that
the listing of what's inside might be helpful to solve our problem.

I don't think there is anything particularly telling here, other than that the behavior you're
seeing still see falls within the range of what is expected (based on what we know).

I would suggest we move this discussion to user@cassandra.apache.org; the mailing list is
a better forum for this sort of discussion. If it becomes apparent that there is indeed a
bug, we can move it back here along with a summary or a pointer to the thread.

Be sure to include the rate of write operations, the size of the writes, the consistency level
being used, how many nodes are involved, along with your most recent configuration.



> OutOfMemory on heavy inserts
> ----------------------------
>
>                 Key: CASSANDRA-1177
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1177
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.6.2
>         Environment: SunOS 5.10, x86 32bit, Jave Hotspot Server VM 11.2-b01 mixed mode
> Sun SDK 1.6.0_12-b04
>            Reporter: Torsten Curdt
>            Priority: Critical
>         Attachments: bug report.zip, commitlog.txt, data.txt
>
>
> We have cluster of 6 Cassandra 0.6.2 nodes running under SunOS (see environment).
> On initial import (using the thrift API) we see some weird behavior of half the cluster.
While cas04-06 look fine as you can see from the attached munin graphs, the other 3 nodes
kept on GCing (see log file) until they became unreachable and went OOM. (This is also why
the stats are so spotty - munin could no longer reach the boxes) We have seen the same behavior
on 0.6.2 and 0.6.1. This started after around 100 million inserts.
> Looking at the hprof (which is of course to big to attach) we see lots of ConcurrentSkipListMap$Node's
and quite some Column objects. Please see the stats attached.
> This looks similar to https://issues.apache.org/jira/browse/CASSANDRA-1014 but we are
not sure it really is the same.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message