incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vasileios Vlachos <vasileiosvlac...@gmail.com>
Subject Cassandra running out of memory?
Date Tue, 10 Apr 2012 13:54:01 GMT
Hello,

We are experimenting a bit with Cassandra lately (version 1.0.7) and we
seem to have some problems with memory. We use EC2 as our test environment
and we have three nodes with 3.7G of memory and 1 core @ 2.4G, all running
Ubuntu server 11.10.

The problem is that the node we hit from our thrift interface dies
regularly (approximately after we store 2-2.5G of data). Error message:
OutOfMemoryError: Java Heap Space and according to the log it in fact used
all of the allocated memory.

The nodes are under relatively constant load and store about 2000-4000 row
keys a minute, which are batched through the Trift interface in 10-30 row
keys at once (with about 50 columns each). The number of reads is very low
with around 1000-2000 a day and only requesting the data of a single row
key. The is currently only one used column family.

The initial thought was that something was wrong in the cassandra-env.sh
file. So, we specified the variables 'system_memory_in_mb' (3760) and the
'system_cpu_cores' (1) according to our nodes' specification. We also
changed the 'MAX_HEAP_SIZE' to 2G and the 'HEAP_NEWSIZE' to 200M (we think
the second is related to the Garbage Collection). Unfortunately, that did
not solve the issue and the node we hit via thrift keeps on dying regularly.

In case you find this useful, swap is off and unevictable memory seems to
be very high on all 3 servers (2.3GB, we usually observe the amount of
unevictable memory on other Linux servers of around 0-16KB) (We are not
quite sure how the unevictable memory ties into Cassandra, its just
something we observed while looking into the problem). The CPU is pretty
much idle the entire time. The heap memory is clearly being reduced once in
a while according to nodetool, but obviously grows over the limit as time
goes by.

Any ideas? Thanks in advance.

Bill

Mime
View raw message