cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: Cassandra in less than 1G of memory?
Date Sun, 16 Jan 2011 21:27:00 GMT
The OS will make it's best guess as to how much memory if can give over to mmapped files. Unfortunately
it will not always makes the best decision, see the information on adding JNA and mlockall()
support in cassandra 0.6.5 http://www.datastax.com/blog/whats-new-cassandra-065

As Jonathan says, try setting the disk mode to standard to see the difference. 

WRT the resident memory for the process, not all memory allocation is done on the heap. To
see the non heap usage connect to the processing using JConsole and take a look at the Memory
tab. For example on my box now Cassandra has 110M of heap memory and 20M of non heap. AFAIK
memory such as the class definitions are not included in the heap memory usage. 

Hope that helps. 
Aaron


On 15 Jan, 2011,at 08:03 PM, Victor Kabdebon <victor.kabdebon@gmail.com> wrote:

Hi Jonathan, hi Edward,

Jonathan : but it looks like mmaping wants to consume the entire memory of my server. It goes
up to 1.7 Gb for a ridiculously small amount of data.
Am I doing something wrong or is there something I should change to prevent this never ending
increase of memory consumption ?
Edward : I am not sure, I will try to see that tomorrow but my disk access mode is standard,
not mmap.

Anyway thank you very much,
Victor K.

PS : here is some hours after the result of ps aux | grep cassandra
root     19093  0.1 30.0 1243940 605060 ?      Sl   Jan11  10:15 /usr/bin/java
-ea -Xms128M -Xmx512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.port=8081 -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.managementjmxremote.authenticate=false -Dstorage-config=bin/../conf -Dcassandra-foreground=yes
-cp bin/../conf:bin/../build/classes:bin/../lib/antlr-3.1.3.jar:bin/../lib/apache-cassandra-0.6.6.jar:bin/../lib/avro-1.2.0-dev.jar:bin/../lib/cassandra-javautils.jar:bin/../lib/clhm-production.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-collections-3.2.1.jar:bin/../lib/commons-io-1.4.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/commons-pool-1.5.4.jar:bin/../lib/google-collections-1.0.jar:bin/../lib/hadoop-core-0.20.1.jar:bin/../lib/hector-0.6.0-14.jar:bin/../lib/high-scale-lib.jar:bin/../lib/ivy-2.1.0.jar:bin/../lib/jackson-core-asl-1.4.0.jar:bin/../lib/jackson-mapper-asl-1.4.0.jar:bin/../lib/jline-0.9.94.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-r917130jar:bin/../lib/log4j-1.2.14.jar:bin/../lib/perf4j-0.9.12.jar:bin/../lib/slf4j-api-1.5.8.jar:bin/../lib/slf4j-log4j12-1.5.8.jar:bin/../lib/uuid-3.1.jar
org.apache.cassandra.thrift.CassandraDaemon


2011/1/15 Jonathan Ellis <jbellis@gmail.com>
mmapping only consumes memory that the OS can afford to feed it.


On Fri, Jan 14, 2011 at 7:29 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
> On Fri, Jan 14, 2011 at 2:13 PM, Victor Kabdebon
> <victor.kabdebon@gmail.com> wrote:
>> Dear rajat,
>>
>> Yes it is possible, I have the same constraints. However I must warn you,
>> from what I see Cassandra memory consumption is not bounded in 0.6X on
>> debian 64 Bit
>>
>> Here is an example of an instance launch in a node :
>>
>> root     19093  0.1 28.3 1210696 570052 ?      Sl   Jan11   9:08
>> /usr/bin/java -ea -Xms128M -Xmx512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
>> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.port=8081
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.managementjmxremote.authenticate=false
>> -Dstorage-config=bin/../conf -Dcassandra-foreground=yes -cp
>> bin/../conf:bin/../build/classes:bin/../lib/antlr-3.1.3.jar:bin/../lib/apache-cassandra-0.6.6.jar:bin/../lib/avro-1.2.0-dev.jar:bin/../lib/cassandra-javautils.jar:bin/../lib/clhm-production.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-collections-3.2.1.jar:bin/../lib/commons-io-1.4.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/commons-pool-1.5.4.jar:bin/../lib/google-collections-1.0.jar:bin/../lib/hadoop-core-0.20.1.jar:bin/../lib/hector-06.0-14.jar:bin/../lib/high-scale-lib.jar:bin/../lib/ivy-2.1.0.jar:bin/../lib/jackson-core-asl-1.4.0.jar:bin/../lib/jackson-mapper-asl-1.4.0.jar:bin/../lib/jline-0.9.94.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-r917130.jar:bin/../lib/log4j-1.2.14.jar:bin/../lib/perf4j-0.9.12.jar:bin/../lib/slf4j-api-1.5.8.jar:bin/../lib/slf4j-log4j12-1.5.8.jar:bin/../lib/uuid-3.1.jar
>> org.apache.cassandra.thrift.CassandraDaemon
>>
>> Look at the second bold value, Xmx indicates the maximum memory that
>> cassandra can use; it is set to be 512, so it could easily fit into 1 Gb.
>> Now look at the first one, 570Mb > 512 Mb. Moreover if I come back in one
>> day the first value will be even higher. Probably around 610 Mb. Actually it
>> increases to the point where I need to restart it otherwise other program
>> are shut down by Linux for cassandra to further expand its memory usage...
>>
>> By the way it's a call to other cassandra users, am I the only one to
>> encounter this problem ?
>>
>> Best regards,
>>
>> Victor K.
>>
>> 2011/1/14 Rajat Chopra <rchopra@makara.com>
>>>
>>> Hello.
>>>
>>>
>>>
>>> According to  JVM heap size topic at
>>> http://wiki.apache.org/cassandra/MemtableThresholds , Cassandra would need
>>> atleast 1G of memory to run. Is it possible to have a running Cassandra
>>> cluster with machines that have less than that memory… say 512M?
>>>
>>> I can live with slow transactions, no compactions etc, but do not want an
>>> OutOfMemory error. The reason for a smaller bound for Cassandra is that I
>>> want to leave room for other processes to run.
>>>
>>>
>>>
>>> Please help with specific parameters to tune.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Rajat
>>>
>>>
>>
>
> -Xmx512M is not an overall memory limit. MMAP'ed files also consume
> memory. Try turning disk access mode to standard not (MMAP or
> MMAP_INDEX_ONLY).
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
    • Unnamed multipart/related (inline, None, 0 bytes)
View raw message