incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Cassandra memory consumption
Date Wed, 09 Feb 2011 02:05:45 GMT
On Tue, Feb 8, 2011 at 4:56 PM, Victor Kabdebon
<victor.kabdebon@gmail.com> wrote:
> I will do that in the future and I will post my results here ( I upgraded
> the server to debian 6 to see if there is any change, so memory is back to
> normal). I will report in a few days.
> In the meantime I am open to any suggestion...
>
> 2011/2/8 Aaron Morton <aaron@thelastpickle.com>
>>
>> When you attach to the JVM with JConsole how much non heap memory and how
>> much heap memory is reported on the memory tab?
>> Xmx controls the total size of the heap memory, which excludes the
>> permanent generation.
>> see
>>
>> http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#generation_sizing
>> and
>>
>> http://blogs.suncom/jonthecollector/entry/presenting_the_permanent_generation
>> Total non-heap memory on a 0.7 box I have is around 27M. You numbers seem
>> large but it would be interesting to know what the JVM is reporting.
>> Aaron
>> On 09 Feb, 2011,at 05:57 AM, Victor Kabdebon <victor.kabdebon@gmail.com>
>> wrote:
>>
>> Information on the system :
>>
>> Debian 5
>> Jvm :
>> victor@testhost:~/database/apache-cassandra-0.6.6$ java -version
>> java version "1.6.0_22"
>> Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
>> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
>>
>> RAM : 2Go
>>
>>
>> 2011/2/8 Victor Kabdebon <victor.kabdebon@gmail.com>
>>>
>>> Sorry Jonathan :
>>>
>>> So most of these informations were taken using the command :
>>>
>>> sudo ps aux | grep cassandra
>>>
>>> For the nodetool information it is :
>>>
>>> /bin/nodetool --host localhost --port 8081 info
>>>
>>>
>>> Regars,
>>>
>>> Victor K.
>>>
>>>
>>> 2011/2/8 Jonathan Ellis <jbellis@gmail.com>
>>>
>>>> I missed the part where you explained where you're getting your numbers
>>>> from.
>>>>
>>>>
>>>> On Tue, Feb 8, 2011 at 9:32 AM, Victor Kabdebon
>>>> <victor.kabdebon@gmail.com> wrote:
>>>> > It is really weird that I am the only one to have this issue.
>>>> > I restarted Cassandra today and already the memory compution is over
>>>> > the
>>>> > limit :
>>>> >
>>>> > root      1739  4.0 24.5 664968 494996 pts/4   SLl  15:51  
0:12
>>>> > /usr/bin/java -ea -Xms128M -Xmx256M -XX:+UseParNewGC
>>>> > -XX:+UseConcMarkSweepGC
>>>> > -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>>>> > -XX:MaxTenuringThreshold=1
>>>> > -XX:CMSInitiatingOccupancyFraction=75
>>>> > -XX:+UseCMSInitiatingOccupancyOnly
>>>> > -XX:+HeapDumpOnOutOfMemoryError
>>>> > -Dcom.sun.management.jmxremote.port=8081
>>>> > -Dcom.sun.management.jmxremotessl=false
>>>> > -Dcom.sun.management.jmxremote.authenticate=false
>>>> > -Dstorage-config=bin/../conf -cp
>>>> >
>>>> > bin/../conf:bin/../build/classes:bin/../lib/antlr-3.1.3.jar:bin/../lib/apache-cassandra-06.6.jar:bin/../lib/avro-1.2.0-dev.jar:bin/../lib/cassandra-javautils.jar:bin/../lib/clhm-production.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-collections-3.2.1.jar:bin/../lib/commons-io-1.4.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/commons-pool-1.5.4.jar:bin/../lib/google-collections-1.0.jar:bin/../lib/hadoop-core-0.20.1.jar:bin/../lib/hector-0.6.0-14.jar:bin/../lib/high-scale-lib.jar:bin/./lib/ivy-2.1.0.jar:bin/../lib/jackson-core-asl-1.4.0.jar:bin/../lib/jackson-mapper-asl-1.4.0.jar:bin/../lib/jline-0.9.94.jar:bin/../lib/jna.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-r917130.jar:bin/../lib/log4j-1.2.14.jar:bin/../lib/perf4j-0.9.12.jar:bin/../lib/slf4j-api-1.5.8.jar:bin/../lib/slf4j-log4j12-1.5.8.jar:bin/../lib/uuid-3.1.jar
>>>> > org.apache.cassandra.thrift.CassandraDaemon
>>>> >
>>>> > It is really an annoying problem if we cannot really foresee memory
>>>> > consumption.
>>>> >
>>>> > Best regards,
>>>> > Victor K
>>>> >
>>>> > 2011/2/8 Victor Kabdebon <victor.kabdebon@gmail.com>
>>>> >>
>>>> >> Dear all,
>>>> >>
>>>> >> Sorry to come back again to this point but I am really worried about
>>>> >> Cassandra memory consumption. I have a single machine that runs
one
>>>> >> Cassandra server. There is almost no data on it but I see a crazy
>>>> >> memory
>>>> >> consumption and it doesn't care at all about the instructions...
>>>> >> Note that I am not using mmap, but "Standard", I use also JNA (inside
>>>> >> lib
>>>> >> folder), i am running on debian 5 64 bits, so a pretty normal
>>>> >> configuration.
>>>> >> I also use Cassandra 0.6.8.
>>>> >>
>>>> >>
>>>> >> Here are the informations I gathered on Cassandra :
>>>> >>
>>>> >> 105      16765  0.1 34.1 1089424 687476 ?      Sl  
Feb02  14:58I
>>>> >> think you are
>>>> >> /usr/bin/java -ea -Xms128M -Xmx256M -XX:+UseParNewGC
>>>> >> -XX:+UseConcMarkSweepGC
>>>> >> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>>>> >> -XX:MaxTenuringThreshold=1
>>>> >> -XX:CMSInitiatingOccupancyFraction=75
>>>> >> -XX:+UseCMSInitiatingOccupancyOnly
>>>> >> -XX:+HeapDumpOnOutOfMemoryError
>>>> >> -Dcom.sunmanagement.jmxremote.port=8081
>>>> >> -Dcom.sun.management.jmxremote.ssl=false
>>>> >> -Dcom.sun.management.jmxremote.authenticate=false
>>>> >> -Dstorage-config=bin/../conf -Dcassandra-foreground=yes -cp
>>>> >>
>>>> >> bin/../conf:bin/../build/classes:bin/../lib/antlr-3.1.3.jar:bin/../lib/apache-cassandra-0.6.6.jar:bin/../lib/avro-1.20-dev.jar:bin/../lib/cassandra-javautils.jar:bin/../lib/clhm-production.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-collections-3.2.1.jar:bin/../lib/commons-io-1.4.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/commons-pool-1.5.4.jar:bin/../lib/google-collections-1.0.jar:bin/../lib/hadoop-core-0.20.1.jar:bin/../lib/hector-0.6.0-14.jar:bin/../lib/high-scale-lib.jar:bin/../lib/ivy-2.1.0.jar:bin/../lib/jackson-core-asl-1.4.0.jar:bin/../lib/jackson-mapper-asl-1.4.0.jar:bin/./lib/jline-0.9.94.jar:bin/../lib/jna.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-r917130.jar:bin/../lib/log4j-1.2.14.jar:bin/../lib/perf4j-0.9.12.jar:bin/../lib/slf4j-api-1.5.8.jar:bin/../lib/slf4j-log4j12-1.5.8.jar:bin/../lib/uuid-3.1.jar
>>>> >> org.apache.cassandra.thrift.CassandraDaemon
>>>> >>
>>>> >> result of nodetool info :
>>>> >>
>>>> >> 116024732779488843382476400091948985708
>>>> >> Load             : 1,94 MB
>>>> >> Generation No    : 1296673772
>>>> >> Uptime (seconds) : 467550
>>>> >> Heap Memory (MB) : 120,26 / 253,94
>>>> >>
>>>> >>
>>>> >> I have about 21 column families, none of them have a lot of
>>>> >> information (
>>>> >> as you see I have 2 Mb of text which is really small). Even if I
set
>>>> >> Xmx at
>>>> >> 256 there is 687M of memory used. Where does this memory come from
?
>>>> >> Bad
>>>> >> garbage collection ? Something that I ignore ?
>>>> >> Thank you for your help I really need to get rid of that problem.
>>>> >>
>>>> >> Best regards,
>>>> >> Victor Kabdebon
>>>> >
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Jonathan Ellis
>>>> Project Chair, Apache Cassandra
>>>> co-founder of DataStax, the source for professional Cassandra support
>>>> http://www.datastax.com
>>>
>>
>
>

Each of your 21 column families will have its own memtable if you have
the default memtable settings your memory usage will grow quite large
over time. Have you tuned down your memtable size?

Mime
View raw message