incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Poziombka, Wade L" <wade.l.poziom...@intel.com>
Subject RE: memory issue on 1.1.0
Date Tue, 05 Jun 2012 04:01:34 GMT
I have repeated the test on two quite large machines 12 core, 64 GB as5 boxes and still observed
the problem.  Interestingly about at the same point.

Anything I can monitor... perhaps I'll hook the Yourkit profiler up to it to see if there
is some kind of leak?

Wade

From: Poziombka, Wade L
Sent: Monday, June 04, 2012 7:23 PM
To: user@cassandra.apache.org
Subject: RE: memory issue on 1.1.0

What JVM settings do you have?
        -Xms8G
        -Xmx8G
        -Xmn800m
        -XX:+HeapDumpOnOutOfMemoryError
        -XX:+UseParNewGC
        -XX:+UseConcMarkSweepGC
        -XX:+CMSParallelRemarkEnabled
        -XX:SurvivorRatio=8
        -XX:MaxTenuringThreshold=1
        -XX:CMSInitiatingOccupancyFraction=75
        -XX:+UseCMSInitiatingOccupancyOnly
        -Djava.rmi.server.hostname=127.0.0.1
        -Djava.net.preferIPv4Stack=true
        -Dcassandra-pidfile=cassandra.pid


What is the machine spec ?
It is an RH AS5 x64
16gb memory
2 CPU cores 2.8 Ghz

As it turns out it is somewhat wimpier than I thought.  While weak on it does have a good
amount of memory.
It is paired with a larger machine.


What settings do you have for key and row cache ?
A: All the defaults. (yaml "template" attached);

Do the CF's have secondary indexes ?
A: Yes one has two.  One of them is used in the key slice used to get the row keys used to
do the further mutations.

How many clients / requests per second ?
A: One client process with 10 threads connected to one of the two nodes in the cluster.  On
thread reading the slice and putting work in a queue.  9 others reading from this queue and
applying the mutations.  Mutations are completing at about 20,000/minute roughly.


From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Monday, June 04, 2012 4:17 PM
To: user@cassandra.apache.org
Subject: Re: memory issue on 1.1.0

Had a look at the log, this message

INFO [ScheduledTasks:1] 2012-06-03 17:49:01,559 StorageService.java (line 2772) Unable to
reduce heap usage since there are no dirty column families
appears correct, it happens after some flush activity and there are not CF's with memtable
data. But the heap is still full.

Overall the server is overloaded, but it seems like it should be handling it better.

What JVM settings do you have? What is the machine spec ?
What settings do you have for key and row cache ?
Do the CF's have secondary indexes ?
How many clients / requests per second ?

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 4/06/2012, at 11:12 AM, Poziombka, Wade L wrote:

Running a very write intensive (new column, delete old column etc.) process and failing on
memory.  Log file attached.

Curiously when I add new data I have never seen this have in past sent hundreds of millions
"new" transactions.  It seems to be when I modify.  my process is as follows

key slice to get columns to modify in batches of 100, in separate threads modify those columns.
 I advance the slice with the start key each with last key in previous batch.  Mutations done
are update a column value in one column family(token), delete column and add new column in
another (pan).

Runs well until after about 5 million rows then it seems to run out of memory.  Note that
these column families are quite small.

WARN [ScheduledTasks:1] 2012-06-03 17:49:01,558 GCInspector.java (line 145) Heap is 0.7967470834946492
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
INFO [ScheduledTasks:1] 2012-06-03 17:49:01,559 StorageService.java (line 2772) Unable to
reduce heap usage since there are no dirty column families
INFO [GossipStage:1] 2012-06-03 17:49:01,999 Gossiper.java (line 797) InetAddress /10.230.34.170
is now UP
INFO [ScheduledTasks:1] 2012-06-03 17:49:10,048 GCInspector.java (line 122) GC for ParNew:
206 ms for 1 collections, 7345969520 used; max is 8506048512
INFO [ScheduledTasks:1] 2012-06-03 17:49:53,187 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
12770 ms for 1 collections, 5714800208 used; max is 8506048512

----------------
Keyspace: keyspace
       Read Count: 50042632
       Read Latency: 0.23157864418482224 ms.
       Write Count: 44948323
       Write Latency: 0.019460829472992797 ms.
       Pending Tasks: 0
               Column Family: pan
               SSTable count: 5
               Space used (live): 1977467326
               Space used (total): 1977467326
               Number of Keys (estimate): 16334848
               Memtable Columns Count: 0
               Memtable Data Size: 0
               Memtable Switch Count: 74
               Read Count: 14985122
               Read Latency: 0.408 ms.
               Write Count: 19972441
               Write Latency: 0.022 ms.
               Pending Tasks: 0
               Bloom Filter False Postives: 829
               Bloom Filter False Ratio: 0.00073
               Bloom Filter Space Used: 37048400
               Compacted row minimum size: 125
               Compacted row maximum size: 149
               Compacted row mean size: 149

               Column Family: token
               SSTable count: 4
               Space used (live): 1250973873
               Space used (total): 1250973873
               Number of Keys (estimate): 14217216
               Memtable Columns Count: 0
               Memtable Data Size: 0
               Memtable Switch Count: 49
               Read Count: 30059563
               Read Latency: 0.167 ms.
               Write Count: 14985488
               Write Latency: 0.014 ms.
               Pending Tasks: 0
               Bloom Filter False Postives: 13642
               Bloom Filter False Ratio: 0.00322
               Bloom Filter Space Used: 28002984
               Compacted row minimum size: 150
               Compacted row maximum size: 258
               Compacted row mean size: 224

               Column Family: counters
               SSTable count: 2
               Space used (live): 561549994
               Space used (total): 561549994
               Number of Keys (estimate): 9985024
               Memtable Columns Count: 0
               Memtable Data Size: 0
               Memtable Switch Count: 38
               Read Count: 4997947
               Read Latency: 0.092 ms.
               Write Count: 9990394
               Write Latency: 0.023 ms.
               Pending Tasks: 0
               Bloom Filter False Postives: 191
               Bloom Filter False Ratio: 0.37525
               Bloom Filter Space Used: 18741152
               Compacted row minimum size: 125
               Compacted row maximum size: 179
               Compacted row mean size: 150

----------------
<cassandra.zip>


Mime
View raw message