Perhaps the deletes: https://issues.apache.org/jira/browse/CASSANDRA-3741
-Brandon
On Sun, Jun 3, 2012 at 6:12 PM, Poziombka, Wade L
<wade.l.poziombka@intel.com> wrote:
> Running a very write intensive (new column, delete old column etc.) process and failing
on memory. Log file attached.
>
> Curiously when I add new data I have never seen this have in past sent hundreds of millions
"new" transactions. It seems to be when I modify. my process is as follows
>
> key slice to get columns to modify in batches of 100, in separate threads modify those
columns. I advance the slice with the start key each with last key in previous batch. Mutations
done are update a column value in one column family(token), delete column and add new column
in another (pan).
>
> Runs well until after about 5 million rows then it seems to run out of memory. Note
that these column families are quite small.
>
> WARN [ScheduledTasks:1] 2012-06-03 17:49:01,558 GCInspector.java (line 145) Heap is 0.7967470834946492
full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up
to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
> INFO [ScheduledTasks:1] 2012-06-03 17:49:01,559 StorageService.java (line 2772) Unable
to reduce heap usage since there are no dirty column families
> INFO [GossipStage:1] 2012-06-03 17:49:01,999 Gossiper.java (line 797) InetAddress /10.230.34.170
is now UP
> INFO [ScheduledTasks:1] 2012-06-03 17:49:10,048 GCInspector.java (line 122) GC for
ParNew: 206 ms for 1 collections, 7345969520 used; max is 8506048512
> INFO [ScheduledTasks:1] 2012-06-03 17:49:53,187 GCInspector.java (line 122) GC for
ConcurrentMarkSweep: 12770 ms for 1 collections, 5714800208 used; max is 8506048512
>
> ----------------
> Keyspace: keyspace
> Read Count: 50042632
> Read Latency: 0.23157864418482224 ms.
> Write Count: 44948323
> Write Latency: 0.019460829472992797 ms.
> Pending Tasks: 0
> Column Family: pan
> SSTable count: 5
> Space used (live): 1977467326
> Space used (total): 1977467326
> Number of Keys (estimate): 16334848
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 74
> Read Count: 14985122
> Read Latency: 0.408 ms.
> Write Count: 19972441
> Write Latency: 0.022 ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 829
> Bloom Filter False Ratio: 0.00073
> Bloom Filter Space Used: 37048400
> Compacted row minimum size: 125
> Compacted row maximum size: 149
> Compacted row mean size: 149
>
> Column Family: token
> SSTable count: 4
> Space used (live): 1250973873
> Space used (total): 1250973873
> Number of Keys (estimate): 14217216
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 49
> Read Count: 30059563
> Read Latency: 0.167 ms.
> Write Count: 14985488
> Write Latency: 0.014 ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 13642
> Bloom Filter False Ratio: 0.00322
> Bloom Filter Space Used: 28002984
> Compacted row minimum size: 150
> Compacted row maximum size: 258
> Compacted row mean size: 224
>
> Column Family: counters
> SSTable count: 2
> Space used (live): 561549994
> Space used (total): 561549994
> Number of Keys (estimate): 9985024
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 38
> Read Count: 4997947
> Read Latency: 0.092 ms.
> Write Count: 9990394
> Write Latency: 0.023 ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 191
> Bloom Filter False Ratio: 0.37525
> Bloom Filter Space Used: 18741152
> Compacted row minimum size: 125
> Compacted row maximum size: 179
> Compacted row mean size: 150
>
> ----------------
|