incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Poziombka, Wade L" <wade.l.poziom...@intel.com>
Subject memory issue on 1.1.0
Date Sun, 03 Jun 2012 23:12:32 GMT
Running a very write intensive (new column, delete old column etc.) process and failing on
memory.  Log file attached.

Curiously when I add new data I have never seen this have in past sent hundreds of millions
"new" transactions.  It seems to be when I modify.  my process is as follows

key slice to get columns to modify in batches of 100, in separate threads modify those columns.
 I advance the slice with the start key each with last key in previous batch.  Mutations done
are update a column value in one column family(token), delete column and add new column in
another (pan).

Runs well until after about 5 million rows then it seems to run out of memory.  Note that
these column families are quite small.

WARN [ScheduledTasks:1] 2012-06-03 17:49:01,558 GCInspector.java (line 145) Heap is 0.7967470834946492
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 INFO [ScheduledTasks:1] 2012-06-03 17:49:01,559 StorageService.java (line 2772) Unable to
reduce heap usage since there are no dirty column families
 INFO [GossipStage:1] 2012-06-03 17:49:01,999 Gossiper.java (line 797) InetAddress /10.230.34.170
is now UP
 INFO [ScheduledTasks:1] 2012-06-03 17:49:10,048 GCInspector.java (line 122) GC for ParNew:
206 ms for 1 collections, 7345969520 used; max is 8506048512
 INFO [ScheduledTasks:1] 2012-06-03 17:49:53,187 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
12770 ms for 1 collections, 5714800208 used; max is 8506048512
 
----------------
Keyspace: keyspace
        Read Count: 50042632
        Read Latency: 0.23157864418482224 ms.
        Write Count: 44948323
        Write Latency: 0.019460829472992797 ms.
        Pending Tasks: 0
                Column Family: pan
                SSTable count: 5
                Space used (live): 1977467326
                Space used (total): 1977467326
                Number of Keys (estimate): 16334848
                Memtable Columns Count: 0
                Memtable Data Size: 0
                Memtable Switch Count: 74
                Read Count: 14985122
                Read Latency: 0.408 ms.
                Write Count: 19972441
                Write Latency: 0.022 ms.
                Pending Tasks: 0
                Bloom Filter False Postives: 829
                Bloom Filter False Ratio: 0.00073
                Bloom Filter Space Used: 37048400
                Compacted row minimum size: 125
                Compacted row maximum size: 149
                Compacted row mean size: 149
 
                Column Family: token
                SSTable count: 4
                Space used (live): 1250973873
                Space used (total): 1250973873
                Number of Keys (estimate): 14217216
                Memtable Columns Count: 0
                Memtable Data Size: 0
                Memtable Switch Count: 49
                Read Count: 30059563
                Read Latency: 0.167 ms.
                Write Count: 14985488
                Write Latency: 0.014 ms.
                Pending Tasks: 0
                Bloom Filter False Postives: 13642
                Bloom Filter False Ratio: 0.00322
                Bloom Filter Space Used: 28002984
                Compacted row minimum size: 150
                Compacted row maximum size: 258
                Compacted row mean size: 224
 
                Column Family: counters
                SSTable count: 2
                Space used (live): 561549994
                Space used (total): 561549994
                Number of Keys (estimate): 9985024
                Memtable Columns Count: 0
                Memtable Data Size: 0
                Memtable Switch Count: 38
                Read Count: 4997947
                Read Latency: 0.092 ms.
                Write Count: 9990394
                Write Latency: 0.023 ms.
                Pending Tasks: 0
                Bloom Filter False Postives: 191
                Bloom Filter False Ratio: 0.37525
                Bloom Filter Space Used: 18741152
                Compacted row minimum size: 125
                Compacted row maximum size: 179
                Compacted row mean size: 150
 
----------------

Mime
View raw message