cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Poziombka, Wade L" <wade.l.poziom...@intel.com>
Subject RE: memory issue on 1.1.0
Date Tue, 05 Jun 2012 18:56:45 GMT
Alas, upgrading to 1.1.1 did not solve my issue.

-----Original Message-----
From: Brandon Williams [mailto:driftx@gmail.com] 
Sent: Monday, June 04, 2012 11:24 PM
To: user@cassandra.apache.org
Subject: Re: memory issue on 1.1.0

Perhaps the deletes: https://issues.apache.org/jira/browse/CASSANDRA-3741

-Brandon

On Sun, Jun 3, 2012 at 6:12 PM, Poziombka, Wade L <wade.l.poziombka@intel.com> wrote:
> Running a very write intensive (new column, delete old column etc.) process and failing
on memory.  Log file attached.
>
> Curiously when I add new data I have never seen this have in past sent 
> hundreds of millions "new" transactions.  It seems to be when I 
> modify.  my process is as follows
>
> key slice to get columns to modify in batches of 100, in separate threads modify those
columns.  I advance the slice with the start key each with last key in previous batch.  Mutations
done are update a column value in one column family(token), delete column and add new column
in another (pan).
>
> Runs well until after about 5 million rows then it seems to run out of memory.  Note
that these column families are quite small.
>
> WARN [ScheduledTasks:1] 2012-06-03 17:49:01,558 GCInspector.java (line 
> 145) Heap is 0.7967470834946492 full.  You may need to reduce memtable 
> and/or cache sizes.  Cassandra will now flush up to the two largest 
> memtables to free up memory.  Adjust flush_largest_memtables_at 
> threshold in cassandra.yaml if you don't want Cassandra to do this 
> automatically
>  INFO [ScheduledTasks:1] 2012-06-03 17:49:01,559 StorageService.java 
> (line 2772) Unable to reduce heap usage since there are no dirty 
> column families
>  INFO [GossipStage:1] 2012-06-03 17:49:01,999 Gossiper.java (line 797) 
> InetAddress /10.230.34.170 is now UP
>  INFO [ScheduledTasks:1] 2012-06-03 17:49:10,048 GCInspector.java 
> (line 122) GC for ParNew: 206 ms for 1 collections, 7345969520 used; 
> max is 8506048512
>  INFO [ScheduledTasks:1] 2012-06-03 17:49:53,187 GCInspector.java 
> (line 122) GC for ConcurrentMarkSweep: 12770 ms for 1 collections, 
> 5714800208 used; max is 8506048512
>
> ----------------
> Keyspace: keyspace
>        Read Count: 50042632
>        Read Latency: 0.23157864418482224 ms.
>        Write Count: 44948323
>        Write Latency: 0.019460829472992797 ms.
>        Pending Tasks: 0
>                Column Family: pan
>                SSTable count: 5
>                Space used (live): 1977467326
>                Space used (total): 1977467326
>                Number of Keys (estimate): 16334848
>                Memtable Columns Count: 0
>                Memtable Data Size: 0
>                Memtable Switch Count: 74
>                Read Count: 14985122
>                Read Latency: 0.408 ms.
>                Write Count: 19972441
>                Write Latency: 0.022 ms.
>                Pending Tasks: 0
>                Bloom Filter False Postives: 829
>                Bloom Filter False Ratio: 0.00073
>                Bloom Filter Space Used: 37048400
>                Compacted row minimum size: 125
>                Compacted row maximum size: 149
>                Compacted row mean size: 149
>
>                Column Family: token
>                SSTable count: 4
>                Space used (live): 1250973873
>                Space used (total): 1250973873
>                Number of Keys (estimate): 14217216
>                Memtable Columns Count: 0
>                Memtable Data Size: 0
>                Memtable Switch Count: 49
>                Read Count: 30059563
>                Read Latency: 0.167 ms.
>                Write Count: 14985488
>                Write Latency: 0.014 ms.
>                Pending Tasks: 0
>                Bloom Filter False Postives: 13642
>                Bloom Filter False Ratio: 0.00322
>                Bloom Filter Space Used: 28002984
>                Compacted row minimum size: 150
>                Compacted row maximum size: 258
>                Compacted row mean size: 224
>
>                Column Family: counters
>                SSTable count: 2
>                Space used (live): 561549994
>                Space used (total): 561549994
>                Number of Keys (estimate): 9985024
>                Memtable Columns Count: 0
>                Memtable Data Size: 0
>                Memtable Switch Count: 38
>                Read Count: 4997947
>                Read Latency: 0.092 ms.
>                Write Count: 9990394
>                Write Latency: 0.023 ms.
>                Pending Tasks: 0
>                Bloom Filter False Postives: 191
>                Bloom Filter False Ratio: 0.37525
>                Bloom Filter Space Used: 18741152
>                Compacted row minimum size: 125
>                Compacted row maximum size: 179
>                Compacted row mean size: 150
>
> ----------------

Mime
View raw message