cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: memory issue on 1.1.0
Date Mon, 04 Jun 2012 21:16:51 GMT
Had a look at the log, this message 

> INFO [ScheduledTasks:1] 2012-06-03 17:49:01,559 StorageService.java (line 2772) Unable
to reduce heap usage since there are no dirty column families
appears correct, it happens after some flush activity and there are not CF's with memtable
data. But the heap is still full. 

Overall the server is overloaded, but it seems like it should be handling it better. 

What JVM settings do you have? What is the machine spec ? 
What settings do you have for key and row cache ? 
Do the CF's have secondary indexes ?
How many clients / requests per second ? 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 4/06/2012, at 11:12 AM, Poziombka, Wade L wrote:

> Running a very write intensive (new column, delete old column etc.) process and failing
on memory.  Log file attached.
> 
> Curiously when I add new data I have never seen this have in past sent hundreds of millions
"new" transactions.  It seems to be when I modify.  my process is as follows
> 
> key slice to get columns to modify in batches of 100, in separate threads modify those
columns.  I advance the slice with the start key each with last key in previous batch.  Mutations
done are update a column value in one column family(token), delete column and add new column
in another (pan).
> 
> Runs well until after about 5 million rows then it seems to run out of memory.  Note
that these column families are quite small.
> 
> WARN [ScheduledTasks:1] 2012-06-03 17:49:01,558 GCInspector.java (line 145) Heap is 0.7967470834946492
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
> INFO [ScheduledTasks:1] 2012-06-03 17:49:01,559 StorageService.java (line 2772) Unable
to reduce heap usage since there are no dirty column families
> INFO [GossipStage:1] 2012-06-03 17:49:01,999 Gossiper.java (line 797) InetAddress /10.230.34.170
is now UP
> INFO [ScheduledTasks:1] 2012-06-03 17:49:10,048 GCInspector.java (line 122) GC for ParNew:
206 ms for 1 collections, 7345969520 used; max is 8506048512
> INFO [ScheduledTasks:1] 2012-06-03 17:49:53,187 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
12770 ms for 1 collections, 5714800208 used; max is 8506048512
> 
> ----------------
> Keyspace: keyspace
>        Read Count: 50042632
>        Read Latency: 0.23157864418482224 ms.
>        Write Count: 44948323
>        Write Latency: 0.019460829472992797 ms.
>        Pending Tasks: 0
>                Column Family: pan
>                SSTable count: 5
>                Space used (live): 1977467326
>                Space used (total): 1977467326
>                Number of Keys (estimate): 16334848
>                Memtable Columns Count: 0
>                Memtable Data Size: 0
>                Memtable Switch Count: 74
>                Read Count: 14985122
>                Read Latency: 0.408 ms.
>                Write Count: 19972441
>                Write Latency: 0.022 ms.
>                Pending Tasks: 0
>                Bloom Filter False Postives: 829
>                Bloom Filter False Ratio: 0.00073
>                Bloom Filter Space Used: 37048400
>                Compacted row minimum size: 125
>                Compacted row maximum size: 149
>                Compacted row mean size: 149
> 
>                Column Family: token
>                SSTable count: 4
>                Space used (live): 1250973873
>                Space used (total): 1250973873
>                Number of Keys (estimate): 14217216
>                Memtable Columns Count: 0
>                Memtable Data Size: 0
>                Memtable Switch Count: 49
>                Read Count: 30059563
>                Read Latency: 0.167 ms.
>                Write Count: 14985488
>                Write Latency: 0.014 ms.
>                Pending Tasks: 0
>                Bloom Filter False Postives: 13642
>                Bloom Filter False Ratio: 0.00322
>                Bloom Filter Space Used: 28002984
>                Compacted row minimum size: 150
>                Compacted row maximum size: 258
>                Compacted row mean size: 224
> 
>                Column Family: counters
>                SSTable count: 2
>                Space used (live): 561549994
>                Space used (total): 561549994
>                Number of Keys (estimate): 9985024
>                Memtable Columns Count: 0
>                Memtable Data Size: 0
>                Memtable Switch Count: 38
>                Read Count: 4997947
>                Read Latency: 0.092 ms.
>                Write Count: 9990394
>                Write Latency: 0.023 ms.
>                Pending Tasks: 0
>                Bloom Filter False Postives: 191
>                Bloom Filter False Ratio: 0.37525
>                Bloom Filter Space Used: 18741152
>                Compacted row minimum size: 125
>                Compacted row maximum size: 179
>                Compacted row mean size: 150
> 
> ----------------
> <cassandra.zip>


Mime
View raw message