incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chip Salzenberg <c...@pobox.com>
Subject Re: 0.7.0beta2 with 128G memory = bizarre tiny files. help?
Date Wed, 27 Oct 2010 09:21:20 GMT
I should add that the disk free space also changed, so the data
directory had a lot more space available than before:

    /dev/cciss/c0d1       2.8T  202M  2.6T   1% /var/lib/cassandra/data


On 10/27/2010 2:17 AM, Chip Salzenberg wrote:
> I recently tested 0.7.0beta2 with good results on a reasonably powerful
> machine: 8 Xeon cores (16 if you count hyperthreads), 64G memory, and
> some nice HP RAID.  So far so good.  But when I took the same config and
> moved it to a basically identical box with 128G of memory, cassandra
> started responding to writes by creating a plethora of teeny tiny
> sstables -- something it had not done before.  For example:
>
>  INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,881 Memtable.java (line
> 150) Writing Memtable-Standard1@950886315(651 bytes, 18 operations)
>  INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,882 ColumnFamilyStore.java
> (line 459) switching in a fresh Memtable for Standard1 at
> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
> position=27598)
>  INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,883 ColumnFamilyStore.java
> (line 771) Enqueuing flush of Memtable-Standard1@1383310803(384 bytes,
> 10 operations)
>  INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,902 Memtable.java (line
> 157) Completed flushing
> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db
>  INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,903 Memtable.java (line
> 150) Writing Memtable-Standard1@996627145(360 bytes, 10 operations)
>  INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
> (line 459) switching in a fresh Memtable for Standard1 at
> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
> position=31446)
>  INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
> (line 771) Enqueuing flush of Memtable-Standard1@1909350010(957 bytes,
> 26 operations)
>  INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,932 Memtable.java (line
> 157) Completed flushing
> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db
>  INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,933 Memtable.java (line
> 150) Writing Memtable-Standard1@1383310803(384 bytes, 10 operations)
>  INFO [CompactionExecutor:1] 2010-10-27 01:53:27,934
> CompactionManager.java (line 233) Compacting
> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-5-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db')]
>  INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
> (line 459) switching in a fresh Memtable for Standard1 at
> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
> position=33074)
>  INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
> (line 771) Enqueuing flush of Memtable-Standard1@595066677(396 bytes, 11
> operations)
>  INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:28,037 Memtable.java (line
> 157) Completed flushing
> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-9-Data.db
>
> I tried lowering the available memory reported by bin/cassandra:
>
>             system_memory_in_mb=`free -m | awk '/Mem:/ {print $2}'`
>             [ "$system_memory_in_mb" -gt 65536 ] &&
> system_memory_in_mb=65536      #<<<< new
>
> Didn't help.  I also tried maually setting the max memtable sizes:
>
>     binary_memtable_throughput_in_mb: 512
>     memtable_throughput_in_mb: 4096
>
> Didn't help.
>
> Help?
>

Mime
View raw message