incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chip Salzenberg <c...@pobox.com>
Subject Re: 0.7.0beta2 with 128G memory = bizarre tiny files. help?
Date Wed, 27 Oct 2010 21:02:34 GMT
That's what you're looking at.  Sorry for the confusion.  I left all the
files in place for the upgrade.  What I meant was that back when b2 was
running, "import" was how I had created Standard1 in the first place.

On 10/27/2010 1:54 PM, Jonathan Ellis wrote:
> Sorry, I should have been more clear: rc1 is upgrade-in-place
> compatible with b2, so I would have liked to see what the settings
> were before running it through an export/import cycle.
>
> These settings look fine and I doubt you will see bizarre tiny files.
>
> On Wed, Oct 27, 2010 at 2:52 PM, Chip Salzenberg <chip@pobox.com> wrote:
>> OK, after upgrading to "apache-cassandra-2010-10-27_12-30-38", here's
>> what "show keyspaces" says about the relevant keyspace:
>>
>>    ColumnFamily: Standard1
>>      Columns sorted by: org.apache.cassandra.db.marshal.BytesType
>>      Row cache size / save period: 0.0/0
>>      Key cache size / save period: 200000.0/3600
>>      Memtable thresholds: 9.590625/2046/60
>>      GC grace seconds: 864000
>>      Compaction min/max thresholds: 4/32
>>
>> I created it with "schematool import" off of the stock config in b2.
>>
>>
>> On 10/27/2010 11:36 AM, Jonathan Ellis wrote:
>>> Not in b2.
>>> On Oct 27, 2010 11:08 AM, "Chip Salzenberg" <chip@pobox.com> wrote:
>>>> Are these values available in JMX? If so, which ones are they? I'd
>>>> prefer not to possibly destroy the evidence by upgrading.
>>>>
>>>> On 10/27/2010 6:50 AM, Jonathan Ellis wrote:
>>>>> Connect with the cli and run "show keyspaces." It sounds like
>>>>> Cassandra somehow picked ludicrously low memtable thresholds. There
>>>>> will be a line like
>>>>>
>>>>> Memtable thresholds: 0.2953125/63/60
>>>>>
>>>>> This may not be in the b2 cli, in which case you will need to upgrade
>>>>> to the nightly from
>>>>>
>>> http://hudson.zones.apache.org/hudson/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/
>>>>> (essentially the same as rc1 being voted on).
>>>>>
>>>>> On Wed, Oct 27, 2010 at 4:17 AM, Chip Salzenberg <chip@pobox.com>
wrote:
>>>>>> I recently tested 0.7.0beta2 with good results on a reasonably powerful
>>>>>> machine: 8 Xeon cores (16 if you count hyperthreads), 64G memory,
and
>>>>>> some nice HP RAID. So far so good. But when I took the same config
and
>>>>>> moved it to a basically identical box with 128G of memory, cassandra
>>>>>> started responding to writes by creating a plethora of teeny tiny
>>>>>> sstables -- something it had not done before. For example:
>>>>>>
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,881 Memtable.java
(line
>>>>>> 150) Writing Memtable-Standard1@950886315(651 bytes, 18 operations)
>>>>>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,882 ColumnFamilyStore.java
>>>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>>>
>>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>>>> position=27598)
>>>>>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,883 ColumnFamilyStore.java
>>>>>> (line 771) Enqueuing flush of Memtable-Standard1@1383310803(384 bytes,
>>>>>> 10 operations)
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,902 Memtable.java
(line
>>>>>> 157) Completed flushing
>>>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,903 Memtable.java
(line
>>>>>> 150) Writing Memtable-Standard1@996627145(360 bytes, 10 operations)
>>>>>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>>>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>>>
>>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>>>> position=31446)
>>>>>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>>>>>> (line 771) Enqueuing flush of Memtable-Standard1@1909350010(957 bytes,
>>>>>> 26 operations)
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,932 Memtable.java
(line
>>>>>> 157) Completed flushing
>>>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,933 Memtable.java
(line
>>>>>> 150) Writing Memtable-Standard1@1383310803(384 bytes, 10 operations)
>>>>>> INFO [CompactionExecutor:1] 2010-10-27 01:53:27,934
>>>>>> CompactionManager.java (line 233) Compacting
>>>>>>
>>> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-5-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db')]
>>>>>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>>>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>>>
>>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>>>> position=33074)
>>>>>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>>>>>> (line 771) Enqueuing flush of Memtable-Standard1@595066677(396 bytes,
11
>>>>>> operations)
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:28,037 Memtable.java
(line
>>>>>> 157) Completed flushing
>>>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-9-Data.db
>>>>>>
>>>>>> I tried lowering the available memory reported by bin/cassandra:
>>>>>>
>>>>>> system_memory_in_mb=`free -m | awk '/Mem:/ {print $2}'`
>>>>>> [ "$system_memory_in_mb" -gt 65536 ] &&
>>>>>> system_memory_in_mb=65536 #<<<< new
>>>>>>
>>>>>> Didn't help. I also tried maually setting the max memtable sizes:
>>>>>>
>>>>>> binary_memtable_throughput_in_mb: 512
>>>>>> memtable_throughput_in_mb: 4096
>>>>>>
>>>>>> Didn't help.
>>>>>>
>>>>>> Help?
>>>>>>
>
>

Mime
View raw message