incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sylvain Lebresne <sylv...@datastax.com>
Subject Re: Key count mismatch in cluster for a column family
Date Wed, 26 Oct 2011 14:51:53 GMT
On Wed, Oct 26, 2011 at 3:50 PM, Ramesh Natarajan <ramesh25@gmail.com> wrote:
> I did some more analysis and I think the compaction for this CF stopped after we did
a update column family to increase the key cache. Other CF compactions were going on without
any issues.
> I did another update column family to the same CF with same values as before and the
compaction started again.
> It it interesting to note that this happened on only one of the node in a 8 node cluster.
Also it is the same node where the command was issued using the cli.
>
> Is it normal to issue update column family to increase the key cache when cassandra is
actively serving traffic in a production environment?
>
> Are there any known issues/interactions with compaction and updating column family in
1.0?

If the update happens to change and set to 0 the
min_compaction_threshold or max_compaction_threshold, then that would
disable compaction. Now of course it should not do that unless you've
asked it to, and if it did, it should disable compaction cluster-wide,
otherwise that would be a bug. But that's the only way I see for an
update column family to stop compaction.

--
Sylvain

>
> Thanks
> Ramesh
>
> Ramesh Natarajan <ramesh25@gmail.com> wrote:
>
>>Looks like compaction for this column family stopped after some time.
>>The last message for this column family in the system.log is
>>
>>
>>INFO [MigrationStage:1] 2011-10-25 16:57:00,385 Migration.java (line
>>119) Applying migration 43f106c0-ff54-11e0-0000-68877f281daf Update
>>column family to
>>org.apache.cassandra.config.CFMetaData@86c50e4[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=1.3E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@7f32ad0d,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
>>lim=3 cap=3],column_metadata={},compactionStrategyClass=class
>>org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
>> INFO [MigrationStage:1] 2011-10-25 16:57:00,389
>>ColumnFamilyStore.java (line 664) Enqueuing flush of
>>Memtable-Migrations@1386279942(8806/11007 serialized/live bytes, 1
>>ops)
>> INFO [FlushWriter:307] 2011-10-25 16:57:00,389 Memtable.java (line
>>237) Writing Memtable-Migrations@1386279942(8806/11007 serialized/live
>>bytes, 1 ops)
>> INFO [MigrationStage:1] 2011-10-25 16:57:00,389
>>ColumnFamilyStore.java (line 664) Enqueuing flush of
>>Memtable-Schema@1156898891(4336/5420 serialized/live bytes, 3 ops)
>> INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
>>273) Completed flushing
>>/var/lib/cassandra/data/system/Migrations-h-48-Data.db (8870 bytes)
>> INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
>>237) Writing Memtable-Schema@1156898891(4336/5420 serialized/live
>>bytes, 3 ops)
>> INFO [FlushWriter:307] 2011-10-25 16:57:00,413 Memtable.java (line
>>273) Completed flushing
>>/var/lib/cassandra/data/system/Schema-h-48-Data.db (4486 bytes)
>> INFO [CompactionExecutor:23] 2011-10-25 16:57:00,929
>>CompactionTask.java (line 119) Compacting
>>[SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9016-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9046-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9042-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9039-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9043-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9045-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9044-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9015-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9040-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9054-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9047-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9041-Data.db')]
>> INFO [MigrationStage:1] 2011-10-25 16:57:06,693 Migration.java (line
>>119) Applying migration 47b1b840-ff54-11e0-0000-68877f281daf Update
>>column family to
>>org.apache.cassandra.config.CFMetaData@23de1953[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=1.5E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@4a50aa8a,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
>>lim=3 cap=3],column_metadata={},compactionStrategyClass=class
>>org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
>> INFO [MigrationStage:1] 2011-10-25 16:57:06,694
>>ColumnFamilyStore.java (line 664) Enqueuing flush of
>>Memtable-Migrations@1978429475(8806/11007 serialized/live bytes, 1
>>ops)
>> INFO [FlushWriter:307] 2011-10-25 16:57:06,694 Memtable.java (line
>>237) Writing Memtable-Migrations@1978429475(8806/11007 serialized/live
>>bytes, 1 ops)
>>
>>
>>
>>The schema for this CF is
>>
>>create column family uid
>>  with column_type = 'Standard'
>>  and comparator = 'BytesType'
>>  and default_validation_class = 'BytesType'
>>  and key_validation_class = 'BytesType'
>>  and rows_cached = 0.0
>>  and row_cache_save_period = 0
>>  and row_cache_keys_to_save = 2147483647
>>  and keys_cached = 1.5E7
>>  and key_cache_save_period = 14400
>>  and read_repair_chance = 1.0
>>  and gc_grace = 3600
>>  and min_compaction_threshold = 4
>>  and max_compaction_threshold = 32
>>  and replicate_on_write = true
>>  and row_cache_provider = 'SerializingCacheProvider'
>>  and compaction_strategy =
>>'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
>>  and compaction_strategy_options = {'sstable_size_in_mb' : '10'};
>>
>>
>>
>>Any idea why compaction stopped automatically for  this CF?
>>
>>thanks
>>Ramesh
>>
>>
>>
>>
>>
>>On Wed, Oct 26, 2011 at 5:23 AM, Sylvain Lebresne <sylvain@datastax.com> wrote:
>>> The estimate for the number of keys is computed by summing the key
>>> estimate for each sstable of the CF. For each sstable, the estimate
>>> should be fairly good. However, that's when we sum all the sstable estimates
>>> that we can loose potentially a lot of precision if there is a lot of rows that
>>> have parts in different sstables. But that in turn would suggest a problem
>>> with compaction lacking badly behind, especially with leveled compaction.
>>>
>>> --
>>> Sylvain
>>>
>>> On Wed, Oct 26, 2011 at 3:58 AM, Terry Cumaranatunge <cumarana@gmail.com>
wrote:
>>>> I have a cluster of 8 nodes all running 1.0. The stats shown on the 1st node
>>>> on one of the CFs for the number of keys is much larger than expected. The
>>>> first node shows the key count estimate to be 9.2M whereas the rest report
>>>> ~650K on each node. The 650K is in the correct neighborhood of the number
of
>>>> keys that have been inserted. The counts are comparable for all other CFs
>>>> across the cluster. I'm using Level compaction, but no compression.
>>>>
>>>> The 'nodetool ring' shows that the load is equal across all nodes. What
>>>> could cause this large disparity in the number of keys? Is this just a stats
>>>> issue or does this suggest a functional problem?
>>>>
>>>> 1st node:
>>>>         Column Family: uid
>>>>                 SSTable count: 395
>>>>                 Space used (live): 13752624444
>>>>                 Space used (total): 5482088532
>>>>                 Number of Keys (estimate): 9215104
>>>>                 Memtable Columns Count: 514952
>>>>                 Memtable Data Size: 295213448
>>>>                 Memtable Switch Count: 290
>>>>                 Read Count: 193102511
>>>>                 Read Latency: 0.146 ms.
>>>>                 Write Count: 176934874
>>>>                 Write Latency: 0.018 ms.
>>>>                 Pending Tasks: 0
>>>>                 Key cache capacity: 8302131
>>>>                 Key cache size: 8302131
>>>>                 Key cache hit rate: 0.8644664668071792
>>>>                 Row cache: disabled
>>>>                 Compacted row minimum size: 87
>>>>                 Compacted row maximum size: 7007506
>>>>                 Compacted row mean size: 8944
>>>> 2nd node:
>>>>         Column Family: uid
>>>>                 SSTable count: 402
>>>>                 Space used (live): 13723958304
>>>>                 Space used (total): 4044833220
>>>>                 Number of Keys (estimate): 652928
>>>>                 Memtable Columns Count: 170290
>>>>                 Memtable Data Size: 102378904
>>>>                 Memtable Switch Count: 272
>>>>                 Read Count: 192463595
>>>>                 Read Latency: 0.289 ms.
>>>>                 Write Count: 176527238
>>>>                 Write Latency: 0.014 ms.
>>>>                 Pending Tasks: 0
>>>>                 Key cache capacity: 8783058
>>>>                 Key cache size: 8783058
>>>>                 Key cache hit rate: 0.7865727464740025
>>>>                 Row cache: disabled
>>>>                 Compacted row minimum size: 87
>>>>                 Compacted row maximum size: 7007506
>>>>                 Compacted row mean size: 12151
>>>> 3rd node:
>>>>               Column Family: uid
>>>>                 SSTable count: 401
>>>>                 Space used (live): 13204714872
>>>>                 Space used (total): 4030024144
>>>>                 Number of Keys (estimate): 675968
>>>>                 Memtable Columns Count: 42881
>>>>                 Memtable Data Size: 30992298
>>>>                 Memtable Switch Count: 304
>>>>                 Read Count: 190769879
>>>>                 Read Latency: 0.224 ms.
>>>>                 Write Count: 175381826
>>>>                 Write Latency: 0.014 ms.
>>>>                 Pending Tasks: 0
>>>>                 Key cache capacity: 8920108
>>>>                 Key cache size: 8920108
>>>>                 Key cache hit rate: 0.8053563128870577
>>>>                 Row cache: disabled
>>>>                 Compacted row minimum size: 87
>>>>                 Compacted row maximum size: 4866323
>>>>                 Compacted row mean size: 12074
>>>>
>>>
>

Mime
View raw message