incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramesh Natarajan <rames...@gmail.com>
Subject Re: Key count mismatch in cluster for a column family
Date Wed, 26 Oct 2011 12:44:21 GMT
Looks like compaction for this column family stopped after some time.
The last message for this column family in the system.log is


INFO [MigrationStage:1] 2011-10-25 16:57:00,385 Migration.java (line
119) Applying migration 43f106c0-ff54-11e0-0000-68877f281daf Update
column family to
org.apache.cassandra.config.CFMetaData@86c50e4[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=1.3E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@7f32ad0d,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
lim=3 cap=3],column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
 INFO [MigrationStage:1] 2011-10-25 16:57:00,389
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Migrations@1386279942(8806/11007 serialized/live bytes, 1
ops)
 INFO [FlushWriter:307] 2011-10-25 16:57:00,389 Memtable.java (line
237) Writing Memtable-Migrations@1386279942(8806/11007 serialized/live
bytes, 1 ops)
 INFO [MigrationStage:1] 2011-10-25 16:57:00,389
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Schema@1156898891(4336/5420 serialized/live bytes, 3 ops)
 INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/system/Migrations-h-48-Data.db (8870 bytes)
 INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
237) Writing Memtable-Schema@1156898891(4336/5420 serialized/live
bytes, 3 ops)
 INFO [FlushWriter:307] 2011-10-25 16:57:00,413 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/system/Schema-h-48-Data.db (4486 bytes)
 INFO [CompactionExecutor:23] 2011-10-25 16:57:00,929
CompactionTask.java (line 119) Compacting
[SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9016-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9046-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9042-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9039-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9043-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9045-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9044-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9015-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9040-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9054-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9047-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9041-Data.db')]
 INFO [MigrationStage:1] 2011-10-25 16:57:06,693 Migration.java (line
119) Applying migration 47b1b840-ff54-11e0-0000-68877f281daf Update
column family to
org.apache.cassandra.config.CFMetaData@23de1953[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=1.5E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@4a50aa8a,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
lim=3 cap=3],column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
 INFO [MigrationStage:1] 2011-10-25 16:57:06,694
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Migrations@1978429475(8806/11007 serialized/live bytes, 1
ops)
 INFO [FlushWriter:307] 2011-10-25 16:57:06,694 Memtable.java (line
237) Writing Memtable-Migrations@1978429475(8806/11007 serialized/live
bytes, 1 ops)



The schema for this CF is

create column family uid
  with column_type = 'Standard'
  and comparator = 'BytesType'
  and default_validation_class = 'BytesType'
  and key_validation_class = 'BytesType'
  and rows_cached = 0.0
  and row_cache_save_period = 0
  and row_cache_keys_to_save = 2147483647
  and keys_cached = 1.5E7
  and key_cache_save_period = 14400
  and read_repair_chance = 1.0
  and gc_grace = 3600
  and min_compaction_threshold = 4
  and max_compaction_threshold = 32
  and replicate_on_write = true
  and row_cache_provider = 'SerializingCacheProvider'
  and compaction_strategy =
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
  and compaction_strategy_options = {'sstable_size_in_mb' : '10'};



Any idea why compaction stopped automatically for  this CF?

thanks
Ramesh





On Wed, Oct 26, 2011 at 5:23 AM, Sylvain Lebresne <sylvain@datastax.com> wrote:
> The estimate for the number of keys is computed by summing the key
> estimate for each sstable of the CF. For each sstable, the estimate
> should be fairly good. However, that's when we sum all the sstable estimates
> that we can loose potentially a lot of precision if there is a lot of rows that
> have parts in different sstables. But that in turn would suggest a problem
> with compaction lacking badly behind, especially with leveled compaction.
>
> --
> Sylvain
>
> On Wed, Oct 26, 2011 at 3:58 AM, Terry Cumaranatunge <cumarana@gmail.com> wrote:
>> I have a cluster of 8 nodes all running 1.0. The stats shown on the 1st node
>> on one of the CFs for the number of keys is much larger than expected. The
>> first node shows the key count estimate to be 9.2M whereas the rest report
>> ~650K on each node. The 650K is in the correct neighborhood of the number of
>> keys that have been inserted. The counts are comparable for all other CFs
>> across the cluster. I'm using Level compaction, but no compression.
>>
>> The 'nodetool ring' shows that the load is equal across all nodes. What
>> could cause this large disparity in the number of keys? Is this just a stats
>> issue or does this suggest a functional problem?
>>
>> 1st node:
>>         Column Family: uid
>>                 SSTable count: 395
>>                 Space used (live): 13752624444
>>                 Space used (total): 5482088532
>>                 Number of Keys (estimate): 9215104
>>                 Memtable Columns Count: 514952
>>                 Memtable Data Size: 295213448
>>                 Memtable Switch Count: 290
>>                 Read Count: 193102511
>>                 Read Latency: 0.146 ms.
>>                 Write Count: 176934874
>>                 Write Latency: 0.018 ms.
>>                 Pending Tasks: 0
>>                 Key cache capacity: 8302131
>>                 Key cache size: 8302131
>>                 Key cache hit rate: 0.8644664668071792
>>                 Row cache: disabled
>>                 Compacted row minimum size: 87
>>                 Compacted row maximum size: 7007506
>>                 Compacted row mean size: 8944
>> 2nd node:
>>         Column Family: uid
>>                 SSTable count: 402
>>                 Space used (live): 13723958304
>>                 Space used (total): 4044833220
>>                 Number of Keys (estimate): 652928
>>                 Memtable Columns Count: 170290
>>                 Memtable Data Size: 102378904
>>                 Memtable Switch Count: 272
>>                 Read Count: 192463595
>>                 Read Latency: 0.289 ms.
>>                 Write Count: 176527238
>>                 Write Latency: 0.014 ms.
>>                 Pending Tasks: 0
>>                 Key cache capacity: 8783058
>>                 Key cache size: 8783058
>>                 Key cache hit rate: 0.7865727464740025
>>                 Row cache: disabled
>>                 Compacted row minimum size: 87
>>                 Compacted row maximum size: 7007506
>>                 Compacted row mean size: 12151
>> 3rd node:
>>               Column Family: uid
>>                 SSTable count: 401
>>                 Space used (live): 13204714872
>>                 Space used (total): 4030024144
>>                 Number of Keys (estimate): 675968
>>                 Memtable Columns Count: 42881
>>                 Memtable Data Size: 30992298
>>                 Memtable Switch Count: 304
>>                 Read Count: 190769879
>>                 Read Latency: 0.224 ms.
>>                 Write Count: 175381826
>>                 Write Latency: 0.014 ms.
>>                 Pending Tasks: 0
>>                 Key cache capacity: 8920108
>>                 Key cache size: 8920108
>>                 Key cache hit rate: 0.8053563128870577
>>                 Row cache: disabled
>>                 Compacted row minimum size: 87
>>                 Compacted row maximum size: 4866323
>>                 Compacted row mean size: 12074
>>
>

Mime
View raw message