cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paulo Motta <pauloricard...@gmail.com>
Subject Re: 3k sstables during a repair incremental !!
Date Wed, 10 Feb 2016 17:40:58 GMT
Are you using vnodes by any chance? If so, how many? How many nodes and
what's the replication factor? How was data inserted (at what consistency
level)?

Streaming might create a large number of sstables with vnodes (see
CASSANDRA-10495), so in case data is inconsistent between nodes (detected
during the validation phase) a large number of sstables might be created
during the streaming phase of the repair. This might happen with or without
incremental repairs.

There is also a migration step from non-incremental to incremental repair
that must be followed for LCS tables to avoid recompacting all sstables
again. You may find more information in the migration procedure on
http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesMigration.html

2016-02-10 14:16 GMT-03:00 Jean Carlo <jean.jeancarl48@gmail.com>:

> Hello Kai
>
> This is for *cf3*
>
> nodetool cfstats pns_nonreg_bench.cf3 -H
> Keyspace: pns_nonreg_bench
>     Read Count: 23594
>     Read Latency: 1.2980987539204882 ms.
>     Write Count: 148161
>     Write Latency: 0.04608940274431193 ms.
>     Pending Flushes: 0
>         Table: cf3
>         SSTable count: 11489
>         SSTables in each level: [11485/4, 4, 0, 0, 0, 0, 0, 0, 0]
>         Space used (live): 954.89 MB
>         Space used (total): 954.89 MB
>         Space used by snapshots (total): 0 bytes
>         Off heap memory used (total): 3.74 MB
>         SSTable Compression Ratio: 0.4245371280396339
>         Number of keys (estimate): 1051613
>         Memtable cell count: 0
>         Memtable data size: 0 bytes
>         Memtable off heap memory used: 0 bytes
>         Memtable switch count: 889
>         Local read count: 23594
>         Local read latency: 1.299 ms
>         Local write count: 148161
>         Local write latency: 0.047 ms
>         Pending flushes: 0
>         Bloom filter false positives: 16609
>         Bloom filter false ratio: 0.00000
>         Bloom filter space used: 1.64 MB
>         Bloom filter off heap memory used: 1.55 MB
>         Index summary off heap memory used: 1.88 MB
>         Compression metadata off heap memory used: 318.56 KB
>         Compacted partition minimum bytes: 643 bytes
>         Compacted partition maximum bytes: 924 bytes
>         Compacted partition mean bytes: 914 bytes
>         Average live cells per slice (last five minutes): 1.0
>         Maximum live cells per slice (last five minutes): 1.0
>         Average tombstones per slice (last five minutes): 0.0
>         Maximum tombstones per slice (last five minutes): 0.0
>
> ----------------
>
> This is for *cf1*
>
> nodetool cfstats pns_nonreg_bench.cf1 -H
> Keyspace: pns_nonreg_bench
>     Read Count: 24207
>     Read Latency: 0.9072712851654481 ms.
>     Write Count: 148380
>     Write Latency: 0.04561746866154468 ms.
>     Pending Flushes: 0
>         Table: cf1
>         SSTable count: 1942
>
> It seems that nodetool does not retrieve the rest of the information, I
> think it is due to compactions
>
> nodetool tpstats
> CompactionExecutor                2        83           1424
> 0                 0
>
>
>
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Wed, Feb 10, 2016 at 6:00 PM, Kai Wang <depend@gmail.com> wrote:
>
>> Jean,
>>
>> What does your cfstats look like? Especially "SSTables in each level"
>> line.
>>
>> On Wed, Feb 10, 2016 at 8:33 AM, Jean Carlo <jean.jeancarl48@gmail.com>
>> wrote:
>>
>>> Hello guys!
>>>
>>> I am testing the repair inc in my custer cassandra. I am doing my test
>>> over these tables
>>>
>>> *CREATE TABLE pns_nonreg_bench.cf3* (
>>>     s text,
>>>     sp int,
>>>     d text,
>>>     dp int,
>>>     m map<text, text>,
>>>     t timestamp,
>>>     PRIMARY KEY (s, sp, d, dp)
>>> ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>>>
>>> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
>>> *LeveledCompactionStrategy*'}
>>>     AND compression = {'sstable_compression':
>>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>>>
>>> *CREATE TABLE pns_nonreg_bench.cf1* (
>>>     ise text PRIMARY KEY,
>>>     int_col int,
>>>     text_col text,
>>>     ts_col timestamp,
>>>     uuid_col uuid
>>> ) WITH bloom_filter_fp_chance = 0.01
>>>  AND compaction = {'class': 'org.apache.cassandra.db.compaction.
>>> *LeveledCompactionStrategy*'}
>>>     AND compression = {'sstable_compression':
>>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>>>
>>>
>>>
>>> *table cf1        Space used (live): 665.7 MB*
>>>
>>> *table cf2*
>>> *        Space used (live): 697.03 MB*
>>>
>>> It happens that when I do repair -inc -par on theses tables, *cf2 got a
>>> pick of 3k sstables*. When the repair finish, it takes 30 min or more
>>> to finish all the compactations and return to 6 sstable.
>>>
>>> I am a little concern about if this will happen on production. is it
>>> normal?
>>>
>>> Saludos
>>>
>>> Jean Carlo
>>>
>>> "The best way to predict the future is to invent it" Alan Kay
>>>
>>
>>
>

Mime
View raw message