cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: Constant blocking read repair for such a tiny table
Date Tue, 15 Oct 2019 19:29:39 GMT
Hello Patrick,

Still in trouble with this? I must admit I'm really puzzled by your issue.
I have no real idea of what's going on. Would you share with us the output
of:

- nodetool status <keyspace>
- nodetool describecluster
- nodetool gossipinfo
- nodetool tpstats

Also you said the app is running for a long time, with no changes. What
about Cassandra? Any recent operations?

I hope that with this information we might be able to understand better and
finally be able to help.

-----------------------
Alain Rodriguez - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Le ven. 4 oct. 2019 à 00:25, Patrick Lee <patrickclee0207@gmail.com> a
écrit :

> this table was actually leveled compaction before, just changed it to size
> tiered yesterday while researching this.
>
> On Thu, Oct 3, 2019 at 4:31 PM Patrick Lee <patrickclee0207@gmail.com>
> wrote:
>
>> its not really time series data.   and it's not updated very often, it
>> would have some updates but pretty infrequent. this thing should be super
>> fast, on avg it's like 1 to 2ms p99 currently but if they double - triple
>> the traffic on that table latencies go upward to 20ms to 50ms.. the only
>> odd thing i see is just that there are constant read repairs that follow
>> the same traffic pattern on the reads, which shows constant writes on the
>> table (from the read repairs), which after read repair or just normal full
>> repairs (all full through reaper, never ran any incremental repair) i would
>> expect it to not have any mismatches.  the other 5 tables they use on the
>> cluster can have the same level traffic all very simple select from table
>> by partition key which returns a single record
>>
>> On Thu, Oct 3, 2019 at 4:21 PM John Belliveau <belliveau.john@gmail.com>
>> wrote:
>>
>>> Hi Patrick,
>>>
>>>
>>>
>>> Is this time series data? If so, I have run into issues with repair on
>>> time series data using the SizeTieredCompactionStrategy. I have had
>>> better luck using the TimeWindowCompactionStrategy.
>>>
>>>
>>>
>>> John
>>>
>>>
>>>
>>> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
>>> Windows 10
>>>
>>>
>>>
>>> *From: *Patrick Lee <patrickclee0207@gmail.com>
>>> *Sent: *Thursday, October 3, 2019 5:14 PM
>>> *To: *user@cassandra.apache.org
>>> *Subject: *Constant blocking read repair for such a tiny table
>>>
>>>
>>>
>>> I have a cluster that is running 3.11.4 ( was upgraded a while back from
>>> 2.1.16 ).  what I see is a steady rate of read repair which is about 10%
>>> constantly on only this 1 table.  Repairs have been run (actually several
>>> times).  The table does not have a lot of writes to it so after repair, or
>>> even after a read repair I would expect it to be fine.  the reason i'm
>>> having to dig into this so much is for the fact that under a much large
>>> traffic load than their normal traffic, latencies are higher than the app
>>> team wants
>>>
>>>
>>>
>>> I mean this thing is tiny, it's a 12x12 cluster but this 1 table is like
>>> 1GB per node on disk.
>>>
>>>
>>>
>>> the application team is doing reads at LOCAL_QUORUM and I can simulate
>>> this on that cluster by running a query using quorum and/or local_quorum
>>> and in the trace can see every time running the query it comes back with a
>>> DigestMismatchException no matter how many times I run it. that record
>>> hasn't been updated by the application for several months.
>>>
>>>
>>>
>>> repairs are scheduled and run every 7 days via reaper, recently in the
>>> past week this table has been repaired at least 3 times.  every time there
>>> are mismatches and data streams back and forth but yet still a constant
>>> rate of read repairs.
>>>
>>>
>>>
>>> curious if anyone has any recommendations to look info further or have
>>> experienced anything like this?
>>>
>>>
>>>
>>> this node has been up for 24 hours.. this is the netstats for read
>>> repairs
>>>
>>> Mode: NORMAL
>>> Not sending any streams.
>>> Read Repair Statistics:
>>> Attempted: 7481
>>> Mismatch (Blocking): 11425375
>>> Mismatch (Background): 17
>>> Pool Name                    Active   Pending      Completed   Dropped
>>> Large messages                  n/a         0           1232         0
>>> Small messages                  n/a         0      395903678         0
>>> Gossip messages                 n/a         0         603746         0
>>>
>>>
>>>
>>> example of the schema... some modifications have been made to reduce
>>> read_reapair and speculative_retry while troubleshooting..
>>>
>>> CREATE TABLE keyspace.table1 (
>>>
>>>     item bigint,
>>>
>>>     price int,
>>>
>>>     start_date timestamp,
>>>
>>>     end_date timestamp,
>>>
>>>     created_date timestamp,
>>>
>>>     cost decimal,
>>>
>>>     list decimal,
>>>
>>>     item_id int,
>>>
>>>     modified_date timestamp,
>>>
>>>     status int,
>>>
>>>     PRIMARY KEY ((item, price), start_date, end_date)
>>>
>>> ) WITH CLUSTERING ORDER BY (start_date ASC, end_date ASC)
>>>
>>>     AND read_repair_chance = 0.0
>>>
>>>     AND dclocal_read_repair_chance = 0.0
>>>
>>>     AND gc_grace_seconds = 864000
>>>
>>>     AND bloom_filter_fp_chance = 0.01
>>>
>>>     AND caching = { 'keys' : 'ALL', 'rows_per_partition' : 'NONE' }
>>>
>>>     AND comment = ''
>>>
>>>     AND compaction = { 'class' :
>>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>>> 'max_threshold' : 32, 'min_threshold' : 4 }
>>>
>>>     AND compression = { 'chunk_length_in_kb' : 4, 'class' :
>>> 'org.apache.cassandra.io.compress.LZ4Compressor' }
>>>
>>>     AND default_time_to_live = 0
>>>
>>>     AND speculative_retry = 'NONE'
>>>
>>>     AND min_index_interval = 128
>>>
>>>     AND max_index_interval = 2048
>>>
>>>     AND crc_check_chance = 1.0
>>>
>>>     AND cdc = false
>>>
>>>     AND memtable_flush_period_in_ms = 0;
>>>
>>>
>>>
>>

Mime
View raw message