cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: [EXTERNAL] Re: Cassandra rate dropping over long term test
Date Tue, 07 Aug 2018 15:25:52 GMT
What's the test do? Pseudocode + schema?

On Tue, Aug 7, 2018 at 8:22 AM, Mihai Stanescu <mihai.stanescu@gmail.com>
wrote:

> Is it possible your client/test is doing something iterative, where the
>> longer it runs, the more it reads or writes?
>
>
> It is possible but i don't see this reflected in the graphs as the number
> of serviced requests remains the same but the IOPS starts growing after a
> big compaction and also when read IOPS start growing i don't see an
> increase in the number of serviced requests. On the contrary as the CPU
> increases also the number of serviced requests drops.
>
> I could not find a metric for received requests but since nothing was
> dropped i guess this would mean that all are serviced and it does not look
> like the client is piling up.
>
> I think its more like what Rahul Singh said. I think the size based
> compaction keeps  on making bigger and bigger sstable which is anyway not
> needed. I am not sure exactly if there is a limit on how big SStable can
> get and when it stops compacting.
>
> I have switched to Time window compactor and its running much better, i
> will leave it to do the 24h test. IMO it would compact 80% of the traffic,
> the rest will expire at the expense of some disk. For the rest TTL is the
> say to go which is fine for me.
>
> I will update the progress here
>
>
> On Tue, Aug 7, 2018 at 5:13 PM, Jeff Jirsa <jjirsa@gmail.com> wrote:
>
>> I'd be surprised if there's a bug with long lived connections, but it's
>> possible.
>>
>> Is it possible your client/test is doing something iterative, where the
>> longer it runs, the more it reads or writes?
>>
>> On Tue, Aug 7, 2018 at 2:32 AM, Mihai Stanescu <mihai.stanescu@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I collected more metrics see below.
>>>
>>> It seems the cluster is destabilizing but i cannot clearly  see why.
>>>
>>> One weard bit is that if i restart the client everything is ok again.
>>> Could it be that there are some problems with the session which is kept
>>> open for too long? or something? The rate of requests hitting the server
>>> seems ok and seems the statements are executed ok.
>>>
>>>
>>> There is a decription of the table
>>>
>>> CREATE TABLE demodb.topic_message (
>>>     topic_name text,
>>>     message_index int,
>>>     crated_at bigint,
>>>     message blob,
>>>     tag text,
>>>     type text,
>>>     uuid text,
>>>     PRIMARY KEY (topic_name, message_index)
>>> ) WITH CLUSTERING ORDER BY (message_index ASC)
>>>     AND bloom_filter_fp_chance = 0.01
>>>     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>>     AND comment = ''
>>>     AND compaction = {'class': 'org.apache.cassandra.db.compa
>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>> 'min_threshold': '4'}
>>>     AND compression = {'chunk_length_in_kb': '64', 'class': '
>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>>     AND crc_check_chance = 1.0
>>>     AND dclocal_read_repair_chance = 0.1
>>>     AND default_time_to_live = 0
>>>     AND gc_grace_seconds = 0
>>>     AND max_index_interval = 2048
>>>     AND memtable_flush_period_in_ms = 0
>>>     AND min_index_interval = 128
>>>     AND read_repair_chance = 0.0
>>>     AND speculative_retry = '99PERCENTILE';
>>>
>>>
>>> message_index is an incrementing sequence per topic_name.
>>>
>>>
>>> I wonder if you are building up tombstones with the deletes
>>>
>>>
>>> There should be just one tombstone per entry give our model. The access
>>> pattern is also kind of time localized because the operations to
>>> read/delete are done on recent inserted rows.
>>>
>>> Actually you can see that the red line actually drops when the read IOPS
>>> start.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Any warnings in your system.log for reading through too many tombstones?
>>>
>>>
>>> Nope
>>>
>>> One thing to notice is that after we restarted the client the cluster
>>> recovered.
>>>
>>>
>>> Regards,
>>> Mihai
>>>
>>>
>>> On Fri, Aug 3, 2018 at 10:35 PM, Durity, Sean R <
>>> SEAN_R_DURITY@homedepot.com> wrote:
>>>
>>>> I wonder if you are building up tombstones with the deletes. Can you
>>>> share your data model? Are the deleted rows using the same partition key
as
>>>> new rows? Any warnings in your system.log for reading through too many
>>>> tombstones?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Sean Durity
>>>>
>>>>
>>>>
>>>> *From:* Mihai Stanescu <mihai.stanescu@gmail.com>
>>>> *Sent:* Friday, August 03, 2018 12:03 PM
>>>> *To:* user@cassandra.apache.org
>>>> *Subject:* [EXTERNAL] Re: Cassandra rate dropping over long term test
>>>>
>>>>
>>>>
>>>> I looked at the compaction history on the affected node when it was
>>>> affected and it was not affected.
>>>>
>>>>
>>>>
>>>> The number of compactions is fairly similar and the amount of work also.
>>>>
>>>>
>>>>
>>>> *Not affected time*
>>>>
>>>> [root@cassandra7 ~]# nodetool compactionhistory | grep 02T22
>>>>
>>>> fda43ca0-9696-11e8-8efb-25b020ed0402 demodb        topic_message
>>>> 2018-08-02T22:59:47.946 433124864  339496194  {1:3200576, 2:2025936,
>>>> 3:262919}
>>>>
>>>> 8a83e2c0-9696-11e8-8efb-25b020ed0402 demodb        topic_message
>>>> 2018-08-02T22:56:34.796 133610579  109321990  {1:1574352, 2:434814}
>>>>
>>>>
>>>> 01811e20-9696-11e8-8efb-25b020ed0402 demodb        topic_message
>>>> 2018-08-02T22:52:44.930 132847372  108175388  {1:1577164, 2:432508}
>>>>
>>>>
>>>>
>>>> *Experiencing more ioread*
>>>>
>>>> [root@cassandra7 ~]# nodetool compactionhistory | grep 03T12
>>>>
>>>> 389aa220-970c-11e8-8efb-25b020ed0402 demodb        topic_message
>>>> 2018-08-03T12:58:57.986 470326446  349948622  {1:2590960, 2:2600102,
>>>> 3:298369}
>>>>
>>>> 81fe6f10-970b-11e8-8efb-25b020ed0402 demodb        topic_message
>>>> 2018-08-03T12:53:51.617 143850880  115555226  {1:1686260, 2:453627}
>>>>
>>>>
>>>> ce418e30-970a-11e8-8efb-25b020ed0402 demodb        topic_message
>>>> 2018-08-03T12:48:50.067 147035600  119201638  {1:1742318, 2:452226}
>>>>
>>>>
>>>>
>>>> During a read operation the row can mostly be in one sstable since was
>>>> only inserted and then read so its strange.
>>>>
>>>>
>>>>
>>>> We have a partition key and then a clustering key.
>>>>
>>>>
>>>>
>>>> Rows that are written should be in kernel buffers and the rows which
>>>> are lost to delete are never read again either so the kernel should have
>>>> only the most recent data.
>>>>
>>>>
>>>>
>>>> I remain puzzled
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Aug 3, 2018 at 3:58 PM, Jeff Jirsa <jjirsa@gmail.com> wrote:
>>>>
>>>> Probably Compaction
>>>>
>>>> Cassandra data files are immutable
>>>>
>>>> The write path first appends to a commitlog, then puts data into the
>>>> memtable. When the memtable hits a threshold, it’s flushed to data files
on
>>>> disk (let’s call the first one “1”, second “2” and so on)
>>>>
>>>> Over time we build up multiple data files on disk - when Cassandra
>>>> reads, it will merge data in those files to give you the result you expect,
>>>> choosing the latest value for each column
>>>>
>>>> But it’s usually wasteful to lots of files around, and that merging is
>>>> expensive, so compaction combines those data files behind the scenes in a
>>>> background thread.
>>>>
>>>> By default they’re combined when 4 or more files are approximately the
>>>> same size, so if your write rate is such that you fill and flush the
>>>> memtable every 5 minutes, compaction will likely happen at least every 20
>>>> minutes (sometimes more). This is called size tiered compaction; there are
>>>> 4 strategies but size tiered is default and easiest to understand.
>>>>
>>>> You’re seeing mostly writes because the reads are likely in page cache
>>>> (the kernel doesn’t need to go to disk to read the files, it’s got them
in
>>>> memory for serving normal reads).
>>>>
>>>> --
>>>> Jeff Jirsa
>>>>
>>>>
>>>> > On Aug 3, 2018, at 12:30 AM, Mihai Stanescu <mihai.stanescu@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi all,
>>>> >
>>>> > I am perftesting cassandra over a longrun in a cluster of 8 nodes and
>>>> i noticed the rate of service drops.
>>>> > Most of the nodes have the CPU between 40-65% however one of the
>>>> nodes has a higher CPU and also started performing a lot of read IOPS as
>>>> seen in the image. (green is read IOPS)
>>>> >
>>>> > My test has a mixed rw scenario.
>>>> > 1. insert row
>>>> > 2. after 60 seconds read row
>>>> > 3. delete row.
>>>> >
>>>> > The rate of inserts is bigger than the rate of deletes so some delete
>>>> will not happen.
>>>> >
>>>> > I have checked the client it it does not accumulate RAM, GC is a
>>>> straight line so o don't understand whats going on.
>>>> >
>>>> > Any hints?
>>>> >
>>>> > Regards,
>>>> > MIhai
>>>> >
>>>> > <image.png>
>>>> >
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> The information in this Internet Email is confidential and may be
>>>> legally privileged. It is intended solely for the addressee. Access to this
>>>> Email by anyone else is unauthorized. If you are not the intended
>>>> recipient, any disclosure, copying, distribution or any action taken or
>>>> omitted to be taken in reliance on it, is prohibited and may be unlawful.
>>>> When addressed to our clients any opinions or advice contained in this
>>>> Email are subject to the terms and conditions expressed in any applicable
>>>> governing The Home Depot terms of business or client engagement letter. The
>>>> Home Depot disclaims all responsibility and liability for the accuracy and
>>>> content of this attachment and for any damages or losses arising from any
>>>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>>>> items of a destructive nature, which may be contained in this attachment
>>>> and shall not be liable for direct, indirect, consequential or special
>>>> damages in connection with this e-mail message or its attachment.
>>>>
>>>
>>>
>>
>

Mime
View raw message