cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antoine Bonavita <anto...@stickyads.tv>
Subject Re: Help diagnosing performance issue
Date Wed, 25 Nov 2015 16:05:03 GMT
Sebastian (and others, help is always appreciated),

After 24h OK, read latencies started to degrade (up to 20ms) and I had 
to ramp down volumes again.

The degradation is clearly linked to the number read IOPs which went up 
to 1.65k/s after 24h.

If anybody can give me hints on what I should look at, I'm very happy to 
do so.

A.

On 11/23/2015 12:07 PM, Antoine Bonavita wrote:
> Sebastian,
>
> I tried to ramp up volume with this new setting and ran into the same
> problems.
>
> After that I restarted my nodes. This pretty much instantly got read
> latencies back to normal (< 5ms) on the 32G nodes.
>
> I am currently ramping up volumes again and here is what I am seeing on
> 32G nodes:
> * Read latencies are OK (<5ms)
> * A lot of read IOPS (~ 400 read/s)
> * I enabled logging for the DateCompactionStrategy and I get only this
> kind of lines :
> DEBUG [CompactionExecutor:186] 2015-11-23 12:02:45,915
> DateTieredCompactionStrategy.java:137 - Compaction buckets are []
> DEBUG [CompactionExecutor:186] 2015-11-23 12:03:16,704
> DateTieredCompactionStrategy.java:137 - Compaction buckets are
> [[BigTableReader(path='/var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-6452-big-Data.db')]]
>
> * When I run pcstats I still get about 100 *-Data.db files loaded at 15%
> (which is what I was seeing with max_sstable_age_days set at 5).
>
> I'm really happy with the first item in my list but the other items seem
> to indicate something is still wrong and it does not look like it's
> compaction.
>
> Any help would be truly appreciated.
>
> A.
>
> On 11/20/2015 12:58 AM, Antoine Bonavita wrote:
>> Sebastian,
>>
>> I took into account your suggestion and set max_sstable_age_days to 1.
>>
>> I left the TTL at 432000 and the gc_grace_seconds at 172800. So, I
>> expect SSTable older than 7 days to get deleted. Am I right ?
>>
>> I did not change dclocal_read_repair_chance because I have only one DC
>> at this point in time. Did you mean that I should set read_repair_chance
>> to 0 ?
>>
>> Thanks again for your time and help. Really appreciated.
>>
>> A.
>>
>>
>> On 11/19/2015 02:36 AM, Sebastian Estevez wrote:
>>>     When you say drop you mean reduce the value (to 1 day for example),
>>>     not "don't set the value", right ?
>>>
>>>
>>> Yes.
>>>
>>>     If I set max sstable age days to 1, my understanding is that
>>>     SSTables with expired data (5 days) are not going to be compacted
>>>     ever. And therefore my disk usage will keep growing forever. Did I
>>>     miss something here ?
>>>
>>>
>>> We will expire sstables who's highest TTL is beyond gc_grace_seconds as
>>> of CASSANDRA-5228
>>> <https://issues.apache.org/jira/browse/CASSANDRA-5228>. This is nice
>>> because the sstable is just dropped for free, no need to scan it and
>>> remove tombstones which is very expensive and DTCS will guarantee that
>>> all the data within an sstable is close together in time.
>>>
>>>     So, if I set max sstable age days to 1, I have to run repairs at
>>>     least once a day, correct ?
>>>
>>>     I'm afraid I don't get your point about painful compactions.
>>>
>>>
>>> I was referring to the problems described here CASSANDRA-9644
>>> <https://issues.apache.org/jira/browse/CASSANDRA-9644>
>>>
>>>
>>>
>>>
>>> All the best,
>>>
>>>
>>> datastax_logo.png <http://www.datastax.com/>
>>>
>>> Sebastián Estévez
>>>
>>> Solutions Architect |954 905 8615 | sebastian.estevez@datastax.com
>>> <mailto:sebastian.estevez@datastax.com>
>>>
>>> linkedin.png <https://www.linkedin.com/company/datastax>facebook.png
>>> <https://www.facebook.com/datastax>twitter.png
>>> <https://twitter.com/datastax>g+.png
>>> <https://plus.google.com/+Datastax/about><http://feeds.feedburner.com/datastax>
>>>
>>>
>>> <http://goog_410786983>
>>>
>>>
>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>
>>>
>>> DataStax is the fastest, most scalable distributed database technology,
>>> delivering Apache Cassandra to the world’s most innovative enterprises.
>>> Datastax is built to be agile, always-on, and predictably scalable to
>>> any size. With more than 500 customers in 45 countries, DataStax is the
>>> database technology and transactional backbone of choice for the worlds
>>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>
>>> On Wed, Nov 18, 2015 at 5:53 PM, Antoine Bonavita <antoine@stickyads.tv
>>> <mailto:antoine@stickyads.tv>> wrote:
>>>
>>>     Sebastian,
>>>
>>>     Your help is very much appreciated. I re-read the blog post and also
>>>     https://labs.spotify.com/2014/12/18/date-tiered-compaction/ but some
>>>     things are still confusing me.
>>>
>>>     Please see my questions inline below.
>>>
>>>     On 11/18/2015 04:21 PM, Sebastian Estevez wrote:
>>>
>>>         Yep, I think you've mixed up your DTCS levers. I would read, or
>>>         re-read
>>>         Marcus's post
>>>         http://www.datastax.com/dev/blog/datetieredcompactionstrategy
>>>
>>>         *TL;DR:*
>>>
>>>            * *base_time_seconds*  is the size of your initial window
>>>            * *max_sstable_age_days* is the time after which you stop
>>>         compacting
>>>              sstables
>>>            * *default_time_to_live* is the time after which data
>>> expires and
>>>              sstables will start to become available for GC. (432000 is
>>>         5 days)
>>>
>>>
>>>              Could it be that compaction is putting those in cache
>>>         constantly?
>>>
>>>
>>>         Yep, you'll keep compacting sstables until they're 10 days old
>>>         per your
>>>         current settings and when you compact there are reads and then
>>>         writes.
>>>
>>>
>>>
>>>         If you aren't doing any updates and most of your reads are
>>> within 1
>>>         hour, you can probably afford to drop max sstable age days.
>>>
>>>     When you say drop you mean reduce the value (to 1 day for example),
>>>     not "don't set the value", right ?
>>>
>>>     If I set max sstable age days to 1, my understanding is that
>>>     SSTables with expired data (5 days) are not going to be compacted
>>>     ever. And therefore my disk usage will keep growing forever. Did I
>>>     miss something here ?
>>>
>>>         Just make
>>>         sure you're doing your repairs more often than the max sstable
>>>         age days
>>>         to avoid some painful compactions.
>>>
>>>     So, if I set max sstable age days to 1, I have to run repairs at
>>>     least once a day, correct ?
>>>     I'm afraid I don't get your point about painful compactions.
>>>
>>>         Along the same lines, you should probably set
>>>         dclocal_read_repair_chance
>>>         to 0
>>>
>>>     Will try that.
>>>
>>>
>>>              Regarding the heap configuration, both are very similar
>>>
>>>
>>>         Probably unrelated but, is there a reason why they're not
>>> identical?
>>>         Especially the different new gen size could have gc
>>> implications.
>>>
>>>     Both are calculated by cassandra-env.sh. If my bash skills are still
>>>     intact, the NewGen size difference comes from the number of cores:
>>>     the 64G machine has 12 cores where the 32G machine has 8 cores (I
>>>     did not even realize this before looking into this, that's why I did
>>>     not mention it in my previous emails).
>>>
>>>     Thanks a lot for your help.
>>>
>>>     A.
>>>
>>>
>>>
>>>
>>>
>>>         All the best,
>>>
>>>
>>>         datastax_logo.png <http://www.datastax.com/>
>>>
>>>         Sebastián Estévez
>>>
>>>         Solutions Architect |954 905 8615 <tel:954%20905%208615> |
>>>         sebastian.estevez@datastax.com
>>>         <mailto:sebastian.estevez@datastax.com>
>>>         <mailto:sebastian.estevez@datastax.com
>>>         <mailto:sebastian.estevez@datastax.com>>
>>>
>>>         linkedin.png
>>> <https://www.linkedin.com/company/datastax>facebook.png
>>>         <https://www.facebook.com/datastax>twitter.png
>>>         <https://twitter.com/datastax>g+.png
>>>
>>> <https://plus.google.com/+Datastax/about><http://feeds.feedburner.com/datastax>
>>>
>>>
>>>         <http://goog_410786983>
>>>
>>>
>>>         <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>
>>>
>>>         DataStax is the fastest, most scalable distributed database
>>>         technology,
>>>         delivering Apache Cassandra to the world’s most innovative
>>>         enterprises.
>>>         Datastax is built to be agile, always-on, and predictably
>>>         scalable to
>>>         any size. With more than 500 customers in 45 countries, DataStax
>>>         is the
>>>         database technology and transactional backbone of choice for the
>>>         worlds
>>>         most innovative companies such as Netflix, Adobe, Intuit, and
>>> eBay.
>>>
>>>         On Wed, Nov 18, 2015 at 6:44 AM, Antoine Bonavita
>>>         <antoine@stickyads.tv <mailto:antoine@stickyads.tv>
>>>         <mailto:antoine@stickyads.tv <mailto:antoine@stickyads.tv>>>
>>> wrote:
>>>
>>>              Sebastian, Robet,
>>>
>>>              First, a big thank you to both of you for your help.
>>>
>>>              It looks like you were right. I used pcstat (awesome tool,
>>>         thanks
>>>              for that as well) and it appears some files I would not
>>>         expect to be
>>>              in cache actually are. Here is a sample of my output
>>>         (edited for
>>>              convenience, adding the file timestamp from the OS):
>>>
>>>              *
>>>
>>>
>>> /var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5951-big-Data.db
>>>
>>>
>>>              - 000.619 % - Nov 16 12:25
>>>              *
>>>
>>>
>>> /var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5954-big-Data.db
>>>
>>>
>>>              - 000.681 % - Nov 16 13:44
>>>              *
>>>
>>>
>>> /var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5955-big-Data.db
>>>
>>>
>>>              -  000.610 % - Nov 16 14:11
>>>              *
>>>
>>>
>>> /var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5956-big-Data.db
>>>
>>>
>>>              - 015.621 % - Nov 16 14:26
>>>              *
>>>
>>>
>>> /var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5957-big-Data.db
>>>
>>>
>>>              - 015.558 % - Nov 16 14:50
>>>
>>>              The SSTables that come before are all at about 0% and the
>>>         ones that
>>>              come after it are all at about 15%.
>>>
>>>              As you can see the first SSTable at 15% date back from 24h.
>>>         Given my
>>>              application I'm pretty sure those are not from the reads
>>>         (reads of
>>>              data older than 1h is definitely under 0.1% of reads).
>>>         Could it be
>>>              that compaction is putting those in cache constantly ?
>>>              If so, then I'm probably confused on the meaning/effect of
>>>              max_sstable_age_days (set at 10 in my case) and
>>>         base_time_seconds
>>>              (not set in my case so the default of 3600 applies). I
>>>         would not
>>>              expect any compaction to happen beyond the first hour and
>>>         the 10
>>>              days is here to make sure data still gets expired and
>>> SSTables
>>>              removed (thus releasing disk space). I don't see where the
>>>         24h come
>>>              from.
>>>              If you guys can shed some light on this, it would be
>>>         awesome. I'm
>>>              sure I got something wrong.
>>>
>>>              Regarding the heap configuration, both are very similar:
>>>              * 32G machine: -Xms8049M -Xmx8049M -Xmn800M
>>>              * 64G machine: -Xms8192M -Xmx8192M -Xmn1200M
>>>              I think we can rule that out.
>>>
>>>              Thanks again for you help, I truly appreciate it.
>>>
>>>              A.
>>>
>>>              On 11/17/2015 08:48 PM, Robert Coli wrote:
>>>
>>>                  On Tue, Nov 17, 2015 at 11:08 AM, Sebastian Estevez
>>>                  <sebastian.estevez@datastax.com
>>>         <mailto:sebastian.estevez@datastax.com>
>>>                  <mailto:sebastian.estevez@datastax.com
>>>         <mailto:sebastian.estevez@datastax.com>>
>>>                  <mailto:sebastian.estevez@datastax.com
>>>         <mailto:sebastian.estevez@datastax.com>
>>>                  <mailto:sebastian.estevez@datastax.com
>>>         <mailto:sebastian.estevez@datastax.com>>>>
>>>                  wrote:
>>>
>>>                       You're sstables are probably falling out of page
>>>         cache on the
>>>                       smaller nodes and your slow disks are killing your
>>>         latencies.
>>>
>>>
>>>                  +1 most likely.
>>>
>>>                  Are the heaps the same size on both machines?
>>>
>>>                  =Rob
>>>
>>>
>>>              --
>>>              Antoine Bonavita (antoine@stickyads.tv
>>>         <mailto:antoine@stickyads.tv>
>>>              <mailto:antoine@stickyads.tv
>>>         <mailto:antoine@stickyads.tv>>) - CTO StickyADS.tv
>>>              Tel: +33 6 34 33 47 36 <tel:%2B33%206%2034%2033%2047%2036>
>>>         <tel:%2B33%206%2034%2033%2047%2036>/+33 9 50
>>>              68 21 32 <tel:%2B33%209%2050%2068%2021%2032>
>>>              NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN |
>>>         MADRID
>>>
>>>
>>>
>>>     --
>>>     Antoine Bonavita (antoine@stickyads.tv
>>>     <mailto:antoine@stickyads.tv>) - CTO StickyADS.tv
>>>     Tel: +33 6 34 33 47 36 <tel:%2B33%206%2034%2033%2047%2036>/+33 9 50
>>>     68 21 32 <tel:%2B33%209%2050%2068%2021%2032>
>>>     NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN | MADRID
>>>
>>>
>>
>

-- 
Antoine Bonavita (antoine@stickyads.tv) - CTO StickyADS.tv
Tel: +33 6 34 33 47 36/+33 9 50 68 21 32
NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN | MADRID

Mime
View raw message