cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antoine Bonavita <anto...@stickyads.tv>
Subject Re: Help diagnosing performance issue
Date Thu, 19 Nov 2015 23:58:51 GMT
Sebastian,

I took into account your suggestion and set max_sstable_age_days to 1.

I left the TTL at 432000 and the gc_grace_seconds at 172800. So, I 
expect SSTable older than 7 days to get deleted. Am I right ?

I did not change dclocal_read_repair_chance because I have only one DC 
at this point in time. Did you mean that I should set read_repair_chance 
to 0 ?

Thanks again for your time and help. Really appreciated.

A.


On 11/19/2015 02:36 AM, Sebastian Estevez wrote:
>     When you say drop you mean reduce the value (to 1 day for example),
>     not "don't set the value", right ?
>
>
> Yes.
>
>     If I set max sstable age days to 1, my understanding is that
>     SSTables with expired data (5 days) are not going to be compacted
>     ever. And therefore my disk usage will keep growing forever. Did I
>     miss something here ?
>
>
> We will expire sstables who's highest TTL is beyond gc_grace_seconds as
> of CASSANDRA-5228
> <https://issues.apache.org/jira/browse/CASSANDRA-5228>. This is nice
> because the sstable is just dropped for free, no need to scan it and
> remove tombstones which is very expensive and DTCS will guarantee that
> all the data within an sstable is close together in time.
>
>     So, if I set max sstable age days to 1, I have to run repairs at
>     least once a day, correct ?
>
>     I'm afraid I don't get your point about painful compactions.
>
>
> I was referring to the problems described here CASSANDRA-9644
> <https://issues.apache.org/jira/browse/CASSANDRA-9644>
>
>
>
>
> All the best,
>
>
> datastax_logo.png <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect |954 905 8615 | sebastian.estevez@datastax.com
> <mailto:sebastian.estevez@datastax.com>
>
> linkedin.png <https://www.linkedin.com/company/datastax>facebook.png
> <https://www.facebook.com/datastax>twitter.png
> <https://twitter.com/datastax>g+.png
> <https://plus.google.com/+Datastax/about><http://feeds.feedburner.com/datastax>
> <http://goog_410786983>
>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to
> any size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Wed, Nov 18, 2015 at 5:53 PM, Antoine Bonavita <antoine@stickyads.tv
> <mailto:antoine@stickyads.tv>> wrote:
>
>     Sebastian,
>
>     Your help is very much appreciated. I re-read the blog post and also
>     https://labs.spotify.com/2014/12/18/date-tiered-compaction/ but some
>     things are still confusing me.
>
>     Please see my questions inline below.
>
>     On 11/18/2015 04:21 PM, Sebastian Estevez wrote:
>
>         Yep, I think you've mixed up your DTCS levers. I would read, or
>         re-read
>         Marcus's post
>         http://www.datastax.com/dev/blog/datetieredcompactionstrategy
>
>         *TL;DR:*
>
>            * *base_time_seconds*  is the size of your initial window
>            * *max_sstable_age_days* is the time after which you stop
>         compacting
>              sstables
>            * *default_time_to_live* is the time after which data expires and
>              sstables will start to become available for GC. (432000 is
>         5 days)
>
>
>              Could it be that compaction is putting those in cache
>         constantly?
>
>
>         Yep, you'll keep compacting sstables until they're 10 days old
>         per your
>         current settings and when you compact there are reads and then
>         writes.
>
>
>
>         If you aren't doing any updates and most of your reads are within 1
>         hour, you can probably afford to drop max sstable age days.
>
>     When you say drop you mean reduce the value (to 1 day for example),
>     not "don't set the value", right ?
>
>     If I set max sstable age days to 1, my understanding is that
>     SSTables with expired data (5 days) are not going to be compacted
>     ever. And therefore my disk usage will keep growing forever. Did I
>     miss something here ?
>
>         Just make
>         sure you're doing your repairs more often than the max sstable
>         age days
>         to avoid some painful compactions.
>
>     So, if I set max sstable age days to 1, I have to run repairs at
>     least once a day, correct ?
>     I'm afraid I don't get your point about painful compactions.
>
>         Along the same lines, you should probably set
>         dclocal_read_repair_chance
>         to 0
>
>     Will try that.
>
>
>              Regarding the heap configuration, both are very similar
>
>
>         Probably unrelated but, is there a reason why they're not identical?
>         Especially the different new gen size could have gc implications.
>
>     Both are calculated by cassandra-env.sh. If my bash skills are still
>     intact, the NewGen size difference comes from the number of cores:
>     the 64G machine has 12 cores where the 32G machine has 8 cores (I
>     did not even realize this before looking into this, that's why I did
>     not mention it in my previous emails).
>
>     Thanks a lot for your help.
>
>     A.
>
>
>
>
>
>         All the best,
>
>
>         datastax_logo.png <http://www.datastax.com/>
>
>         Sebastián Estévez
>
>         Solutions Architect |954 905 8615 <tel:954%20905%208615> |
>         sebastian.estevez@datastax.com
>         <mailto:sebastian.estevez@datastax.com>
>         <mailto:sebastian.estevez@datastax.com
>         <mailto:sebastian.estevez@datastax.com>>
>
>         linkedin.png <https://www.linkedin.com/company/datastax>facebook.png
>         <https://www.facebook.com/datastax>twitter.png
>         <https://twitter.com/datastax>g+.png
>         <https://plus.google.com/+Datastax/about><http://feeds.feedburner.com/datastax>
>         <http://goog_410786983>
>
>
>         <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
>
>         DataStax is the fastest, most scalable distributed database
>         technology,
>         delivering Apache Cassandra to the world’s most innovative
>         enterprises.
>         Datastax is built to be agile, always-on, and predictably
>         scalable to
>         any size. With more than 500 customers in 45 countries, DataStax
>         is the
>         database technology and transactional backbone of choice for the
>         worlds
>         most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
>         On Wed, Nov 18, 2015 at 6:44 AM, Antoine Bonavita
>         <antoine@stickyads.tv <mailto:antoine@stickyads.tv>
>         <mailto:antoine@stickyads.tv <mailto:antoine@stickyads.tv>>> wrote:
>
>              Sebastian, Robet,
>
>              First, a big thank you to both of you for your help.
>
>              It looks like you were right. I used pcstat (awesome tool,
>         thanks
>              for that as well) and it appears some files I would not
>         expect to be
>              in cache actually are. Here is a sample of my output
>         (edited for
>              convenience, adding the file timestamp from the OS):
>
>              *
>
>         /var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5951-big-Data.db
>              - 000.619 % - Nov 16 12:25
>              *
>
>         /var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5954-big-Data.db
>              - 000.681 % - Nov 16 13:44
>              *
>
>         /var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5955-big-Data.db
>              -  000.610 % - Nov 16 14:11
>              *
>
>         /var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5956-big-Data.db
>              - 015.621 % - Nov 16 14:26
>              *
>
>         /var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5957-big-Data.db
>              - 015.558 % - Nov 16 14:50
>
>              The SSTables that come before are all at about 0% and the
>         ones that
>              come after it are all at about 15%.
>
>              As you can see the first SSTable at 15% date back from 24h.
>         Given my
>              application I'm pretty sure those are not from the reads
>         (reads of
>              data older than 1h is definitely under 0.1% of reads).
>         Could it be
>              that compaction is putting those in cache constantly ?
>              If so, then I'm probably confused on the meaning/effect of
>              max_sstable_age_days (set at 10 in my case) and
>         base_time_seconds
>              (not set in my case so the default of 3600 applies). I
>         would not
>              expect any compaction to happen beyond the first hour and
>         the 10
>              days is here to make sure data still gets expired and SSTables
>              removed (thus releasing disk space). I don't see where the
>         24h come
>              from.
>              If you guys can shed some light on this, it would be
>         awesome. I'm
>              sure I got something wrong.
>
>              Regarding the heap configuration, both are very similar:
>              * 32G machine: -Xms8049M -Xmx8049M -Xmn800M
>              * 64G machine: -Xms8192M -Xmx8192M -Xmn1200M
>              I think we can rule that out.
>
>              Thanks again for you help, I truly appreciate it.
>
>              A.
>
>              On 11/17/2015 08:48 PM, Robert Coli wrote:
>
>                  On Tue, Nov 17, 2015 at 11:08 AM, Sebastian Estevez
>                  <sebastian.estevez@datastax.com
>         <mailto:sebastian.estevez@datastax.com>
>                  <mailto:sebastian.estevez@datastax.com
>         <mailto:sebastian.estevez@datastax.com>>
>                  <mailto:sebastian.estevez@datastax.com
>         <mailto:sebastian.estevez@datastax.com>
>                  <mailto:sebastian.estevez@datastax.com
>         <mailto:sebastian.estevez@datastax.com>>>>
>                  wrote:
>
>                       You're sstables are probably falling out of page
>         cache on the
>                       smaller nodes and your slow disks are killing your
>         latencies.
>
>
>                  +1 most likely.
>
>                  Are the heaps the same size on both machines?
>
>                  =Rob
>
>
>              --
>              Antoine Bonavita (antoine@stickyads.tv
>         <mailto:antoine@stickyads.tv>
>              <mailto:antoine@stickyads.tv
>         <mailto:antoine@stickyads.tv>>) - CTO StickyADS.tv
>              Tel: +33 6 34 33 47 36 <tel:%2B33%206%2034%2033%2047%2036>
>         <tel:%2B33%206%2034%2033%2047%2036>/+33 9 50
>              68 21 32 <tel:%2B33%209%2050%2068%2021%2032>
>              NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN |
>         MADRID
>
>
>
>     --
>     Antoine Bonavita (antoine@stickyads.tv
>     <mailto:antoine@stickyads.tv>) - CTO StickyADS.tv
>     Tel: +33 6 34 33 47 36 <tel:%2B33%206%2034%2033%2047%2036>/+33 9 50
>     68 21 32 <tel:%2B33%209%2050%2068%2021%2032>
>     NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN | MADRID
>
>

-- 
Antoine Bonavita (antoine@stickyads.tv) - CTO StickyADS.tv
Tel: +33 6 34 33 47 36/+33 9 50 68 21 32
NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN | MADRID

Mime
View raw message