cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Anchlia <mohitanch...@gmail.com>
Subject Re: Question regarding major compaction.
Date Tue, 01 May 2012 19:31:40 GMT
+1

On Tue, May 1, 2012 at 12:06 PM, Edward Capriolo <edlinuxguru@gmail.com>wrote:

> Also there are some tickets in JIRA to impose a max sstable size and
> some other related optimizations that I think got stuck behind levelDB
> in coolness factor. Not every use case is good for leveled so adding
> more tools and optimizations of the Size Tiered tables would be
> awesome.
>
> On Tue, May 1, 2012 at 10:15 AM, Jason Rutherglen
> <jason.rutherglen@gmail.com> wrote:
> > I wonder if TieredMergePolicy [1] could be used in Cassandra for
> compaction?
> >
> > 1.
> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
> >
> > On Tue, May 1, 2012 at 6:38 AM, Edward Capriolo <edlinuxguru@gmail.com>
> wrote:
> >> Henrik,
> >>
> >> There are use cases where major compaction works well like yours and
> >> mine. Essentially cases with a high amount of churn, updates and
> >> deletes we get a lot of benefit from forced tombstone removal in the
> >> form of less physical data.
> >>
> >> However we end up with really big sstables that naturally will never
> >> get compacted away since they are so much bigger then the other
> >> tables. So we get stuck always major compacting forever.
> >>
> >> Cassandra needs un compact for people like us so we can turn 1 big
> >> sstable into multiple smaller ones. Or a major compaction that takes
> >> in multiple sstables and produces multiple output tables nicely
> >> organized for bloom filter hits and tombstone free.
> >>
> >> Edward
> >>
> >> On Tue, May 1, 2012 at 7:31 AM, Henrik Schröder <skrolle@gmail.com>
> wrote:
> >>> But what's the difference between doing an extra read from that One Big
> >>> File, than doing an extra read from whatever SSTable happen to be
> largest in
> >>> the course of automatic minor compaction?
> >>>
> >>> We have a pretty update-heavy application, and doing a major
> compaction can
> >>> remove up to 30% of the used diskspace. That directly translates into
> less
> >>> reads and less SSTables that rows appear in. Everything that's
> unchanged
> >>> since the last major compaction is obviously faster to access, and
> >>> everything that's changed since the last major compaction is about the
> same
> >>> as if we hadn't done it?
> >>>
> >>> So I'm still confused. I don't see a significant difference between
> doing
> >>> the occasional major compaction or leaving it to do automatic minor
> >>> compactions. What am I missing? Reads will "continually degrade" with
> >>> automatic minor compactions as well, won't they?
> >>>
> >>> I can sort of see that if you have a moving active data set, then that
> will
> >>> most probably only exist in the smallest SSTables and frequently be the
> >>> object of minor compactions, and doing a major compaction will move
> all of
> >>> it into the biggest SSTables?
> >>>
> >>>
> >>> /Henrik
> >>>
> >>> On Mon, Apr 30, 2012 at 05:35, aaron morton <aaron@thelastpickle.com>
> wrote:
> >>>>
> >>>> Depends on your definition of significantly, there are a few things
to
> >>>> consider.
> >>>>
> >>>> * Reading from SSTables for a request is a serial operation. Reading
> from
> >>>> 2 SSTables will take twice as long as 1.
> >>>>
> >>>> * If the data in the One Big File™ has been overwritten, reading it
> is a
> >>>> waste of time. And it will continue to be read until it the row is
> compacted
> >>>> away.
> >>>>
> >>>> * You will need to get min_compaction_threshold (CF setting) SSTables
> that
> >>>> big before automatic compaction will pickup the big file.
> >>>>
> >>>> On the other side: Some people do report getting value from nightly
> major
> >>>> compactions. They also manage their cluster to reduce the impact of
> >>>> performing the compactions.
> >>>>
> >>>> Hope that helps.
> >>>>
> >>>> -----------------
> >>>> Aaron Morton
> >>>> Freelance Developer
> >>>> @aaronmorton
> >>>> http://www.thelastpickle.com
> >>>>
> >>>> On 26/04/2012, at 9:37 PM, Fredrik wrote:
> >>>>
> >>>> Exactly, but why would reads be significantly slower over time when
> >>>> including just one more, although sometimes large, SSTable in the
> read?
> >>>>
> >>>> Ji Cheng skrev 2012-04-26 11:11:
> >>>>
> >>>> I'm also quite interested in this question. Here's my understanding
on
> >>>> this problem.
> >>>>
> >>>> 1. If your workload is append-only, doing a major compaction shouldn't
> >>>> affect the read performance too much, because each row appears in one
> >>>> sstable anyway.
> >>>>
> >>>> 2. If your workload is mostly updating existing rows, then more and
> more
> >>>> columns will be obsoleted in that big sstable created by major
> compaction.
> >>>> And that super big sstable won't be compacted until you either have
> another
> >>>> 3 similar-sized sstables or start another major compaction. But I am
> not
> >>>> very sure whether this will be a major problem, because you only end
> up with
> >>>> reading one more sstable. Using size-tiered compaction against
> mostly-update
> >>>> workload itself may result in reading multiple sstables for a single
> row
> >>>> key.
> >>>>
> >>>> Please correct me if I am wrong.
> >>>>
> >>>> Cheng
> >>>>
> >>>>
> >>>> On Thu, Apr 26, 2012 at 3:50 PM, Fredrik
> >>>> <fredrik.l.stigback@sitevision.se> wrote:
> >>>>>
> >>>>> In the tuning documentation regarding Cassandra, it's recomended
not
> to
> >>>>> run major compactions.
> >>>>> I understand what a major compaction is all about but I'd like an
in
> >>>>> depth explanation as to why reads "will continually degrade until
> the next
> >>>>> major compaction is manually invoked".
> >>>>>
> >>>>> From the doc:
> >>>>> "So while read performance will be good immediately following a
major
> >>>>> compaction, it will continually degrade until the next major
> compaction is
> >>>>> manually invoked. For this reason, major compaction is NOT
> recommended by
> >>>>> DataStax."
> >>>>>
> >>>>> Regards
> >>>>> /Fredrik
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
>

Mime
View raw message