carbondata-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From prabhatkashyap <prabhat.kash...@knoldus.in>
Subject Re: What is the difference between Minor Compaction Level 1 and Minor Compaction Level 2?
Date Thu, 08 Jun 2017 09:24:05 GMT
Hi xm_zzc

There are two levels of Minor Compaction:
Level 1: Merging of the segments which are not yet compacted.
Level 2: Merging of the compacted segments again to form a bigger segment.

If you look into carbon.properties there is a property

> carbon.compaction.level.threshold=4,3

This property is set by default. 
Which means it will merge 4 segments at level 1 and 3 segments at level 2.

For example:
If you load 3 times then there will be segments:
Segment 0
Segment 1
Segment 2

And on the fourth load, if auto-compaction is enabled then the first four
segments will merge and will create a new segment with name like Segment 0.1
Similarly, after the eighth load, Level 1 compaction will run again and
create a new segment with a name like Segment 4.1 will be created. (Note:
This time Segment 4 to 7 will be merged). Same will happen for next four
loads and will create a new segment like before Segment 8.1. This is level 1
compaction.

Now if you see the property that for level 2 compaction the default value is
set to 3. As soon as we have three Level 1 compacted segments, level 2
compaction will be triggered and will merge 3 already compacted segments to
generate a new segment like Segment 0.2 (Note: Here Segment 0.1, 4.1 and 8.1
will get merged).

Also, there would be no compaction of the segment after level 2 compaction.
Like 0.2 will not be merged with any other segment.

Also, note that you can run compaction using alter command if the
auto-compaction is not enabled

Minor level 2 compaction is not equal to Major Compaction.

Minor compaction is based on numbers of segment and Major Compaction is
based on the size of segments.

For details, you can refer to documentation:
http://carbondata.apache.org/data-management.html#compacting-data





--
View this message in context: http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/What-is-the-difference-between-Minor-Compaction-Level-1-and-Minor-Compaction-Level-2-tp23p24.html
Sent from the Apache CarbonData User Mailing List mailing list archive at Nabble.com.

Mime
View raw message