incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ying Tang <ivytang0...@gmail.com>
Subject Re: When to call the major compaction ?
Date Thu, 02 Dec 2010 01:57:39 GMT
@Chen Xinli
"and mark old sstables as deleted which will be deleted while jvm gc."
SSTable is on the harddisk , how could jvm gc delete it ? JVM GC is in
charge the using of the space in the memory.

@Nick
The GC in cassandra doesn't refer to jvm gc ? This kind of gc is cassandda's
gc , intend to remove the unused file on harddisk ?



On Wed, Dec 1, 2010 at 10:54 PM, Chen Xinli <chen.daqi@gmail.com> wrote:

>
>
>  2010/12/1 Ying Tang <ivytang0812@gmail.com>
>
>> I'm confused , plz ingore the mail above.
>>  Here is my confusion ,
>>    posterior to 0.6.6/0.7  , minor compaction and major compaction both
>> can clean out rows 'tagged'  tombstones  , and generate a new , without
>> tombstones , sstable .
>>
>
> This is right.
>
>
>>     And the tombstones remains in memory ,waiting to be removed by jvm gc
>> .
>> Am i right?
>>
>
> No! Compactions merge several old sstables into one, and mark old sstables
> as deleted which will be deleted while jvm gc.
> SSTable are files on harddisk, nothing to do with memory. You'd better have
> a look at Google's bigtable paper.
>
>
>>
>>   On Wed, Dec 1, 2010 at 9:10 PM, Ying Tang <ivytang0812@gmail.com>wrote:
>>
>>> 1. So posterior to 0.6.6/0.7 ,  minor compaction and major compaction
>>> both  can clean out rows 'tagged'  tombstones , this kind of clean out
>>> doesn't mead remove it from the disk permanently.
>>>     The real remove is done by the jvm GC ?
>>> 2. The intence of compaction is merging multi sstables into one , clean
>>> out the tombstone , let the un-tombstones  rows be into a new ordered
>>> sstable ?
>>>
>>>
>>>
>>> On Wed, Dec 1, 2010 at 7:30 PM, Sylvain Lebresne <sylvain@yakaz.com>wrote:
>>>
>>>> On Wed, Dec 1, 2010 at 12:11 PM, Ying Tang <ivytang0812@gmail.com>
>>>> wrote:
>>>> > And i have another question , what's the difference between minor
>>>> > compaction and major compaction?
>>>>
>>>> A major compaction is a compaction that compact *all* the SSTables of a
>>>> given
>>>> column family (compaction compacts one CF at a time).
>>>>
>>>> Before https://issues.apache.org/jira/browse/CASSANDRA-1074
>>>> (introduced in 0.6.6 and
>>>> recent 0.7 betas/rcs), major compactions where the only ones that
>>>> removed the
>>>> tombstones (see http://wiki.apache.org/cassandra/DistributedDeletes)
>>>> and this is the
>>>> reason major compaction exists. Now, with #1074, minor compactions
>>>> should remove most
>>>> if not all tombstones, so major compaction are not or much less useful
>>>> (it may depend on your
>>>> workload though as minor can't always delete the tombstones).
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>> >
>>>> > On 12/1/10, Chen Xinli <chen.daqi@gmail.com> wrote:
>>>> >> 2010/12/1 Ying Tang <ivytang0812@gmail.com>
>>>> >>
>>>> >>> Every time cassandra creates a new sstable , it will call the
>>>> >>> CompactionManager.submitMinorIfNeeded  ? And if the number of
>>>> memtables is
>>>> >>> beyond  MinimumCompactionThreshold  , the minor compaction will
be
>>>> called.
>>>> >>> And there is also a method named CompactionManager.submitMajor
, and
>>>> the
>>>> >>> call relationship is :
>>>> >>>
>>>> >>> NodeCmd -- > NodeProbe -->StorageService.forceTableCompaction
-->
>>>> >>> Table.forceCompaction -->CompactionManager.performMajor -->
>>>> >>> CompactionManager.submitMajor
>>>> >>>
>>>> >>> ColumnFamilyStore.forceMajorCompaction -->
>>>> CompactionManager.performMajor
>>>> >>> --> CompactionManager.submitMajor
>>>> >>>
>>>> >>>
>>>> >>> HintedHandOffManager
>>>> >>>  --> CompactionManager.submitMajor
>>>> >>>
>>>> >>> So i have 3 questions:
>>>> >>> 1. Once a new sstable has been created ,
>>>> >>> CompactionManager.submitMinorIfNeeded  will be called ,
>>>> minorCompaction
>>>> >>> maybe called .
>>>> >>>     But when will the majorCompaction be called ? Just the NodeCmd
?
>>>> >>>
>>>> >>
>>>> >> Yes, majorCompaction must be called manually from NodeCmd
>>>> >>
>>>> >>
>>>> >>> 2. Which jobs will minorCompaction and majorCompaction do ?
>>>> >>>     Will minorCompaction delete the data that have been marked
as
>>>> deleted
>>>> >>> ?
>>>> >>>     And how about the major compaction ?
>>>> >>>
>>>> >>
>>>> >> Compaction only mark sstables as deleted. Deletion will be done
when
>>>> there
>>>> >> are full gc, or node restarted.
>>>> >>
>>>> >>
>>>> >>> 3. When gc be called ? Every time compaction been called?
>>>> >>>
>>>> >>
>>>> >> GC has nothing to do with compaction, you may mistake the two
>>>> conceptions
>>>> >>
>>>> >>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Best regards,
>>>> >>>
>>>> >>> Ivy Tang
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Chen Xinli
>>>> >>
>>>> >
>>>> >
>>>> > --
>>>> > Best regards,
>>>> >
>>>> > Ivy Tang
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Ivy Tang
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Ivy Tang
>>
>>
>>
>>
>
>
> --
> Best Regards,
> Chen Xinli
>



-- 
Best regards,

Ivy Tang

Mime
View raw message