incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Xinli <chen.d...@gmail.com>
Subject Re: When to call the major compaction ?
Date Thu, 02 Dec 2010 02:57:41 GMT
You are right, jvm gc is for memory.
In cassandra, there is a small trick called *PhantomReference*, which will
be called when jvm gc. And deletion is actually done in PhantomReference.

2010/12/2 Ying Tang <ivytang0812@gmail.com>

> @Chen Xinli
> "and mark old sstables as deleted which will be deleted while jvm gc."
> SSTable is on the harddisk , how could jvm gc delete it ? JVM GC is in
> charge the using of the space in the memory.
>
> @Nick
> The GC in cassandra doesn't refer to jvm gc ? This kind of gc is
> cassandda's gc , intend to remove the unused file on harddisk ?
>
>
>
> On Wed, Dec 1, 2010 at 10:54 PM, Chen Xinli <chen.daqi@gmail.com> wrote:
>
>>
>>
>>  2010/12/1 Ying Tang <ivytang0812@gmail.com>
>>
>>> I'm confused , plz ingore the mail above.
>>>  Here is my confusion ,
>>>    posterior to 0.6.6/0.7  , minor compaction and major compaction both
>>> can clean out rows 'tagged'  tombstones  , and generate a new , without
>>> tombstones , sstable .
>>>
>>
>> This is right.
>>
>>
>>>     And the tombstones remains in memory ,waiting to be removed by jvm gc
>>> .
>>> Am i right?
>>>
>>
>> No! Compactions merge several old sstables into one, and mark old sstables
>> as deleted which will be deleted while jvm gc.
>> SSTable are files on harddisk, nothing to do with memory. You'd better
>> have a look at Google's bigtable paper.
>>
>>
>>>
>>>   On Wed, Dec 1, 2010 at 9:10 PM, Ying Tang <ivytang0812@gmail.com>wrote:
>>>
>>>> 1. So posterior to 0.6.6/0.7 ,  minor compaction and major compaction
>>>> both  can clean out rows 'tagged'  tombstones , this kind of clean out
>>>> doesn't mead remove it from the disk permanently.
>>>>     The real remove is done by the jvm GC ?
>>>> 2. The intence of compaction is merging multi sstables into one , clean
>>>> out the tombstone , let the un-tombstones  rows be into a new ordered
>>>> sstable ?
>>>>
>>>>
>>>>
>>>> On Wed, Dec 1, 2010 at 7:30 PM, Sylvain Lebresne <sylvain@yakaz.com>wrote:
>>>>
>>>>> On Wed, Dec 1, 2010 at 12:11 PM, Ying Tang <ivytang0812@gmail.com>
>>>>> wrote:
>>>>> > And i have another question , what's the difference between minor
>>>>> > compaction and major compaction?
>>>>>
>>>>> A major compaction is a compaction that compact *all* the SSTables of
a
>>>>> given
>>>>> column family (compaction compacts one CF at a time).
>>>>>
>>>>> Before https://issues.apache.org/jira/browse/CASSANDRA-1074
>>>>> (introduced in 0.6.6 and
>>>>> recent 0.7 betas/rcs), major compactions where the only ones that
>>>>> removed the
>>>>> tombstones (see http://wiki.apache.org/cassandra/DistributedDeletes)
>>>>> and this is the
>>>>> reason major compaction exists. Now, with #1074, minor compactions
>>>>> should remove most
>>>>> if not all tombstones, so major compaction are not or much less useful
>>>>> (it may depend on your
>>>>> workload though as minor can't always delete the tombstones).
>>>>>
>>>>> --
>>>>> Sylvain
>>>>>
>>>>> >
>>>>> > On 12/1/10, Chen Xinli <chen.daqi@gmail.com> wrote:
>>>>> >> 2010/12/1 Ying Tang <ivytang0812@gmail.com>
>>>>> >>
>>>>> >>> Every time cassandra creates a new sstable , it will call
the
>>>>> >>> CompactionManager.submitMinorIfNeeded  ? And if the number
of
>>>>> memtables is
>>>>> >>> beyond  MinimumCompactionThreshold  , the minor compaction
will be
>>>>> called.
>>>>> >>> And there is also a method named CompactionManager.submitMajor
,
>>>>> and the
>>>>> >>> call relationship is :
>>>>> >>>
>>>>> >>> NodeCmd -- > NodeProbe -->StorageService.forceTableCompaction
-->
>>>>> >>> Table.forceCompaction -->CompactionManager.performMajor
-->
>>>>> >>> CompactionManager.submitMajor
>>>>> >>>
>>>>> >>> ColumnFamilyStore.forceMajorCompaction -->
>>>>> CompactionManager.performMajor
>>>>> >>> --> CompactionManager.submitMajor
>>>>> >>>
>>>>> >>>
>>>>> >>> HintedHandOffManager
>>>>> >>>  --> CompactionManager.submitMajor
>>>>> >>>
>>>>> >>> So i have 3 questions:
>>>>> >>> 1. Once a new sstable has been created ,
>>>>> >>> CompactionManager.submitMinorIfNeeded  will be called ,
>>>>> minorCompaction
>>>>> >>> maybe called .
>>>>> >>>     But when will the majorCompaction be called ? Just the
NodeCmd
>>>>> ?
>>>>> >>>
>>>>> >>
>>>>> >> Yes, majorCompaction must be called manually from NodeCmd
>>>>> >>
>>>>> >>
>>>>> >>> 2. Which jobs will minorCompaction and majorCompaction do
?
>>>>> >>>     Will minorCompaction delete the data that have been
marked as
>>>>> deleted
>>>>> >>> ?
>>>>> >>>     And how about the major compaction ?
>>>>> >>>
>>>>> >>
>>>>> >> Compaction only mark sstables as deleted. Deletion will be done
when
>>>>> there
>>>>> >> are full gc, or node restarted.
>>>>> >>
>>>>> >>
>>>>> >>> 3. When gc be called ? Every time compaction been called?
>>>>> >>>
>>>>> >>
>>>>> >> GC has nothing to do with compaction, you may mistake the two
>>>>> conceptions
>>>>> >>
>>>>> >>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> Best regards,
>>>>> >>>
>>>>> >>> Ivy Tang
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Chen Xinli
>>>>> >>
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Best regards,
>>>>> >
>>>>> > Ivy Tang
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Ivy Tang
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Ivy Tang
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Chen Xinli
>>
>
>
>
> --
> Best regards,
>
> Ivy Tang
>
>
>
>


-- 
Best Regards,
Chen Xinli

Mime
View raw message