hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lars hofhansl <la...@apache.org>
Subject Re: Is it feasible to delete qualified tombstones during minor compaction?
Date Thu, 14 Mar 2013 18:05:31 GMT
That is true. The memstore is special, though, in the sense that it will contain only stuff
newer than in any storefile (new in the sense of when it was inserted rather than it's timestamp,
and so I say in another thread there's still weirdness).

There is no such separation of new vs old between the store files until we get your striped
compactions.


Back to the original idea, though :)  The bloomfilters are per HFile, right? (not sure now)
If so, one would still have to check each file.


-- Lars



________________________________
 From: Sergey Shelukhin <sergey@hortonworks.com>
To: dev@hbase.apache.org; lars hofhansl <larsh@apache.org> 
Sent: Thursday, March 14, 2013 10:46 AM
Subject: Re: Is it feasible to delete qualified tombstones during minor compaction?
 

Actually, major compaction still doesn't look at memstore so the window for reappearing puts
is still there, it's just small. I am +1 on the OP idea :)


On Thu, Mar 14, 2013 at 10:00 AM, lars hofhansl <larsh@apache.org> wrote:

Unless you can guarantee that all KVs affected by a delete marker have been deleted you cannot
remove the delete marker. That's a fact :)
>That means in order to remove a delete marker you have to look everywhere where a potentially
affected KV could reside. Currently that is everywhere and hence only a major compaction can
do that.
>
>Sergey is working on striped compactions where will be able (hopefully) to limit where
we have to look for affected to something smaller than "everything".
>
>-- Lars
>
>
>
>________________________________
> From: Ted Yu <yuzhihong@gmail.com>
>To: dev@hbase.apache.org
>Sent: Thursday, March 14, 2013 9:41 AM
>Subject: Re: Is it feasible to delete qualified tombstones during minor compaction?
>
>
>That is correct.
>
>On Thu, Mar 14, 2013 at 9:28 AM, ramkrishna vasudevan <
>ramkrishna.s.vasudevan@gmail.com> wrote:
>
>> So seeing the final patch it means that whatever Liang has asked for is not
>> available right?
>>
>> Regards
>> Ram
>>
>> On Thu, Mar 14, 2013 at 9:55 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>> > The issue was HBASE-7902
>> >
>> > The final patch was just mostly renaming:
>> >
>> > -          Collections.singletonList(memstoreScanner),
>> > ScanType.MINOR_COMPACT,
>> > +          Collections.singletonList(memstoreScanner),
>> > ScanType.COMPACT_RETAIN_DELETES,
>> >
>> > Cheers
>> >
>> > On Thu, Mar 14, 2013 at 9:22 AM, Anoop John <anoop.hbase@gmail.com>
>> wrote:
>> >
>> > > Hi
>> > >          I have seen some issue raised by Sergey regarding removing
>> > Delete
>> > > markers during minor compaction. Not remembering the issue id and
>> details
>> > > :(
>> > >
>> > > -Anoop-
>> > >
>> > >
>> > > On Thu, Mar 14, 2013 at 3:11 PM, 谢良 <xieliang@xiaomi.com> wrote:
>> > >
>> > > > Currently, all tombstones only get cleaned up with major
>> compactions. I
>> > > > guess the original
>> > > > design probably considered that we need to ensure whether there are
>> > other
>> > > > versions of
>> > > > the deleted key in other storefiles which are not included in minor
>> > > > compaction? If that's
>> > > > true, i have a thought, maybe we could delete a bunch of tombstones
>> per
>> > > > bloom filter, in
>> > > > other words, we can try to query the deleted key in other
>> storefiles's
>> > > > bloom filter info to determine
>> > > > whether we can remove it safely or not.
>> > > > I am not familiar with compaction&bloomfilter internal, it's just
a
>> raw
>> > > > thought right now:)
>> > > >
>> > > > Regards,
>> > > > Liang
>> > >
>> >
>>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message