hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From st...@duboce.net
Subject Re: Review Request: Fix periodic major compaction (HBASE-2990) and expiration check (HBASE-3083)
Date Thu, 11 Nov 2010 16:04:44 GMT


> On 2010-11-11 06:23:59, Kannan Muthukkaruppan wrote:
> > Last major should be tracked on per region+CF basis, rather than just a per region
basis, right?
> > 
> > Nicolas's & Amit's recent changes to compaction algorithm already makes it such
that different column families within the same region could end up having their major compactions
done at different times. [A minor which includes all files will implicitly get promoted to
a major].
> > 
> > Is there some boolean state/flag in the HFile's meta data already that tells us
 if the file was the result of a major compaction or not?
> > 
> > If so, isn't it sufficient to use the timestamp of the file which has the "major
compaction" tag to determine when a particular region+CF was last major compacted?
> >

Kannan, yes, there is a flag on end of HFile already which says if file is result of a major
compaction and yes, it would make more sense to do this on a per-Store level (though up to
this its been at the Region level).  And thanks, your note reminds me that all is different
since the new compaction code went in.   I'm thinking the latter probably even 'fixes' the
original issue.  Lets see.  Meantime, I think we should move this issue out of 0.90.  We can
then take the time with the implementation.


- stack


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1089/#review1906
-----------------------------------------------------------


On 2010-11-10 17:04:39, Jonathan Gray wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/1089/
> -----------------------------------------------------------
> 
> (Updated 2010-11-10 17:04:39)
> 
> 
> Review request for hbase, stack and khemani.
> 
> 
> Summary
> -------
> 
> This is a somewhat misguided attempt.  It's not done but it shows the fairly simple change
to the actual major compaction code.
> 
> The hard part is:
> 
> +    long lastMajor = region.getRegionInfo().getRegionData().getLastMajor();
> 
> And the question is where to persist that.
> 
> This patch adds a new class called HRegionData into HRegionInfo that contains any number
of key-value pairs of data that get persisted with the HRI.  Not really sure how I ended up
there but this data seemed like an odd-man-out so adding another field seemed weird.  We also
need some kind of versioning in HRI so we can add stuff w/o migrating.  There's some versioned
stuff in HRData.
> 
> Just looking for some feedback / ideas.
> 
> 
> This addresses bugs HBASE-2990 and HBASE-3083.
>     http://issues.apache.org/jira/browse/HBASE-2990
>     http://issues.apache.org/jira/browse/HBASE-3083
> 
> 
> Diffs
> -----
> 
>   trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1033780 
>   trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1033780 
> 
> Diff: http://review.cloudera.org/r/1089/diff
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Jonathan
> 
>


Mime
View raw message