hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingjie Lai" <mjla...@gmail.com>
Subject Re: Review Request: HBASE-2468: Improvements to prewarm META cache on clients.
Date Wed, 09 Jun 2010 22:39:37 GMT


> On 2010-06-07 14:23:42, stack wrote:
> > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 96
> > <http://review.hbase.org/r/98/diff/5/?file=944#file944line96>
> >
> >     getRowOrBefore is an expensive call.  Are we sure we are not calling this too
often?

I agree it is an expensive call. 

However I don't think it would bring any performance penalty for existing and potential use
cases:
Use case 1 -- existing MetaScanner users: since this method is newly added, existing users
won't be affected; 
Use case 2 -- hbase clients when locating a region :  
1) if prefetch is on, it calls this MetaScanner with [table + row combination], which calls
getRowOrBefore() to get current region info, then number of following regions from meta. After
that, the client can get the region info directly from cache. 
2) if prefetch is disabled (current behavior), it eventually calls similar method getClosestRowBefore()
to get desired region. 

So no matter prefetch is on or not, getRowOrBefore(or getClosestRowBefore) eventually is called.
The only difference is whether to scan following regions from meta or not. 

For future MetaScanner users which scan from one region with desired use table row, it has
to take the effort since it is the expected behavior. 


- Mingjie


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/98/#review144
-----------------------------------------------------------


On 2010-06-02 01:13:42, Mingjie Lai wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.hbase.org/r/98/
> -----------------------------------------------------------
> 
> (Updated 2010-06-02 01:13:42)
> 
> 
> Review request for hbase.
> 
> 
> Summary
> -------
> 
> HBASE-2468: Improvements to prewarm META cache on clients.
> 
> Changes:
> 1. Add new HTable methods which support region info de/serialation, and region cache
prewarm: 
> - void serializeRegionInfo(): clients could perform a large scan for all the meta for
the table, serialize the meta to a file. MR job can ship a copy of the meta for the table
in the DistributedCache
> - Map<HRegionInfo, HServerAddress> deserializeRegionInfo(): MR job can deserialize
the region info from the DistributedCache 
> - prewarmRegionCache(Map<HRegionInfo, HServerAddress> regionMap): MR job can prewarm
local region cache by the deserialized region info.
> 
> 2. For each client, each region cache read-miss could trigger read-ahead some number
of rows in META. This option could be turned on and off for one particular table. 
> 
> 
> This addresses bug HBASE-2468.
>     http://issues.apache.org/jira/browse/HBASE-2468
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/hadoop/hbase/client/HConnection.java 853164d 
>   src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java ed18092 
>   src/main/java/org/apache/hadoop/hbase/client/HTable.java 7ec95cb 
>   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java d3a0c07 
>   src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 95e494a 
> 
> Diff: http://review.hbase.org/r/98/diff
> 
> 
> Testing
> -------
> 
> Unit tests passed locally for me. 
> 
> 
> Thanks,
> 
> Mingjie
> 
>


Mime
View raw message