hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray" <jg...@apache.org>
Subject Re: Review Request: HBASE-2468: Improvements to prewarm META cache on clients.
Date Fri, 11 Jun 2010 22:38:56 GMT


> On 2010-06-10 08:53:41, Jonathan Gray wrote:
> > src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 1079
> > <http://review.hbase.org/r/98/diff/5/?file=943#file943line1079>
> >
> >     I guess these are static because of how HTables all share a single HCM per conf.
 The setting of prefetching is set at the HCM level not the HTable level, however clients
are usually not exposed to HCM and only deal with HTable.
> >     
> >     We should probably make it clear in the javadoc for these methods that they
apply to all HTable instances, though that may be clear from being static.
> >     
> >     Maybe since these are more advanced calls, they shouldn't be in HTable?  If
we provide proper documentation, it should be easy enough for a user to grab the HCM and apply
the config at that level?
> 
> Mingjie Lai wrote:
>     > I guess these are static because of how HTables all share a single HCM per conf...
>     Yes. 
>     
>     > Maybe since these are more advanced calls, they shouldn't be in HTable?
>     Two alternatives:
>     1) HCM: as jgray said, ``however clients are usually not exposed to HCM and only
deal with HTable.''
>     2) HBaseAdmin: it is a more reasonable design choice since these operation are at
HCM level. 
>     3) or, make it a configuration. It would be one global configuration applied to all
tables, and cannot be changed dynamically. 
>     
>     I like 2) better, but not really sure whether we want to expose it there or not.

>     
>     What do you think?

Adding it to HBaseAdmin could make sense.  This one is a bit of an odd one because it's a
client-side configuration parameter done at the per-client-jvm level.  Typically we have per-query
or per-htable-instance configs.  HBaseAdmin is generally made up of remote administration
commands not local client config.

If we provide sufficient javadoc (including as a class comment on HTable) it doesn't matter
so much where we put it.  Since it's distinct from what's currently in HTable and HBaseAdmin,
maybe it does make sense as a static in HCM?


- Jonathan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/98/#review176
-----------------------------------------------------------


On 2010-06-09 15:50:59, Mingjie Lai wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.hbase.org/r/98/
> -----------------------------------------------------------
> 
> (Updated 2010-06-09 15:50:59)
> 
> 
> Review request for hbase, Todd Lipcon and stack.
> 
> 
> Summary
> -------
> 
> HBASE-2468: Improvements to prewarm META cache on clients.
> 
> Changes:
> 1. Add new HTable methods which support region info de/serialation, and region cache
prewarm: 
> - void serializeRegionInfo(): clients could perform a large scan for all the meta for
the table, serialize the meta to a file. MR job can ship a copy of the meta for the table
in the DistributedCache
> - Map<HRegionInfo, HServerAddress> deserializeRegionInfo(): MR job can deserialize
the region info from the DistributedCache 
> - prewarmRegionCache(Map<HRegionInfo, HServerAddress> regionMap): MR job can prewarm
local region cache by the deserialized region info.
> 
> 2. For each client, each region cache read-miss could trigger read-ahead some number
of rows in META. This option could be turned on and off for one particular table. 
> 
> 
> This addresses bug HBASE-2468.
>     http://issues.apache.org/jira/browse/HBASE-2468
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/hadoop/hbase/client/HConnection.java 853164d 
>   src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java ed18092 
>   src/main/java/org/apache/hadoop/hbase/client/HTable.java 7ec95cb 
>   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java d3a0c07 
>   src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 95e494a 
> 
> Diff: http://review.hbase.org/r/98/diff
> 
> 
> Testing
> -------
> 
> Unit tests passed locally for me. 
> 
> 
> Thanks,
> 
> Mingjie
> 
>


Mime
View raw message