Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Message-ID: <31787422.31291274483301073.JavaMail.jira@thor>
Date: Fri, 21 May 2010 19:08:21 -0400 (EDT)
From: "Jonathan Gray (JIRA)" <jira@apache.org>
To: hbase-issues@hadoop.apache.org
Subject: [jira] Commented: (HBASE-2468) Improvements to prewarm META cache
 on clients
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870215#action_12870215 ] 

Jonathan Gray commented on HBASE-2468:
--------------------------------------

I'm unsure if I like the approach of aggressively pre-warming though you don't ask for it (prefetch on HTable instantiation, and auto look-ahead in locateRegionInMeta)

I think I would prefer adding an additional constructor w/ a warm boolean to determine whether META should be warmed for that table or not.

I don't quite understand the prefetching in locateRegionInMeta().  If you want to pre-cache meta, you should warm the HTable on construction.  Is this to try to update neighboring regions in case they got stale?  Since 10 seems arbitrary, if we do have this behavior it should probably be configurable.

Grabbing 9 extra rows could incur several additional block reads across multiple files.  It would also eliminate any chances of using blooms or other optimizations if the META retrievals were never single row reads.  So there is a cost to this being the default behavior.

Otherwise the patch itself looks good.  I'm in the process of changing MetaScanner a bit for some master stuff but should be okay.

Good stuff, this has been a long-desired feature!

> Improvements to prewarm META cache on clients
> ---------------------------------------------
>
>                 Key: HBASE-2468
>                 URL: https://issues.apache.org/jira/browse/HBASE-2468
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: client
>            Reporter: Todd Lipcon
>            Assignee: Mingjie Lai
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2468-trunk.patch
>
>
> A couple different use cases cause storms of reads to META during startup. For example, a large MR job will cause each map task to hit meta since it starts with an empty cache.
> A couple possible improvements have been proposed:
>  - MR jobs could ship a copy of META for the table in the DistributedCache
>  - Clients could prewarm cache by doing a large scan of all the meta for the table instead of random reads for each miss
>  - Each miss could fetch ahead some number of rows in META

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.