hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2468) Improvements to prewarm META cache on clients
Date Sat, 22 May 2010 22:45:17 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870332#action_12870332

Andrew Purtell commented on HBASE-2468:

bq. A single table could be made up of many thousands even hundreds of thousands of regions?
It could be significant. Even if 1000 regions it could add a huge overhead to instantiating
an HTable a user may only use for a single Get.

Yes but this issue talks about a more likely use case, where someone runs a MR job over a
table and causes a storm of individual Gets to META. 

bq. I love these features and have wanted them for a while. Just don't think we should change
default HTable behavior significantly.

Then for the general case that's fine but TIF and TOF should set some flag to true which causes
the prefetch on open to kick in. Reasonable?

> Improvements to prewarm META cache on clients
> ---------------------------------------------
>                 Key: HBASE-2468
>                 URL: https://issues.apache.org/jira/browse/HBASE-2468
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: client
>            Reporter: Todd Lipcon
>            Assignee: Mingjie Lai
>             Fix For: 0.21.0
>         Attachments: HBASE-2468-trunk.patch
> A couple different use cases cause storms of reads to META during startup. For example,
a large MR job will cause each map task to hit meta since it starts with an empty cache.
> A couple possible improvements have been proposed:
>  - MR jobs could ship a copy of META for the table in the DistributedCache
>  - Clients could prewarm cache by doing a large scan of all the meta for the table instead
of random reads for each miss
>  - Each miss could fetch ahead some number of rows in META

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message