hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5913) Speed up the full scan of META
Date Wed, 02 May 2012 17:08:50 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266714#comment-13266714

Hadoop QA commented on HBASE-5913:

-1 overall.  Here are the results of testing the latest attachment 
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit

     -1 core tests.  The patch failed these unit tests:

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1725//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1725//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1725//console

This message is automatically generated.
> Speed up the full scan of META
> ------------------------------
>                 Key: HBASE-5913
>                 URL: https://issues.apache.org/jira/browse/HBASE-5913
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>         Attachments: 5913-v2.txt, HBASE-5913.patch
> In the master, we will do the full scan of META in some situations
> for example,
> 1.master start up
> 2.CatalogJanitor do the full scan per 5 mins
> 3.ServerShutdownHandler, getServerUserRegions for dead server.
> For the online applications, we should try the best to reduce the process time of ServerShutdownHandler
in the situation 3. 
> However, we found MetaReader#getServerUserRegions take 14mins for 10w regions in our
production environment.
> And it is caused by two reasons:
> The first, we don't use cache and get one row per next() when fully scan .META.
> The second, "hbase.ipc.client.tcpnodelay" is false as default, and in our environment
it take 40ms for per next() (It is related to the length of row in the .META. , if someone
also found, could try to set it true)
> For this issue, I think we could set the caching when do the full scan of META

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message