hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HBASE-830) Debugging HCM.locateRegionInMeta is painful
Date Fri, 15 Aug 2008 18:09:44 GMT

     [ https://issues.apache.org/jira/browse/HBASE-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

stack updated HBASE-830:

    Attachment: 830-v2-shortertimeouts.patch

I think that the pause should be shorter still. Rather than 8 seconds, it should be 2 seconds.
Attached file changes the pause to 2 seconds and removes the 64 off the end of RETRY_BACKOFF.

At 2 seconds all is 'snappier' - creating, disablng, etc. - and after ten retries we're at
about 2 minutes which is long-enough in my opinion and is < TaskTracker timeout of ten
minutes so we'll see error in MR logs rather than child killed messages.

At 8 seconds, with RETRY_BACKOFF as it was, we're waiting > 17 minutes (if I did my math

This patch passes all unit tests. Also tried it in a big MR job. More logging if client DEBUG
is enabled but at least now you have a better clue whats going on.

> Debugging HCM.locateRegionInMeta is painful
> -------------------------------------------
>                 Key: HBASE-830
>                 URL: https://issues.apache.org/jira/browse/HBASE-830
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 0.2.0
>            Reporter: Jean-Daniel Cryans
>            Priority: Minor
>             Fix For: 0.2.1, 0.3.0
>         Attachments: 830-v2-shortertimeouts.patch, hbase-826-v1.patch
> I've been debugging a case where a bunch of reduces were hanging for no apparent reason
and then get killed because they did not do anything for 600 seconds. I figured that it's
because we are stuck in a very long waiting time due to retry backoffs. 
> {code}
> public static int RETRY_BACKOFF[] = { 1, 1, 1, 1, 2, 4, 8, 16, 32, 64 };
> {code}
> That means we wait 10 sec, 10 sec, 10, 10, ... then 640 sec. That's a long time, do we
really need that much time to finally be warned that there's a bug in HBase? 
> Also, the places where we get this:
> {code}
> LOG.debug("reloading table servers because: " + t.getMessage());
> {code}
> should be more verbose. I my logs these are caused by a table not found but the only
thing I see is "reloading table servers because: tableName".

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message