Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 3 Jul 2012 18:29:20 +0000 (UTC)
From: "Jean-Daniel Cryans (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <1349200071.532.1341340160228.JavaMail.jiratomcat@issues-vm>
In-Reply-To: <1943104904.407.1341273177446.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Commented] (HBASE-6309) [MTTR] Do NN operations outside of
 the ZK EventThread in SplitLogManager
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-6309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405969#comment-13405969 ] 

Jean-Daniel Cryans commented on HBASE-6309:
-------------------------------------------

Hey Ram, yeah it's pretty much the same although I like my description of the problem better :)

IMO we should move everything that talks to ZK and NN out of that path.
                
> [MTTR] Do NN operations outside of the ZK EventThread in SplitLogManager
> ------------------------------------------------------------------------
>
>                 Key: HBASE-6309
>                 URL: https://issues.apache.org/jira/browse/HBASE-6309
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 0.92.1, 0.94.0, 0.96.0
>            Reporter: Jean-Daniel Cryans
>            Priority: Critical
>             Fix For: 0.96.0
>
>
> We found this issue during the leap second cataclysm which prompted a distributed splitting of all our logs.
> I saw that none of the RS were splitting after some time while the master was showing that it wasn't even 30% done. jstack'ing I saw this:
> {noformat}
> "main-EventThread" daemon prio=10 tid=0x00007f6ce46d8800 nid=0x5376 in
> Object.wait() [0x00007f6ce2ecb000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1093)
>         - locked <0x00000005fdd661a0> (a org.apache.hadoop.ipc.Client$Call)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
>         at $Proxy9.rename(Unknown Source)
>         at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>         at $Proxy9.rename(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:759)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:253)
>         at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.moveRecoveredEditsFromTemp(HLogSplitter.java:553)
>         at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.moveRecoveredEditsFromTemp(HLogSplitter.java:519)
>         at org.apache.hadoop.hbase.master.SplitLogManager$1.finish(SplitLogManager.java:138)
>         at org.apache.hadoop.hbase.master.SplitLogManager.getDataSetWatchSuccess(SplitLogManager.java:431)
>         at org.apache.hadoop.hbase.master.SplitLogManager.access$1200(SplitLogManager.java:95)
>         at org.apache.hadoop.hbase.master.SplitLogManager$GetDataAsyncCallback.processResult(SplitLogManager.java:1011)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:571)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> {noformat}
> We are effectively bottlenecking on doing NN operations and whatever else is happening in GetDataAsyncCallback. It was so bad that on our 100 offline cluster it took a few hours for the master to process all the incoming ZK events while the actual splitting took a fraction of that time.
> I'm marking this as critical and against 0.96 but depending on how involved the fix is we might want to backport.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira