hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8764) Some MasterMonitorCallable should retry
Date Fri, 26 Jul 2013 16:01:54 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720911#comment-13720911
] 

stack commented on HBASE-8764:
------------------------------

Here is commit message for this patch updated:
{code}
Add retrying of Master operations; helps when running hbase-it and chaos monkey kills
Master or Master is not yet up ready to take on operations.

Refactors ServerCallable.  ServerCallable had a public call() method and then beside
it a withRetries() and also a withoutRetries().  Confusing.  Also the rpc retrying
with its specific handling of server exception returns was not reusable buried down
in ServerCallable guts.

This patch moves the rpc retrying code out of ServerCallable into a utility
RpcRetryingCaller class (A 'Caller' runs the 'Callable'). ServerCallable shrinks,
implements a new RetryingCallable Interface, and becomes RegionServerCallable, a class
that is just about Calling -- no rpc nor retries, a Callable class with added details
on where the Callable is to be applied (table name and row), -- etc.

This pattern is then applied to Master operations.  Master operations were not retried
previously.  The Master operation Callables are now like RegionServerCallable (though
they need to carry way less detail), implement RetryingCallable, and are passed to
RpcRetryingCaller so they are retried.  Changed some exceptions so they now implement
DoNotRetryException because not all master operations should be retried.


M hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
  Move to new style making calls (create a Callable and a Caller to run the op)

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
  Create a caller on initialization and reuse it doing calls in new style

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
  Javadoc.

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
  Use new style making calls where we first create the Callable and then a
  Caller of the right type to make the call.  Remove the execute methods.
  Have all directly use executeCallable instead.  HBaseAdmin manipulates two
  'admin' rpc Services -- the 'Monitor' service and the 'Admin' service.
  Amend the Callables that were declared as inner classes here so they both
  share a base class -- MasterCallable -- that does shared functionality
  and that implements RetryingCallable.  Also adds support for closing the
  service needed by the backing AsyncProcess.

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionWrapper.java
  Remove deprecated methods.

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
  Use new model calling.

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java
  Implement RegionServerCallable instead of ServerCallable.

A hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java
  ServerCallable but stripped of retrying.

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedException.java
  Javadoc

A hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetryingCallable.java
  Interface implemented by Callables that we can retry.

A hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCaller.java
  This is the retrying functionality broken out of what was ServerCallable.

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
  Implement RegionServerCallable.

A hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCaller.java
  A subclass of RpcRetryingCaller that supplies the type returned when scanning.

D hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
  Replaced by RpcRetryingCaller and RegionServerCallable.

M hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/DoNotRetryIOException.java
  Comment

M hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/HBaseSnapshotException.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/InvalidFamilyOperationException.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/TableExistsException.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/LockTimeoutException.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/TableNotDisabledException.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/TableNotEnabledException.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/TableNotFoundException.java
  Implement DoNotRetryException

M hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionCoprocessorRpcChannel.java
M hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
  New model.

M hbase-common/src/main/java/org/apache/hadoop/hbase/util/Threads.java
  Javadoc.  Perpeturate interrupt.

M hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
  Javadoc.

A hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHBaseAdminNoCluster.java
  Unit test.

Converted the rest of the tests to new model
{code}
                
> Some MasterMonitorCallable should retry
> ---------------------------------------
>
>                 Key: HBASE-8764
>                 URL: https://issues.apache.org/jira/browse/HBASE-8764
>             Project: HBase
>          Issue Type: Bug
>          Components: IPC/RPC
>    Affects Versions: 0.95.1
>            Reporter: Elliott Clark
>            Assignee: stack
>             Fix For: 0.95.2
>
>         Attachments: 8764.txt, 8764v2.txt, 8764v3.txt, 8764v4.txt, 8796v5.txt, 8796v7.txt
>
>
> Calls in the admin that only get status should re-try.
> got a call stack like:
> {code}
> org.apache.hadoop.hbase.exceptions.PleaseHoldException: org.apache.hadoop.hbase.exceptions.PleaseHoldException:
Master is initializing
> 	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2266)
> 	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1610)
> 	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1646)
> 	at org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:20930)
> 	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
> 	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> 	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
> 	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
> 	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:230)
> 	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2705)
> 	at org.apache.hadoop.hbase.client.HBaseAdmin.execute(HBaseAdmin.java:2674)
> 	at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:524)
> 	at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:417)
> 	at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:349)
> 	at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.createSchema(IntegrationTestBigLinkedList.java:437)
> 	at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.runGenerator(IntegrationTestBigLinkedList.java:471)
> 	at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.run(IntegrationTestBigLinkedList.java:505)
> 	at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.runGenerator(IntegrationTestBigLinkedList.java:698)
> 	at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.run(IntegrationTestBigLinkedList.java:748)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithChaosMonkey.testContinuousIngest(IntegrationTestBigLinkedListWithChaosMonkey.java:80)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> 	at org.junit.runners.Suite.runChild(Suite.java:127)
> 	at org.junit.runners.Suite.runChild(Suite.java:26)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> 	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> 	at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
> 	at org.junit.runner.JUnitCore.run(JUnitCore.java:117)
> 	at org.apache.hadoop.hbase.IntegrationTestsDriver.doWork(IntegrationTestsDriver.java:111)
> 	at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:108)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> 	at org.apache.hadoop.hbase.IntegrationTestsDriver.main(IntegrationTestsDriver.java:47)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message