accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-4405) GC collection cycle stuck on waitForFlush RPC to Master
Date Mon, 15 Aug 2016 16:26:20 GMT


Josh Elser commented on ACCUMULO-4405:

bq. I also have to question why the RPC itself from the GC did not time out. I would have
suspected that it would be using the TTimeoutTransport (but perhaps it is also configured
to have an extremely long timeout).

Apparently, RPCs to the Master intentionally do not use the TTimeoutTransport:

    try {
      // Master requests can take a long time: don't ever time out
      MasterClientService.Client client = ThriftUtil.getClientNoTimeout(new MasterClientService.Client.Factory(),
master, context);
      return client;
    } catch (TTransportException tte) {
      Throwable cause = tte.getCause();
      if (null != cause && cause instanceof UnknownHostException) {
        // do not expect to recover from this
        throw new RuntimeException(tte);
      log.debug("Failed to connect to master=" + master + ", will retry... ", tte);
      return null;

> GC collection cycle stuck on waitForFlush RPC to Master
> -------------------------------------------------------
>                 Key: ACCUMULO-4405
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: gc, master
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>             Fix For: 1.8.1
> While testing out 1.8.0rc1, all of the TabletServers had crashed due to an OOME, I believe,
because I temporarily ran out of HDFS space because HDFS trash was enabled (trash could not
be cleaned up fast enough for Accumulo generating more trash). I came back to the system after
restarting the TabletServers and found that the GC had not run a new cycle after restarting
the TabletServers. In a jstack of the GC, I saw:
> {noformat}
> "gc" #13 prio=5 os_prio=0 tid=0x00000000021f3800 nid=0x4dd5 runnable [0x00007f6f1ebc0000]
>    java.lang.Thread.State: RUNNABLE
>         at Method)
>         at
>         at
>         at
>         at
>         at
>         at
>         - locked <0x00000000f5b4b750> (a
>         at
>         at org.apache.thrift.transport.TTransport.readAll(
>         at org.apache.thrift.transport.TFramedTransport.readFrame(
>         at
>         at org.apache.thrift.transport.TTransport.readAll(
>         at org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.readAll(
>         at org.apache.thrift.protocol.TCompactProtocol.readByte(
>         at org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(
>         at org.apache.thrift.TServiceClient.receiveBase(
>         at org.apache.accumulo.core.master.thrift.MasterClientService$Client.recv_waitForFlush(
>         at org.apache.accumulo.core.master.thrift.MasterClientService$Client.waitForFlush(
>         at org.apache.accumulo.core.client.impl.TableOperationsImpl._flush(
>         at org.apache.accumulo.core.client.impl.TableOperationsImpl.compact(
>         at org.apache.accumulo.core.client.impl.TableOperationsImpl.compact(
>         at org.apache.accumulo.core.client.impl.TableOperationsImpl.compact(
>         at
>         at org.apache.accumulo.gc.SimpleGarbageCollector.main(
>         at org.apache.accumulo.gc.GCExecutable.execute(
>         at org.apache.accumulo.start.Main$
>         at
>    Locked ownable synchronizers:
>         - None
> {noformat}
> The Master was also stuck with an active thread/RPC:
> {noformat}
> "ClientPool 23257" #45412 daemon prio=5 os_prio=0 tid=0x00000000049cc000 nid=0x6401 waiting
on condition [0x00007f8462d94000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.accumulo.core.client.impl.ThriftScanner.pause(
>         at org.apache.accumulo.core.client.impl.ThriftScanner.scan(
>         at org.apache.accumulo.core.client.impl.ScannerIterator$
>         at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(
>         at org.apache.accumulo.core.client.IsolatedScanner$RowBufferingIterator.readRow(
>         at org.apache.accumulo.core.client.IsolatedScanner$RowBufferingIterator.<init>(
>         at org.apache.accumulo.core.client.IsolatedScanner.iterator(
>         at org.apache.accumulo.core.client.RowIterator.<init>(
>         at org.apache.accumulo.master.MasterClientServiceHandler.waitForFlush(
>         at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>         at java.lang.reflect.Method.invoke(
>         at org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(
>         at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(
>         at com.sun.proxy.$Proxy18.waitForFlush(Unknown Source)
>         at org.apache.accumulo.core.master.thrift.MasterClientService$Processor$waitForFlush.getResult(
>         at org.apache.accumulo.core.master.thrift.MasterClientService$Processor$waitForFlush.getResult(
>         at org.apache.thrift.ProcessFunction.process(
>         at org.apache.thrift.TBaseProcessor.process(
>         at org.apache.accumulo.server.rpc.TimedProcessor.process(
>         at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(
>         at org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(
>         at
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
>         at java.util.concurrent.ThreadPoolExecutor$
>         at
>         at
>    Locked ownable synchronizers:
>         - <0x00000006caeb94d8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> It would appear that in the {{waitForFlush}} implementation in MasterClientServiceHandler,
we construct a scanner (with the default timeout) and wrap an IsolatedScanner around that.
We do not set the timeout on the scanner which ends up using a default value of {{Long.MAX_VALUE}}
(essentially, never time out).
> The problem is that in ThriftScanner, the backoff policy for retrying the failed RPC's
starts at 100ms and then, for each subsequent retry loop, doubles the previous value (+/-
10%). In the face of a prolonged inability to scan a Tablet, the value we sleep grows: 100,
~200, ~400, ~800, ~1600ms, etc.
> Because the scan will not time out until Long.MAX_VALUE, we've essentially create an
RPC which never times out. We should apply an upper-limit, on the order of seconds, to the
maximum amount of time that ThriftScanner will sleep before retrying (without needing to change
the overall timeout property)

This message was sent by Atlassian JIRA

View raw message