accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-3609) BalanceInPresenceOfOfflineTableIT seems to be hanging
Date Fri, 20 Feb 2015 22:20:13 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329665#comment-14329665
] 

Josh Elser commented on ACCUMULO-3609:
--------------------------------------

Seeing lots of in ShellServerIT's output

{noformat}
2015-02-20 17:17:47,907 [impl.ThriftTransportPool] WARN : Thread "Thread-41" stuck on IO 
to hostname:55675 (0) for at least 120146 ms
{noformat}

Stuck in

{noformat}
"Thread-41" prio=5 tid=0x00007fda65017000 nid=0x4517 runnable [0x000000010bf93000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00000007c0022eb8> (a java.io.BufferedInputStream)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
        at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.readAll(ThriftTransportPool.java:275)
        at org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:601)
        at org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:470)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
        at org.apache.accumulo.core.master.thrift.FateService$Client.recv_waitForFateOperation(FateService.java:174)
        at org.apache.accumulo.core.master.thrift.FateService$Client.waitForFateOperation(FateService.java:159)
        at org.apache.accumulo.core.client.impl.TableOperationsImpl.waitForFateOperation(TableOperationsImpl.java:267)
        at org.apache.accumulo.core.client.impl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:309)
        at org.apache.accumulo.core.client.impl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:295)
        at org.apache.accumulo.core.client.impl.TableOperationsImpl.doTableFateOperation(TableOperationsImpl.java:1594)
        at org.apache.accumulo.core.client.impl.TableOperationsImpl.delete(TableOperationsImpl.java:681)
        at org.apache.accumulo.core.util.shell.commands.DeleteTableCommand.doTableOp(DeleteTableCommand.java:54)
        at org.apache.accumulo.core.util.shell.commands.TableOperation.execute(TableOperation.java:90)
        at org.apache.accumulo.core.util.shell.commands.DeleteTableCommand.execute(DeleteTableCommand.java:44)
        at org.apache.accumulo.core.util.shell.Shell.execCommand(Shell.java:747)
        at org.apache.accumulo.test.ShellServerIT$TestShell.exec(ShellServerIT.java:150)
        at org.apache.accumulo.test.ShellServerIT$TestShell.exec(ShellServerIT.java:159)
        at org.apache.accumulo.test.ShellServerIT$TestShell.exec(ShellServerIT.java:155)
        at org.apache.accumulo.test.ShellServerIT.exporttableImporttable(ShellServerIT.java:301)
{noformat}

Master is waiting

{noformat}
"ClientPool 43" daemon prio=5 tid=0x00007f9bce000000 nid=0x9227 in Object.wait() [0x000000011f07c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.accumulo.fate.ZooStore.waitForStatusChange(ZooStore.java:356)
        - locked <0x000000010c0b9b20> (a org.apache.accumulo.fate.ZooStore)
        at org.apache.accumulo.fate.AgeOffStore.waitForStatusChange(AgeOffStore.java:210)
        at org.apache.accumulo.fate.Fate.waitForCompletion(Fate.java:191)
        at org.apache.accumulo.master.FateServiceHandler.waitForFateOperation(FateServiceHandler.java:484)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.accumulo.trace.instrument.thrift.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
        at org.apache.accumulo.server.util.RpcWrapper$1.invoke(RpcWrapper.java:47)
        at com.sun.proxy.$Proxy10.waitForFateOperation(Unknown Source)
        at org.apache.accumulo.core.master.thrift.FateService$Processor$waitForFateOperation.getResult(FateService.java:478)
        at org.apache.accumulo.core.master.thrift.FateService$Processor$waitForFateOperation.getResult(FateService.java:462)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:168)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
        at org.apache.accumulo.server.util.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:77)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

> BalanceInPresenceOfOfflineTableIT seems to be hanging
> -----------------------------------------------------
>
>                 Key: ACCUMULO-3609
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3609
>             Project: Accumulo
>          Issue Type: Bug
>          Components: test
>            Reporter: Josh Elser
>             Fix For: 1.7.0
>
>
> I noticed today that BalanceInPresenceOfOfflineTableIT seems to have started failing
for me.
> I see the following repeated ad infinitum in the tserver logs
> {noformat}
> 2015-02-20 15:52:25,830 [util.FileUtil] WARN : Failed to find mid point using indexes,
falling back to data files which is slower. No entries between row_0000040499 and row_00000405
for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> 2015-02-20 15:52:25,833 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find
mid point, no entries between row_0000040499 and row_00000405 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> 2015-02-20 15:52:25,840 [util.FileUtil] WARN : Failed to find mid point using indexes,
falling back to data files which is slower. No entries between row_0000100199 and row_00001002
for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
> 2015-02-20 15:52:25,841 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find
mid point, no entries between row_0000100199 and row_00001002 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
> 2015-02-20 15:52:25,850 [util.FileUtil] WARN : Failed to find mid point using indexes,
falling back to data files which is slower. No entries between row_0000141299 and row_00001413
for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,855 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find
mid point, no entries between row_0000141299 and row_00001413 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,862 [util.FileUtil] WARN : Failed to find mid point using indexes,
falling back to data files which is slower. No entries between row_0000179599 and row_00001796
for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,865 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find
mid point, no entries between row_0000179599 and row_00001796 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,878 [util.FileUtil] WARN : Failed to find mid point using indexes,
falling back to data files which is slower. No entries between row_0000040499 and row_00000405
for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> 2015-02-20 15:52:25,881 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find
mid point, no entries between row_0000040499 and row_00000405 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> 2015-02-20 15:52:25,889 [util.FileUtil] WARN : Failed to find mid point using indexes,
falling back to data files which is slower. No entries between row_0000100199 and row_00001002
for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
> 2015-02-20 15:52:25,891 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find
mid point, no entries between row_0000100199 and row_00001002 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
> 2015-02-20 15:52:25,897 [util.FileUtil] WARN : Failed to find mid point using indexes,
falling back to data files which is slower. No entries between row_0000141299 and row_00001413
for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,900 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find
mid point, no entries between row_0000141299 and row_00001413 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,904 [util.FileUtil] WARN : Failed to find mid point using indexes,
falling back to data files which is slower. No entries between row_0000179599 and row_00001796
for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,906 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find
mid point, no entries between row_0000179599 and row_00001796 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,920 [util.FileUtil] WARN : Failed to find mid point using indexes,
falling back to data files which is slower. No entries between row_0000040499 and row_00000405
for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> 2015-02-20 15:52:25,923 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find
mid point, no entries between row_0000040499 and row_00000405 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> {noformat}
> It does seem to eventually finish (occasionally), so I'm not really sure what's happening
yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message