accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-4063) Thrift server should not accept requests until lock is initially acquired
Date Tue, 19 Jan 2016 17:18:40 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15107032#comment-15107032
] 

Josh Elser commented on ACCUMULO-4063:
--------------------------------------

Here's a snippet on 1.7:

{noformat}
2016-01-15 21:55:18,188 [tserver.TabletServer] WARN : Got getScans message from master before
lock acquired, ignoring...
2016-01-15 21:55:18,191 [tserver.TabletServer$ThriftClientHandler] ERROR: Lock not acquired
java.lang.RuntimeException: Lock not acquired
	at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1410)
	at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.getActiveScans(TabletServer.java:1632)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:39)
	at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:73)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2864)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2848)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
	at java.lang.Thread.run(Thread.java:745)
2016-01-15 21:55:18,200 [thrift.ProcessFunction] ERROR: Internal error processing getActiveScans
org.apache.thrift.TException: Lock not acquired
	at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:80)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2864)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2848)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
	at java.lang.Thread.run(Thread.java:745)
2016-01-15 21:55:20,932 [server.GarbageCollectionLogger] DEBUG: gc ParNew=0.04(+0.00) secs
ConcurrentMarkSweep=0.10(+0.03) secs freemem=1,534,000,088(-18,718,192) totalmem=1,593,180,160
2016-01-15 21:55:21,122 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:21,204 [tserver.TabletServer] INFO : Waiting for tablet server lock
2016-01-15 21:55:21,772 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56575 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:21,887 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56575 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:23,359 [tserver.TabletServer] WARN : Got getScans message from master before
lock acquired, ignoring...
2016-01-15 21:55:23,359 [tserver.TabletServer$ThriftClientHandler] ERROR: Lock not acquired
java.lang.RuntimeException: Lock not acquired
	at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1410)
	at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.getActiveScans(TabletServer.java:1632)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:39)
	at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:73)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2864)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2848)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
	at java.lang.Thread.run(Thread.java:745)
2016-01-15 21:55:23,362 [thrift.ProcessFunction] ERROR: Internal error processing getActiveScans
org.apache.thrift.TException: Lock not acquired
	at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:80)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2864)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2848)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
	at java.lang.Thread.run(Thread.java:745)
2016-01-15 21:55:24,743 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56575 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:26,227 [tserver.TabletServer] INFO : Waiting for tablet server lock
2016-01-15 21:55:26,922 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:28,460 [tserver.TabletServer] WARN : Got getScans message from master before
lock acquired, ignoring...
2016-01-15 21:55:28,461 [tserver.TabletServer$ThriftClientHandler] ERROR: Lock not acquired
java.lang.RuntimeException: Lock not acquired
	at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1410)
	at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.getActiveScans(TabletServer.java:1632)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:39)
	at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:73)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2864)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2848)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
	at java.lang.Thread.run(Thread.java:745)
2016-01-15 21:55:28,466 [thrift.ProcessFunction] ERROR: Internal error processing getActiveScans
org.apache.thrift.TException: Lock not acquired
	at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:80)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2864)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2848)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
	at java.lang.Thread.run(Thread.java:745)
2016-01-15 21:55:30,363 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56600 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:30,481 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56600 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:31,250 [tserver.TabletServer] INFO : Waiting for tablet server lock
2016-01-15 21:55:32,138 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,146 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,150 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,243 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,261 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,265 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,372 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,376 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,448 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,483 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,491 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,600 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,610 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,718 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,723 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,829 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,833 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,862 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,940 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:32,944 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,051 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,054 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,161 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,172 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,278 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,282 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,391 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,396 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,503 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,507 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,544 [tserver.TabletServer] WARN : Got getScans message from master before
lock acquired, ignoring...
2016-01-15 21:55:33,544 [tserver.TabletServer$ThriftClientHandler] ERROR: Lock not acquired
java.lang.RuntimeException: Lock not acquired
	at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1410)
	at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.getActiveScans(TabletServer.java:1632)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:39)
	at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:73)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2864)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2848)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
	at java.lang.Thread.run(Thread.java:745)
2016-01-15 21:55:33,546 [thrift.ProcessFunction] ERROR: Internal error processing getActiveScans
org.apache.thrift.TException: Lock not acquired
	at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:80)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
	at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
	at com.sun.proxy.$Proxy21.getActiveScans(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2864)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$getActiveScans.getResult(TabletClientService.java:2848)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
	at java.lang.Thread.run(Thread.java:745)
2016-01-15 21:55:33,614 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,618 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,667 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,724 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,731 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,837 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,842 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,948 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:33,952 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,058 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,063 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,170 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,173 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,285 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56570 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,289 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,397 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,401 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,510 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,514 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,620 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,624 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,732 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,738 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,844 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,852 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,957 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:34,960 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,067 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,070 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,177 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,181 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,275 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,287 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,293 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,400 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,404 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,428 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56600 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,516 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,524 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.01 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,547 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56600 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,632 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,635 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,744 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56570 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,748 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,855 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,858 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,965 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:35,969 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,076 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,080 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,185 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,189 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,288 [tserver.TabletServer] INFO : Waiting for tablet server lock
2016-01-15 21:55:36,295 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,298 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,404 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,408 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,515 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,519 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,630 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,634 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,742 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,746 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,851 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,855 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,982 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:36,986 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,092 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,098 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,206 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56570 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,210 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,318 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,322 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,429 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,432 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,539 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,543 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,649 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,653 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,759 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,762 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,869 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,873 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,979 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:37,983 [tserver.TabletServer] DEBUG: MultiScanSess 172.22.89.243:56559 0
entries in 0.00 secs (lookup_time:0.00 secs tablets:1 ranges:1) 
2016-01-15 21:55:41,301 [tserver.TabletServer] DEBUG: Obtained tablet server lock /accumulo/744e77bf-8e1e-4770-b8b7-0f1ab1b1d18a/tservers/os-d7-tubtbs-accumulo-1-ks2-1.novalocal:9997/zlock-0000000017
{noformat}

> Thrift server should not accept requests until lock is initially acquired
> -------------------------------------------------------------------------
>
>                 Key: ACCUMULO-4063
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4063
>             Project: Accumulo
>          Issue Type: Bug
>          Components: rpc, tserver
>    Affects Versions: 1.7.0
>            Reporter: Josh Elser
>
> Noticing the following:
> {noformat}
> 2015-11-19 22:42:47,130 [thrift.ProcessFunction] ERROR: Internal error processing flush
> org.apache.thrift.TException: Lock not acquired
>         at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:51)
>         at com.sun.proxy.$Proxy21.flush(Unknown Source)
>         at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$flush.getResult(TabletClientService.java:2671)
>         at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$flush.getResult(TabletClientService.java:2657)
>         at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>         at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
>         at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
>         at org.apache.accumulo.server.rpc.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:78)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>         at java.lang.Thread.run(Thread.java:745)
> 2015-11-19 22:42:47,402 [tserver.TabletServer$ThriftClientHandler] ERROR: org.apache.accumulo.tserver.session.MultiScanSession
cannot be cast to org.apache.accumulo.tserver.session.ScanSession
> java.lang.ClassCastException: org.apache.accumulo.tserver.session.MultiScanSession cannot
be cast to org.apache.accumulo.tserver.session.ScanSession
>         at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.closeScan(TabletServer.java:574)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
>         at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:47)
>         at com.sun.proxy.$Proxy21.closeScan(Unknown Source)
>         at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeScan.getResult(TabletClientService.java:2309)
>         at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeScan.getResult(TabletClientService.java:2295)
>         at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>         at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
>         at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
>         at org.apache.accumulo.server.rpc.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:78)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}
> a {{flush}} rpc came in and was denied, but then a {{closeScan}} came in and threw an
unrelated error (ignore the classcast for now).
> Are there _any_ RPCs that a tabletserver should answer before it holds its lock?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message