accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: GC and Monitor Unable to scan login as !SYSTEM
Date Tue, 28 Jul 2015 16:26:52 GMT
Accumulo has an internal user which it uses for RPCs between Accumulo 
serverse. The "!SYSTEM" user is this user. This user also has 
credentials but it is computed from a number of things, one of which is 
the value of all properties beginning with "instance." in accumulo-site.xml.

If you are seeing errors, it is likely that you did not correctly update 
accumulo-site.xml on _all_ nodes in the system and, as such, some nodes 
are expecting one "password" for !SYSTEM with other nodes expecting a 
different one. Ensure that you have consistent configuration across all 
nodes and restart Accumulo.

I think you have fooled yourself that the second exception is due to you 
reading from a user table. Note again, the user in question is the 
!SYSTEM user. You are not this user.

Webb, Ryan L. wrote:
> Below are two stack traces – one is from the GC and the latter is form
> the monitor when I try to look at a specific table.
>
> I am using 1.6.0-CHD5.1.0 on CentOS 6.4
>
> This happened after I attempted to fix the FQDN issue with bulk table
> loading.
>
> Is there a way to reset the password? How did this get changed?
>
> Unable to scan metadata table
>
> java.lang.RuntimeException:
> org.apache.accumulo.core.client.AccumuloSecurityException: Error
> BAD_CREDENTIALS for user !SYSTEM on table accumulo.root(ID:+r) -
> Username or Password is Invalid
>
> at
> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:187)
>
> at
> org.apache.accumulo.server.util.MetadataTableUtil.getLogEntries(MetadataTableUtil.java:547)
>
> at
> org.apache.accumulo.server.util.MetadataTableUtil$LogEntryIterator.<init>(MetadataTableUtil.java:615)
>
> at
> org.apache.accumulo.server.util.MetadataTableUtil.getLogEntries(MetadataTableUtil.java:652)
>
> at
> org.apache.accumulo.gc.GarbageCollectWriteAheadLogs.removeMetadataEntries(GarbageCollectWriteAheadLogs.java:280)
>
> at
> org.apache.accumulo.gc.GarbageCollectWriteAheadLogs.collect(GarbageCollectWriteAheadLogs.java:125)
>
> at
> org.apache.accumulo.gc.SimpleGarbageCollector.run(SimpleGarbageCollector.java:568)
>
> at
> org.apache.accumulo.gc.SimpleGarbageCollector.main(SimpleGarbageCollector.java:154)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.accumulo.start.Main$1.run(Main.java:141)
>
> at java.lang.Thread.run(Thread.java:744)
>
> Caused by: org.apache.accumulo.core.client.AccumuloSecurityException:
> Error BAD_CREDENTIALS for user !SYSTEM on table accumulo.root(ID:+r) -
> Username or Password is Invalid
>
> at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:454)
>
> at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:277)
>
> at
> org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:84)
>
> at
> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:177)
>
> ... 13 more
>
> Caused by: ThriftSecurityException(user:!SYSTEM, code:BAD_CREDENTIALS)
>
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startScan_result$startScan_resultStandardScheme.read(TabletClientService.java:5100)
>
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startScan_result$startScan_resultStandardScheme.read(TabletClientService.java:5077)
>
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startScan_result.read(TabletClientService.java:5000)
>
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
>
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:226)
>
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:202)
>
> at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:399)
>
> ... 16 more
>
> java.lang.RuntimeException:
> org.apache.accumulo.core.client.AccumuloSecurityException: Error
> BAD_CREDENTIALS for user !SYSTEM on table accumulo.metadata(ID:!0) -
> Username or Password is Invalid
>
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.hasNext(TabletServerBatchReaderIterator.java:208)
>
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.hasNext(MetaDataTableScanner.java:117)
>
> at
> org.apache.accumulo.monitor.servlets.TablesServlet.doTableDetails(TablesServlet.java:158)
>
> at
> org.apache.accumulo.monitor.servlets.TablesServlet.pageBody(TablesServlet.java:75)
>
> at
> org.apache.accumulo.monitor.servlets.BasicServlet.doGet(BasicServlet.java:65)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>
> at org.mortbay.jetty.Server.handle(Server.java:326)
>
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>
> at
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
> Caused by: org.apache.accumulo.core.client.AccumuloSecurityException:
> Error BAD_CREDENTIALS for user !SYSTEM on table accumulo.metadata(ID:!0)
> - Username or Password is Invalid
>
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:716)
>
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:372)
>
> at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>
> at java.lang.Thread.run(Thread.java:744)
>
> Caused by: ThriftSecurityException(user:!SYSTEM, code:BAD_CREDENTIALS)
>
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startMultiScan_result$startMultiScan_resultStandardScheme.read(TabletClientService.java:8613)
>
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startMultiScan_result$startMultiScan_resultStandardScheme.read(TabletClientService.java:8590)
>
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startMultiScan_result.read(TabletClientService.java:8529)
>
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
>
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startMultiScan(TabletClientService.java:311)
>
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startMultiScan(TabletClientService.java:291)
>
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:658)
>
> ... 7 more
>

Mime
View raw message