accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russ Weeks <rwe...@newbrightidea.com>
Subject Re: Bad GC credentials after moving namenode
Date Fri, 06 Mar 2015 18:28:48 GMT
Thanks Sean, thanks Josh.

To be precise: we're running Accumulo 1.6.1.2.2.0.0-2041 provided by HDP
with a hotfix for ACCUMULO-3312.

I did an md5sum on all my accumulo-site.xml's and they all matched... I
have a feeling that I just didn't restart the GC properly. I gave it a good
kick, things seem OK now.

Just curious Sean, which part of that stack trace diverges from the Apache
Accumulo distribution?

Regards,
-Russ

On Fri, Mar 6, 2015 at 10:06 AM, Josh Elser <josh.elser@gmail.com> wrote:

> The !SYSTEM user's credentials include a hashing of all properties in
> accumulo-site.xml that begin with 'instance.'. This includes
> instance.secret and has two goals: 1) prevent rogue servers from
> maliciously entering the Accumulo instance. 2) prevent servers with bad
> configurations from accidentally entering the Accumulo instance.
>
> BAD_CREDENTIALS on !SYSTEM likely means that the new host has (or the
> processing on that host are using) a different version of accumulo-site.xml
> than the other servers.
>
> Trying `diff`'ing accumulo-site.xml on the host running your GC and one of
> the hosts running a TabletServer. If they appear to be accurate, you can
> check the print-out of site configuration in the debug.log for the GC and
> see if it somehow picked up a different version of accumulo-site.xml than
> you expected.
>
>
> Russ Weeks wrote:
>
>> Hi, folks,
>>
>> I just finished moving my namenode, following the instructions at [1]. I
>> think I got everything just about right but now I see these errors in
>> the monitor. They seem to be associated with the GC process.
>>
>> Unable to scan metadata table
>>         java.lang.RuntimeException: org.apache.accumulo.core.client.AccumuloSecurityException:
>> Error BAD_CREDENTIALS for user !SYSTEM on table accumulo.root(ID:+r) -
>> Username or Password is Invalid
>>         at org.apache.accumulo.core.client.impl.ScannerIterator.
>> hasNext(ScannerIterator.java:187)
>>         at org.apache.accumulo.server.util.MetadataTableUtil.
>> getLogEntries(MetadataTableUtil.java:554)
>>         at org.apache.accumulo.server.util.MetadataTableUtil$
>> LogEntryIterator.<init>(MetadataTableUtil.java:622)
>>         at org.apache.accumulo.server.util.MetadataTableUtil.
>> getLogEntries(MetadataTableUtil.java:659)
>>         at org.apache.accumulo.gc.GarbageCollectWriteAheadLogs.
>> removeMetadataEntries(GarbageCollectWriteAheadLogs.java:321)
>>         at org.apache.accumulo.gc.GarbageCollectWriteAheadLogs.collect(
>> GarbageCollectWriteAheadLogs.java:151)
>>         at org.apache.accumulo.gc.SimpleGarbageCollector.run(
>> SimpleGarbageCollector.java:608)
>>         at org.apache.accumulo.gc.SimpleGarbageCollector.main(
>> SimpleGarbageCollector.java:159)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke(
>> NativeMethodAccessorImpl.java:57)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>         at org.apache.accumulo.start.Main$1.run(Main.java:141)
>>         at java.lang.Thread.run(Thread.java:745)
>>         Caused by: org.apache.accumulo.core.client.AccumuloSecurityException:
>> Error BAD_CREDENTIALS for user !SYSTEM on table accumulo.root(ID:+r) -
>> Username or Password is Invalid
>>         at org.apache.accumulo.core.client.impl.ThriftScanner.
>> scan(ThriftScanner.java:458)
>>         at org.apache.accumulo.core.client.impl.ThriftScanner.
>> scan(ThriftScanner.java:279)
>>         at org.apache.accumulo.core.client.impl.ScannerIterator$
>> Reader.run(ScannerIterator.java:84)
>>         at org.apache.accumulo.core.client.impl.ScannerIterator.
>> hasNext(ScannerIterator.java:177)
>>                 ... 13 more
>>         Caused by: ThriftSecurityException(user:!SYSTEM,
>> code:BAD_CREDENTIALS)
>>         at org.apache.accumulo.core.tabletserver.thrift.
>> TabletClientService$startScan_result$startScan_resultStandardScheme.read(
>> TabletClientService.java:6684)
>>         at org.apache.accumulo.core.tabletserver.thrift.
>> TabletClientService$startScan_result$startScan_resultStandardScheme.read(
>> TabletClientService.java:6661)
>>         at org.apache.accumulo.core.tabletserver.thrift.
>> TabletClientService$startScan_result.read(TabletClientService.java:6584)
>>         at org.apache.thrift.TServiceClient.receiveBase(
>> TServiceClient.java:78)
>>         at org.apache.accumulo.core.tabletserver.thrift.
>> TabletClientService$Client.recv_startScan(TabletClientService.java:232)
>>         at org.apache.accumulo.core.tabletserver.thrift.
>> TabletClientService$Client.startScan(TabletClientService.java:208)
>>         at org.apache.accumulo.core.client.impl.ThriftScanner.
>> scan(ThriftScanner.java:403)
>>                 ... 16 more
>>
>>
>> Any ideas? Everything seems to be working, eg. I can connect using the
>> shell and browse the contents of my tables.
>>
>> 1:
>> http://accumulo.apache.org/1.6/accumulo_user_manual.html#_
>> instance_volumes_replacements
>>
>

Mime
View raw message