Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (athena.apache.org: domain of josh.elser@gmail.com
 designates 209.85.220.44 as permitted sender)
Message-ID: <54F9F52D.4020702@gmail.com>
Date: Fri, 06 Mar 2015 10:42:53 -0800
From: Josh Elser <josh.elser@gmail.com>
User-Agent: Postbox 3.0.11 (Macintosh/20140602)
MIME-Version: 1.0
To: user@accumulo.apache.org
Subject: Re: Bad GC credentials after moving namenode
References: 
 <CAPVnKEnExaM_X7pXpA22EzO-0VYHc55F6DbLq_8P4t3SBgU9Jw@mail.gmail.com>
 <54F9ECC3.1080301@gmail.com>
 <CAPVnKEnJDX-RN8NsYCCrEO93uyQgggoQ-jxUuJo=pxD7EhQydw@mail.gmail.com>
In-Reply-To: 
 <CAPVnKEnJDX-RN8NsYCCrEO93uyQgggoQ-jxUuJo=pxD7EhQydw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

This might help answer some questions: 
https://github.com/hortonworks/accumulo-release. Check out the 
branches/tags drop-down. You should be able to work backwards.

HDP-2.2 contained all of 1.6.1 and also contained a few bug fixes 
cherry-picked in after we released it in Apache. The data-center 
replication code was also included.

Russ Weeks wrote:
> Thanks Sean, thanks Josh.
>
> To be precise: we're running Accumulo 1.6.1.2.2.0.0-2041 provided by HDP
> with a hotfix for ACCUMULO-3312.
>
> I did an md5sum on all my accumulo-site.xml's and they all matched... I
> have a feeling that I just didn't restart the GC properly. I gave it a
> good kick, things seem OK now.
>
> Just curious Sean, which part of that stack trace diverges from the
> Apache Accumulo distribution?
>
> Regards,
> -Russ
>
> On Fri, Mar 6, 2015 at 10:06 AM, Josh Elser <josh.elser@gmail.com
> <mailto:josh.elser@gmail.com>> wrote:
>
>     The !SYSTEM user's credentials include a hashing of all properties
>     in accumulo-site.xml that begin with 'instance.'. This includes
>     instance.secret and has two goals: 1) prevent rogue servers from
>     maliciously entering the Accumulo instance. 2) prevent servers with
>     bad configurations from accidentally entering the Accumulo instance.
>
>     BAD_CREDENTIALS on !SYSTEM likely means that the new host has (or
>     the processing on that host are using) a different version of
>     accumulo-site.xml than the other servers.
>
>     Trying `diff`'ing accumulo-site.xml on the host running your GC and
>     one of the hosts running a TabletServer. If they appear to be
>     accurate, you can check the print-out of site configuration in the
>     debug.log for the GC and see if it somehow picked up a different
>     version of accumulo-site.xml than you expected.
>
>
>     Russ Weeks wrote:
>
>         Hi, folks,
>
>         I just finished moving my namenode, following the instructions
>         at [1]. I
>         think I got everything just about right but now I see these
>         errors in
>         the monitor. They seem to be associated with the GC process.
>
>         Unable to scan metadata table
>                  java.lang.RuntimeException:
>         org.apache.accumulo.core.__client.__AccumuloSecurityException:
>         Error BAD_CREDENTIALS for user !SYSTEM on table
>         accumulo.root(ID:+r) - Username or Password is Invalid
>                  at
>         org.apache.accumulo.core.__client.impl.ScannerIterator.__hasNext(ScannerIterator.java:__187)
>                  at
>         org.apache.accumulo.server.__util.MetadataTableUtil.__getLogEntries(__MetadataTableUtil.java:554)
>                  at
>         org.apache.accumulo.server.__util.MetadataTableUtil$__LogEntryIterator.<init>(__MetadataTableUtil.java:622)
>                  at
>         org.apache.accumulo.server.__util.MetadataTableUtil.__getLogEntries(__MetadataTableUtil.java:659)
>                  at
>         org.apache.accumulo.gc.__GarbageCollectWriteAheadLogs.__removeMetadataEntries(__GarbageCollectWriteAheadLogs.__java:321)
>                  at
>         org.apache.accumulo.gc.__GarbageCollectWriteAheadLogs.__collect(__GarbageCollectWriteAheadLogs.__java:151)
>                  at
>         org.apache.accumulo.gc.__SimpleGarbageCollector.run(__SimpleGarbageCollector.java:__608)
>                  at
>         org.apache.accumulo.gc.__SimpleGarbageCollector.main(__SimpleGarbageCollector.java:__159)
>                  at
>         sun.reflect.__NativeMethodAccessorImpl.__invoke0(Native Method)
>                  at
>         sun.reflect.__NativeMethodAccessorImpl.__invoke(__NativeMethodAccessorImpl.java:__57)
>                  at
>         sun.reflect.__DelegatingMethodAccessorImpl.__invoke(__DelegatingMethodAccessorImpl.__java:43)
>                  at java.lang.reflect.Method.__invoke(Method.java:606)
>                  at org.apache.accumulo.start.__Main$1.run(Main.java:141)
>                  at java.lang.Thread.run(Thread.__java:745)
>                  Caused by:
>         org.apache.accumulo.core.__client.__AccumuloSecurityException:
>         Error BAD_CREDENTIALS for user !SYSTEM on table
>         accumulo.root(ID:+r) - Username or Password is Invalid
>                  at
>         org.apache.accumulo.core.__client.impl.ThriftScanner.__scan(ThriftScanner.java:458)
>                  at
>         org.apache.accumulo.core.__client.impl.ThriftScanner.__scan(ThriftScanner.java:279)
>                  at
>         org.apache.accumulo.core.__client.impl.ScannerIterator$__Reader.run(ScannerIterator.__java:84)
>                  at
>         org.apache.accumulo.core.__client.impl.ScannerIterator.__hasNext(ScannerIterator.java:__177)
>                          ... 13 more
>                  Caused by: ThriftSecurityException(user:!__SYSTEM,
>         code:BAD_CREDENTIALS)
>                  at
>         org.apache.accumulo.core.__tabletserver.thrift.__TabletClientService$startScan___result$startScan___resultStandardScheme.read(__TabletClientService.java:6684)
>                  at
>         org.apache.accumulo.core.__tabletserver.thrift.__TabletClientService$startScan___result$startScan___resultStandardScheme.read(__TabletClientService.java:6661)
>                  at
>         org.apache.accumulo.core.__tabletserver.thrift.__TabletClientService$startScan___result.read(__TabletClientService.java:6584)
>                  at
>         org.apache.thrift.__TServiceClient.receiveBase(__TServiceClient.java:78)
>                  at
>         org.apache.accumulo.core.__tabletserver.thrift.__TabletClientService$Client.__recv_startScan(__TabletClientService.java:232)
>                  at
>         org.apache.accumulo.core.__tabletserver.thrift.__TabletClientService$Client.__startScan(TabletClientService.__java:208)
>                  at
>         org.apache.accumulo.core.__client.impl.ThriftScanner.__scan(ThriftScanner.java:403)
>                          ... 16 more
>
>
>         Any ideas? Everything seems to be working, eg. I can connect
>         using the
>         shell and browse the contents of my tables.
>
>         1:
>         http://accumulo.apache.org/1.__6/accumulo_user_manual.html#___instance_volumes_replacements
>         <http://accumulo.apache.org/1.6/accumulo_user_manual.html#_instance_volumes_replacements>
>
>