Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 85E7D18889 for ; Thu, 16 Jul 2015 03:05:59 +0000 (UTC) Received: (qmail 66599 invoked by uid 500); 16 Jul 2015 03:05:53 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 66542 invoked by uid 500); 16 Jul 2015 03:05:53 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 66531 invoked by uid 99); 16 Jul 2015 03:05:53 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2015 03:05:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6B192C0098 for ; Thu, 16 Jul 2015 03:05:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.879 X-Spam-Level: ** X-Spam-Status: No, score=2.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id bwJSGx_5As-X for ; Thu, 16 Jul 2015 03:05:51 +0000 (UTC) Received: from mail-oi0-f53.google.com (mail-oi0-f53.google.com [209.85.218.53]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 5436C43DDB for ; Thu, 16 Jul 2015 03:05:51 +0000 (UTC) Received: by oihq81 with SMTP id q81so41986508oih.2 for ; Wed, 15 Jul 2015 20:05:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=IXDZ21wksa051mMGbEn/MKDTwtv0m7xX1OuX9uBwkJw=; b=Xbf5cAJDDGistjLKbLbl8CN/qFO3pX3YQ2YZMlim8MzuZYt+/d0g0jBqGt1Xkopb+8 uBVUGyhgoR6lslmAA32dh8j2DrFbDhdo9o/z3v3vjbBdFex7+JWA7QZ8BtbDXGhzL6z+ x3+8qf2NV309pPCBGX3fuS+ubS8xiOmh2Q3DBCSoK4/7E05aXnjcJd60e+4AK/JNbgjt YUwBKHFvnPD0XKo2ffg+y4DKUjBbGTB1F+2SQnlKJMjDebw+/kBbLRZeGZL9Iq9nlS6c Y2o9XgsI/1febrzsU8nffBrToVu2Fsjxrbw2qTIOrhAOX26YkIFDTh+XyoUrMindGRlV F93Q== MIME-Version: 1.0 X-Received: by 10.202.79.211 with SMTP id d202mr6322012oib.120.1437015900513; Wed, 15 Jul 2015 20:05:00 -0700 (PDT) Received: by 10.76.28.102 with HTTP; Wed, 15 Jul 2015 20:05:00 -0700 (PDT) In-Reply-To: <559C20EE.6000201@gmail.com> References: <559C20EE.6000201@gmail.com> Date: Wed, 15 Jul 2015 23:05:00 -0400 Message-ID: Subject: Re: Failed to find an available server in the list of servers From: pundu tech To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary=001a113d709c496d58051af5564a --001a113d709c496d58051af5564a Content-Type: text/plain; charset=UTF-8 Josh, I had miss this email from you before. So I have done as you suggested. Let me summarize what I have done. 1- Followed https://blogs.apache.org/accumulo/entry/generating_keystores_for_configuring_accumulo I have a master (master)node and 4 slaves (slave1, slave2, slave3, slave4) I have created certificates for the 5 nodes and I have also created certificate for a client which is sitting in slave1. 2-Since I am running the shell from slave1 I have created a client.conf file which I pass to the shell via the --config-file parameter. INSTANCE_NAME=comet INSTANCE_RPC_SSL_ENABLED=true INSTANCE_RPC_SSL_CLIENT_AUTH=true INSTANCE_ZK_HOST=slave1,slave2,slave3,slave4 #the trustore is the same along all the nodes since it stores the pub key of the CA RPC_SSL_TRUSTSTORE_PATH=/home/hadoop/accumulo-1.7.0/conf/clientSSL/truststore.jks RPC_SSL_TRUSTSTORE_TYPE=JKS RPC_SSL_TRUSTSTORE_PASSWORD=accumuloAuth RPC_SSL_KEYSTORE_PATH=/home/hadoop/accumulo-1.7.0/conf/clientSSL/client.jks RPC_SSL_KEYSTORE_TYPE=JKS RPC_SSL_KEYSTORE_PASSWORD=mypass 3-I run the shell with --debug and this is what I get: 2015-07-15 22:53:06,380 [impl.ThriftTransportPool] DEBUG: Failed to connect to ssl:slave1:9997 (120000) org.apache.thrift.transport.TTransportException: Error creating the transport at org.apache.accumulo.core.rpc.ThriftUtil.createSSLContext(ThriftUtil.java:371) at org.apache.accumulo.core.rpc.ThriftUtil.createClientTransport(ThriftUtil.java:248) at org.apache.accumulo.core.client.impl.ThriftTransportPool.createNewTransport(ThriftTransportPool.java:478) at org.apache.accumulo.core.client.impl.ThriftTransportPool.getAnyTransport(ThriftTransportPool.java:466) at org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClient.java:141) at org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClient.java:117) at org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClient.java:113) at org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.java:95) at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:61) at org.apache.accumulo.core.client.impl.ConnectorImpl.(ConnectorImpl.java:67) at org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:248) at org.apache.accumulo.shell.Shell.config(Shell.java:362) at org.apache.accumulo.shell.Shell.execute(Shell.java:571) at org.apache.accumulo.start.Main$1.run(Main.java:93) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55) at java.security.KeyStore.load(KeyStore.java:1214) at org.apache.accumulo.core.rpc.ThriftUtil.createSSLContext(ThriftUtil.java:348) ... 14 more Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770) This error repeats for every slave. I have tested the password for every keystore and truststore file in the cluster and it is correct--it is the same everywhere. I am very positive about this at this point. Do you have any suggestion on what else could be wrong? I appreciate your help. I am stuck! pundutech On Tue, Jul 7, 2015 at 2:56 PM, Josh Elser wrote: > Pundu, > > The password to use would be the "root" user's password that you set when > calling `accumulo init`. A limitation of the SSL approach is that it only > uses sets up a secure RPC, it isn't a "complete" security implementation > (as you might get with Kerberos in 1.7). > > Sadly, the error messages for SSL are very sparse when the client fails to > negotiate the handshake with a server. With the Accumulo shell, you can try > passing in the --debug option to get more information. > > Alternatively, try turning up org.apache.accumulo.core.client to DEBUG or > TRACE in $ACCUMULO_CONF_DIR/log4j.properties. > > - Josh > > > pundu tech wrote: > >> I have a SSL enabled-accumulo setup. >> >> I have followed: >> >> https://blogs.apache.org/accumulo/entry/generating_keystores_for_configuring_accumulo >> to the teeth and as far as my undersatnding goes on SSL it is all >> correct. >> >> I have created a $ACCUMULO_HOME/conf/client.conf with the following >> properties >> >> INSTANCE_NAME=accumulo >> >> INSTANCE_RPC_SSL_ENABLED=true >> >> NSTANCE_RPC_SSL_CLIENT_AUTH=true >> >> INSTANCE_ZK_HOST=host1 >> >> RPC_SSL_TRUSTSTORE_PATH=/home/hadoop/truststore.jks >> >> RPC_SSL_TRUSTSTORE_TYPE=JKS >> >> RPC_SSL_TRUSTSTORE_PASSWORD=mypass >> >> RPC_SSL_KEYSTORE_PATH=/home/hadoop/server.jks >> >> RPC_SSL_KEYSTORE_TYPE=JKS >> >> RPC_SSL_KEYSTORE_PASSWORD=mypass >> >> >> but when I try to connect via shell I am prompted for a password. Which >> password is this? It does not seem to be the tracer password (which user >> is "root"). >> >> ./accumulo shell -u root >> >> /usr/local/zookeeper-3.4.6 >> >> /usr/local/jdk1.7.0_79 >> >> Password: ----> ? >> >> >> Thanks >> >> pundu tech >> >> --001a113d709c496d58051af5564a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Josh,=C2=A0
I had miss this email from you before.

So I have done as you suggested.=C2=A0 Let me summari= ze what I have done.

I have a master (master)node and 4 sla= ves (slave1, slave2, slave3, slave4)
I have created certificates = for the 5 nodes and I have also created certificate for a client which is s= itting in slave1.
2-Since I am running the shell from slave1 I ha= ve created a client.conf file which I pass to the shell via the --config-fi= le parameter.

INSTANCE_NAME=3Dcomet

INSTANCE_RPC_SSL_ENABLED=3Dtrue

INSTANCE_RPC_SSL_CLIENT_AUTH=3Dtrue

INSTANCE_ZK_HOST=3Dslave1,slave2,slave3,slav= e4

#the trustore is the same along all the node= s since it stores the pub key of the CA

RPC_SSL_TRUSTSTORE_PATH=3D/home/hadoop/accumulo-1.7.0/conf/clientSSL/= truststore.jks

RPC_SSL_TRUSTSTORE_TYPE=3DJKS

RPC_SSL_TRUSTSTORE_PASSWORD=3DaccumuloAuth

RPC_SSL_KEYSTORE_PATH=3D/home/hadoop/accumul= o-1.7.0/conf/clientSSL/client.jks

RPC_SSL_KEYSTORE_TYPE=3DJKS

RPC_SSL_KEYSTORE_PASSWORD=3Dmypass

3-I run the shell with --debug and this is = what I get:

2015-07-15 22:53:06,38= 0 [impl.ThriftTransportPool] DEBUG: Failed to connect to ssl:slave1:9997 (1= 20000)

org.apache.thrift.transport= .TTransportException: Error creating the transport

= at org.apache.accumulo.core.rpc.T= hriftUtil.createSSLContext(ThriftUtil.java:371)

at org.apache.accumulo.core.rpc.Thri= ftUtil.createClientTransport(ThriftUtil.java:248)

<= span class=3D""> at org.apache.accumulo.core.client= .impl.ThriftTransportPool.createNewTransport(ThriftTransportPool.java:478)<= /span>

at org.ap= ache.accumulo.core.client.impl.ThriftTransportPool.getAnyTransport(ThriftTr= ansportPool.java:466)

at org.apache.accumulo.core.client.impl.ServerClient.getConne= ction(ServerClient.java:141)

at org.apache.accumulo.core.client.impl.ServerClient.ge= tConnection(ServerClient.java:117)

at org.apache.accumulo.core.client.impl.ServerCli= ent.getConnection(ServerClient.java:113)

at org.apache.accumulo.core.client.impl.Ser= verClient.executeRaw(ServerClient.java:95)

at org.apache.accumulo.core.client.impl.S= erverClient.execute(ServerClient.java:61)

at org.apache.accumulo.core.client.impl.Co= nnectorImpl.<init>(ConnectorImpl.java:67)

at org.apache.accumulo.core.client.Z= ooKeeperInstance.getConnector(ZooKeeperInstance.java:248)

at org.apache.accumulo.she= ll.Shell.config(Shell.java:362)

at org.apache.accumulo.shell.Shell.execute(Shell.jav= a:571)

at= org.apache.accumulo.start.Main$1.run(Main.java:93)

at java.lang.Thread.run(Thread.j= ava:745)

Caused by: java.io.IOExce= ption: Keystore was tampered with, or password was incorrect

at sun.security.provide= r.JavaKeyStore.engineLoad(JavaKeyStore.java:772)

at sun.security.provider.JavaKeySto= re$JKS.engineLoad(JavaKeyStore.java:55)

at java.security.KeyStore.load(KeyStore.java= :1214)

at= org.apache.accumulo.core.rpc.ThriftUtil.createSSLContext(ThriftUtil.java:3= 48)

... 1= 4 more

Caused by: java.security.Un= recoverableKeyException: Password verification failed

at sun.s= ecurity.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)

<= /div>

This error repeats for every slave.
I ha= ve tested the password for every keystore and truststore file in the cluste= r and it is correct--it is the same everywhere.=C2=A0 I am very positive ab= out this at this point.=C2=A0 Do you have any suggestion on what else could= be wrong?

I appreciate your help. I am stuck!

pundutech

<= div class=3D"gmail_quote">On Tue, Jul 7, 2015 at 2:56 PM, Josh Elser <j= osh.elser@gmail.com> wrote:
Pundu,

The password to use would be the "root" user's password that = you set when calling `accumulo init`. A limitation of the SSL approach is t= hat it only uses sets up a secure RPC, it isn't a "complete" = security implementation (as you might get with Kerberos in 1.7).

Sadly, the error messages for SSL are very sparse when the client fails to = negotiate the handshake with a server. With the Accumulo shell, you can try= passing in the --debug option to get more information.

Alternatively, try turning up org.apache.accumulo.core.client to DEBUG or T= RACE in $ACCUMULO_CONF_DIR/log4j.properties.

- Josh


pundu tech wrote:
I have a SSL enabled-accumulo setup.

I have followed:
https://blogs.a= pache.org/accumulo/entry/generating_keystores_for_configuring_accumulo<= br> =C2=A0 to the teeth and as far as my undersatnding goes on SSL it is all co= rrect.

I have created a $ACCUMULO_HOME/conf/client.conf with the following
properties

INSTANCE_NAME=3Daccumulo

INSTANCE_RPC_SSL_ENABLED=3Dtrue

NSTANCE_RPC_SSL_CLIENT_AUTH=3Dtrue

INSTANCE_ZK_HOST=3Dhost1

RPC_SSL_TRUSTSTORE_PATH=3D/home/hadoop/truststore.jks

RPC_SSL_TRUSTSTORE_TYPE=3DJKS

RPC_SSL_TRUSTSTORE_PASSWORD=3Dmypass

RPC_SSL_KEYSTORE_PATH=3D/home/hadoop/server.jks

RPC_SSL_KEYSTORE_TYPE=3DJKS

RPC_SSL_KEYSTORE_PASSWORD=3Dmypass


but when I try to connect via shell I am prompted for a password. Which
password is this? It does not seem to be the tracer password (which user is "root").

./accumulo shell -u root

/usr/local/zookeeper-3.4.6

/usr/local/jdk1.7.0_79

Password:=C2=A0 =C2=A0----> ?


Thanks

pundu tech


--001a113d709c496d58051af5564a--