Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D9BAF1075F for ; Wed, 8 May 2013 16:46:19 +0000 (UTC) Received: (qmail 62004 invoked by uid 500); 8 May 2013 16:46:19 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 61919 invoked by uid 500); 8 May 2013 16:46:19 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 61911 invoked by uid 99); 8 May 2013 16:46:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 May 2013 16:46:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mreichman@pixelforensics.com designates 209.85.220.170 as permitted sender) Received: from [209.85.220.170] (HELO mail-vc0-f170.google.com) (209.85.220.170) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 May 2013 16:46:13 +0000 Received: by mail-vc0-f170.google.com with SMTP id gf12so1896275vcb.29 for ; Wed, 08 May 2013 09:45:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pixelforensics.com; s=google; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=fv56pSk3fjAr5XceoRhLDZahEqDe4aY490fj2HYACK8=; b=LmBgZYLCa7CgP6NEQH+MGxcsW3fidwSg4kthSc5QwXOsU90AN6nAJ9nLFl16PAeCPS f5ZhhGXnKvC1/S+hBVsHqXhCwslnLdFc2YV/1OFotTlLGvZIvI3IAk/NwgsBHUx4O0x6 D2PqV9qPksZNqOGA3rWwmSnxo0KnvLNT6tEak= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type:x-gm-message-state; bh=fv56pSk3fjAr5XceoRhLDZahEqDe4aY490fj2HYACK8=; b=NEL3bDJ2gaxhjLYfi51mId4Xac5bI/maL1DgFyCsa5gGlc4TFbHXsA0Ty3uy7sKqPr cLwBUPjgxboPoN060oHlSyAaLTX9gVDaQASdBIHsH5nGmBfM+O2GfM4QgmwScSeoi76E jLocOqlFmf3d5eWa78Y6STnSGYkr3WD8AtVW6f7rBw5v3H38ECCBAi6ufBGUZlaZM5GD V1M8FVkpU4AR09bAsq3PQNOByNUvS/3nnnwbeTchS+2lElNCqEvN6WfKj3yZxEYnfJK7 hkDMW/jQ8as1ZsicNvYvw7Y1t2AYlOKOZ3uIYfIxM0cI4hsU9QrvYyeqWWg1nF4Mkhb6 GgNA== MIME-Version: 1.0 X-Received: by 10.58.85.134 with SMTP id h6mr5251555vez.18.1368031552773; Wed, 08 May 2013 09:45:52 -0700 (PDT) Received: by 10.59.3.230 with HTTP; Wed, 8 May 2013 09:45:52 -0700 (PDT) In-Reply-To: References: Date: Wed, 8 May 2013 11:45:52 -0500 Message-ID: Subject: Re: remote accumulo instance issue From: Marc Reichman To: user@accumulo.apache.org, vines@apache.org Content-Type: multipart/alternative; boundary=047d7b6d9340beb05004dc37aa8b X-Gm-Message-State: ALoCoQmwLWFvZDR3Xpp4VrFMmTfOOkaofc5p/AvvEHCDqHynLzGKW3L/cqkL1xW1Qys5Iee0gOGQ X-Virus-Checked: Checked by ClamAV on apache.org --047d7b6d9340beb05004dc37aa8b Content-Type: text/plain; charset=ISO-8859-1 1.4.1., hadoop 1.0.3. Just for sanity, I ran 'accumulo classpath' on the cluster and am copying those exact files to my client side in case there was a mismatch somewhere. On Wed, May 8, 2013 at 11:43 AM, John Vines wrote: > What version of Accumulo are you running? > > Sent from my phone, please pardon the typos and brevity. > On May 8, 2013 12:38 PM, "Marc Reichman" > wrote: > >> I can't find anything wrong with the networking. Here is the whole error >> with stack trace: >> 2057 [main] WARN org.apache.accumulo.core.client.impl.ServerClient - >> Failed to find an available server in the list of servers: >> [192.168.1.164:9997:9997 (120000), 192.168.1.192:9997:9997 (120000), >> 192.168.1.194:9997:9997 (120000), 192.168.1.162:9997:9997 (120000), >> 192.168.1.190:9997:9997 (120000), 192.168.1.166:9997:9997 (120000), >> 192.168.1.168:9997:9997 (120000), 192.168.1.196:9997:9997 (120000)] >> Exception in thread "main" java.lang.IncompatibleClassChangeError: >> Implementing class >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:615) >> at >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) >> at java.net.URLClassLoader.access$000(URLClassLoader.java:58) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:197) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> at >> org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClient.java:146) >> at >> org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClient.java:123) >> at >> org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.java:105) >> at >> org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:71) >> at >> org.apache.accumulo.core.client.impl.ConnectorImpl.(ConnectorImpl.java:75) >> at >> org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:218) >> at >> org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:206) >> >> Running on JDK 1.6.0_27 >> >> >> On Wed, May 8, 2013 at 10:38 AM, Keith Turner wrote: >> >>> >>> >>> >>> On Wed, May 8, 2013 at 11:09 AM, Marc Reichman < >>> mreichman@pixelforensics.com> wrote: >>> >>>> I have seen this as ticket ACCUMULO-687 which has been marked resolved, >>>> but I still see this issue. >>>> >>>> I am connecting to a remote accumulo instance to query and to launch >>>> mapreduce jobs using AccumuloRowInputFormat, and I'm seeing an error like: >>>> >>>> 91 [main-SendThread(padres.home:2181)] INFO >>>> org.apache.zookeeper.ClientCnxn - Socket connection established to >>>> padres.home/192.168.1.160:2181, initiating session >>>> 166 [main-SendThread(padres.home:2181)] INFO >>>> org.apache.zookeeper.ClientCnxn - Session establishment complete on server >>>> padres.home/192.168.1.160:2181, sessionid = 0x13e7b48f9d17af7, >>>> negotiated timeout = 30000 >>>> 1889 [main] WARN org.apache.accumulo.core.client.impl.ServerClient - >>>> Failed to find an available server in the list of servers: >>>> [192.168.1.164:9997:9997 (120000), 192.168.1.192:9997:9997 (120000), >>>> 192.168.1.194:9997:9997 (120000), 192.168.1.162:9997:9997 (120000), >>>> 192.168.1.190:9997:9997 (120000), 192.168.1.166:9997:9997 (120000), >>>> 192.168.1.168:9997:9997 (120000), 192.168.1.196:9997:9997 (120000)] >>>> >>>> My zookeeper's "tservers" key looks like: >>>> [zk: localhost:2181(CONNECTED) 1] ls >>>> /accumulo/908a756e-1c81-4bea-a4de-675456499a10/tservers >>>> [192.168.1.164:9997, 192.168.1.192:9997, 192.168.1.194:9997, >>>> 192.168.1.162:9997, 192.168.1.190:9997, 192.168.1.166:9997, >>>> 192.168.1.168:9997, 192.168.1.196:9997] >>>> >>>> My masters and slaves file look like: >>>> [hadoop@padres conf]$ cat masters >>>> 192.168.1.160 >>>> [hadoop@padres conf]$ cat slaves >>>> 192.168.1.162 >>>> 192.168.1.164 >>>> 192.168.1.166 >>>> 192.168.1.168 >>>> 192.168.1.190 >>>> 192.168.1.192 >>>> 192.168.1.194 >>>> 192.168.1.196 >>>> >>>> tracers, gc, and monitor are the same as masters. >>>> >>>> I have no issues executing on the master, but I would like to work from >>>> a remote host. The remote host is on a VPN, and its default resolver is NOT >>>> the resolver from the remote network. If I do reverse lookup over the VPN >>>> *using* the remote resolver it shows proper hostnames. >>>> >>>> My concern is that something is causing the "host:port" entry plus the >>>> port to come up with this concatenated view of host:port:port, which is >>>> obviously not going to work. >>>> >>> >>> The second port is nothing to worry about. Its created by concatenating >>> what came from zookeeper with the default tserver port. The location from >>> zookeeper can contain a port. If for some reason the location in >>> zookeeper did not have a port, it would use the default. >>> >>> That second port should probably go away, its being added >>> by vestigial code. We always expect what comes from zookeeper to have port >>> now. >>> >>> >>>> >>>> What else can I try? I previously had hostnames in the >>>> masters/slaves/etc. files but now have the IPs. Should I re-init the >>>> instance to see if it changes anything in zookeeper? >>>> >>> >>> >> --047d7b6d9340beb05004dc37aa8b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
1.4.1., hadoop 1.0.3.

Just for sa= nity, I ran 'accumulo classpath' on the cluster and am copying thos= e exact files to my client side in case there was a mismatch somewhere.


On Wed,= May 8, 2013 at 11:43 AM, John Vines <vines@apache.org> wrote= :

What version of Accumulo are you running?

Sent from my phone, please pardon the typos and brevity.

=
On May 8, 2013 12:38 PM, "Marc Reichman&quo= t; <mr= eichman@pixelforensics.com> wrote:
I can't find anything wrong with the networking. Here = is the whole error with stack trace:
2057 [main] WARN org.apache.a= ccumulo.core.client.impl.ServerClient =A0- Failed to find an available serv= er in the list of servers: [192.168.1.164:9997:9997 (120000), 192.168.1.192= :9997:9997 (120000), 192.168.1.194:9997:9997 (120000), 192.168.1.162:9997:9= 997 (120000), 192.168.1.190:9997:9997 (120000), 192.168.1.166:9997:9997 (12= 0000), 192.168.1.168:9997:9997 (120000), 192.168.1.196:9997:9997 (120000)]<= /div>
Exception in thread "main" java.lang.IncompatibleClassChange= Error: Implementing class
<= /span>at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defin= eClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureCl= assLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLCla= ssLoader.java:283)
at java.net.URLClassLoade= r.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)<= /div>
at java.security.AccessCo= ntroller.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:1= 90)
at java.lang.ClassLoader.= loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301= )
at java.lang.ClassLoader.= loadClass(ClassLoader.java:247)
at org.apache.accumulo.core.client.impl.ServerClient.getConnect= ion(ServerClient.java:146)
at org.apache.accumulo.co= re.client.impl.ServerClient.getConnection(ServerClient.java:123)
= at org.apache.accumulo.core.cl= ient.impl.ServerClient.executeRaw(ServerClient.java:105)
at org.apache.accumulo.co= re.client.impl.ServerClient.execute(ServerClient.java:71)
at org.apache.accumulo.core.client.im= pl.ConnectorImpl.<init>(ConnectorImpl.java:75)
at org.apache.accumulo.co= re.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:218)
<= div> at org.apache.accumulo.cor= e.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:206)

Running on JDK 1.6.0_27


On Wed, May 8, 2013 at 10:3= 8 AM, Keith Turner <keith@deenlo.com> wrote:



On Wed, May 8, 2013 at 11:09 AM= , Marc Reichman <mreichman@pixelforensics.com> wr= ote:
I have seen= this as ticket ACCUMULO-687=A0which has been marked resolved, but I still = see this issue.

I am connecting to a remote accumulo instance to query and t= o launch mapreduce jobs using AccumuloRowInputFormat, and I'm seeing an= error like:

91 [main-SendThread(padres.home:2181)] INFO org.ap= ache.zookeeper.ClientCnxn =A0- Socket connection established to padres.home= /192.168.1.160:2181= , initiating session
166 [main-SendThread(padres.home:2181)] INFO org.apache.zookeeper.Clie= ntCnxn =A0- Session establishment complete on server padres.home/192.168.1.160:2181, sessi= onid =3D 0x13e7b48f9d17af7, negotiated timeout =3D 30000
1889 [main] WARN org.apache.accumulo.core.client.impl.ServerClient =A0= - Failed to find an available server in the list of servers: [192.168.1.164= :9997:9997 (120000), 192.168.1.192:9997:9997 (120000), 192.168.1.194:9997:9= 997 (120000), 192.168.1.162:9997:9997 (120000), 192.168.1.190:9997:9997 (12= 0000), 192.168.1.166:9997:9997 (120000), 192.168.1.168:9997:9997 (120000), = 192.168.1.196:9997:9997 (120000)]

My zookeeper's "tservers" key looks= like:
[zk: localhost:2181(CONNECTED) 1] ls /accumulo/908a75= 6e-1c81-4bea-a4de-675456499a10/tservers

My masters and slaves file look like:
[h= adoop@padres conf]$ cat masters
192.168.1.160
[hadoop@p= adres conf]$ cat slaves
192.168.1.162
192.168.1.164
192.168.1.166
192.168.1.168
192.168.1.190
192.168.1.192
192.168.1.194
192.168.1.196
<= br>
tracers, gc, and monitor are the same as masters.

I have no issues executing on the master, but I would l= ike to work from a remote host. The remote host is on a VPN, and its defaul= t resolver is NOT the resolver from the remote network. If I do reverse loo= kup over the VPN *using* the remote resolver it shows proper hostnames.

My concern is that something is causing the "host:= port" entry plus the port to come up with this concatenated view of ho= st:port:port, which is obviously not going to work.

The second port is nothing to = worry about. Its created by concatenating what came from zookeeper with the= default tserver port. =A0The location from zookeeper can contain a port. = =A0 If for some reason the location in zookeeper did not have a port, it wo= uld use the default. =A0

That second port should probably go away, its being add= ed by=A0vestigial=A0code. =A0We always expect what comes from zookeeper to = have port now.
=A0

What else can I try? I previously had hostnames i= n the masters/slaves/etc. files but now have the IPs. Should I re-init the = instance to see if it changes anything in zookeeper?



--047d7b6d9340beb05004dc37aa8b--