Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5AA3010B3C for ; Wed, 16 Apr 2014 23:21:42 +0000 (UTC) Received: (qmail 42604 invoked by uid 500); 16 Apr 2014 23:21:41 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 42532 invoked by uid 500); 16 Apr 2014 23:21:41 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 42524 invoked by uid 99); 16 Apr 2014 23:21:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Apr 2014 23:21:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of threadedblue@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-we0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Apr 2014 23:21:36 +0000 Received: by mail-we0-f172.google.com with SMTP id t61so11599997wes.17 for ; Wed, 16 Apr 2014 16:21:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=4UU7U5Rn+2euBhFqaTG4ziN3BBvhAgxGMEEV5Xlw/MI=; b=cwu2YRV5rDbugYfbhm5xqecA+SmC8eJTzIeRxKEkK1iO9ysob6hzSshioDHZ3obtLz GHgFjRHXiTriXNDM5Y89x/7Xm+gBpYXii3uKTX02XBeBGsTUCGS+bWokmy2kLD+lxgHB SEA4v1OAypfj/o56rid9bVD3e0u8xfYxpnou9riubqdXK/Z/1qT08koy4Cm/l9wR+Xmr 6VFrjkxQQN28Yu3D6lwOkG89hZUQ02vMjxtXrPUM4d/BVcsOBJBDUAnu86B3yfn9gUKs DSjLBkD4dWpwkHygTt6ZrLS20cGhfQDB77wwb9hXihT0dyKXYL5h0ITOGTFBqGmRi8Uu fmrw== MIME-Version: 1.0 X-Received: by 10.194.191.195 with SMTP id ha3mr3920071wjc.69.1397690475589; Wed, 16 Apr 2014 16:21:15 -0700 (PDT) Received: by 10.194.139.76 with HTTP; Wed, 16 Apr 2014 16:21:15 -0700 (PDT) In-Reply-To: <534F04B9.7040301@gmail.com> References: <534F04B9.7040301@gmail.com> Date: Wed, 16 Apr 2014 19:21:15 -0400 Message-ID: Subject: Re: Accumulo not starting anymore From: Geoffry Roberts To: Accumulo Content-Type: multipart/alternative; boundary=047d7ba970bc4db6f104f7312cd3 X-Virus-Checked: Checked by ClamAV on apache.org --047d7ba970bc4db6f104f7312cd3 Content-Type: text/plain; charset=UTF-8 Here's what I get: Listing from hdfs: drwxr-xr-x - gcr supergroup 0 2014-04-15 11:42 /accumulo/instance_id drwxr-xr-x - gcr supergroup 0 2014-04-15 13:11 /accumulo/recovery drwxr-xr-x - gcr supergroup 0 2014-04-15 11:42 /accumulo/tables drwxr-xr-x - gcr supergroup 0 2014-04-15 11:42 /accumulo/version drwxr-xr-x - gcr supergroup 0 2014-04-15 11:43 /accumulo/wal Zookeeper shell: [zk: localhost:2181(CONNECTED) 0] ls /accumulo Node does not exist: /accumulo [zk: localhost:2181(CONNECTED) 1] ls /accumulo/instances Node does not exist: /accumulo/instances [zk: localhost:2181(CONNECTED) 2] On Wed, Apr 16, 2014 at 6:31 PM, Josh Elser wrote: > Are you sure ZooKeeper is running and wasn't wiped? > > See what kind of results come from `ls /accumulo` and `ls > /accumulo/instances` in the ZooKeeper shell ($ZOOKEEPER_HOME/bin/zkCli.sh). > '/accumulo' should be non-empty, and you should see the instance name you > provided when running `accumulo init` inside '/accumulo/instances' > > > On 4/16/14, 4:15 PM, Geoffry Roberts wrote: > >> All, >> >> Suddenly, Accumulo will no longer start. Log files are not helpful. Is >> there a way to troubleshoot this? >> >> The back story is I upgraded from OSX 10.7 to 10.9. Everything was >> working with 10.7. But with 10.9 Accumulo began to complain of >> insufficient file limits and recommended setting maxfiles to 65536, >> which I did. >> >> Hadoop starts -- version 2.3.0 >> Zookeeper starts -- version 3.4.6 >> Java -- version 1.7.55 >> >> I've included part of a log file just in case. >> >> Thanks mucho >> >> From: master_abend.home.debug.log >> >> t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting >> 2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance >> d9f3a06a-ef06-4860-a08d-9cff805a9249 >> 2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5 >> 2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to talk to >> zookeeper >> 2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to >> localhost:2181 with timeout 30000 with auth >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 >> GMT >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:host.name =abend.home >> >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:java.version=1.7.0_55 >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:java.vendor=Oracle Corporation >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_ >> 55.jdk/Contents/Home/jre >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:java.class.path=/usr/local/accumulo/conf:/usr/ >> local/accumulo/lib/accumulo-start.jar:/usr/local/hadoop/ >> share/hadoop/common/lib/log4j-1.2.17.jar >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:java.library.path=/usr/local/hadoop/lib/native >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:java.io.tmpdir=/var/folders/sb/g6bpj4cd401c1sw566x2r41m0000gn >> /T/ >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:java.compiler= >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:os.name =Mac OS X >> >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:os.arch=x86_64 >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:os.version=10.9.2 >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:user.name =gcr >> >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:user.home=/Users/gcr >> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client >> environment:user.dir=/Users/gcr >> 2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating client >> connection, connectString=localhost:2181 sessionTimeout=30000 >> watcher=org.apache.accumulo.fate.zookeeper.ZooSession$ZooWatcher@14731467 >> 2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket >> connection to server localhost/127.0.0.1:2181 . >> >> Will not attempt to authenticate using SASL (unknown error) >> 2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket connection >> established to localhost/127.0.0.1:2181 , >> >> initiating session >> 2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session >> establishment complete on server localhost/127.0.0.1:2181 >> , sessionid = 0x1456c089e9b0008, negotiated >> >> timeout = 30000 >> 2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set watch >> for Monitor Log4j watcher >> 2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for accumulo to >> be initialized >> 2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for accumulo to >> be initialized >> >> -- >> There are ways and there are ways, >> >> Geoffry Roberts >> > -- There are ways and there are ways, Geoffry Roberts --047d7ba970bc4db6f104f7312cd3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Here's what I get:

Listing from hdfs:


drwxr-xr-x =C2= =A0 - gcr supergroup=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 2014-04-15 11:42 /= accumulo/instance_id

drwxr-xr-x =C2=A0 - gcr = supergroup=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 2014-04-15 13:11 /accumulo/r= ecovery

drwxr-xr-x =C2=A0 - gcr = supergroup=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 2014-04-15 11:42 /accumulo/t= ables

drwxr-xr-x =C2=A0 - gcr = supergroup=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 2014-04-15 11:42 /accumulo/v= ersion

drwxr-xr-x =C2=A0 - gcr = supergroup=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 2014-04-15 11:43 /accumulo/w= al


Zookeeper shell:

[zk: lo= calhost:2181(CONNECTED) 0] ls /accumulo

Node does not exist: /ac= cumulo

[zk: localhost:2181(CONN= ECTED) 1] ls /accumulo/instances

Node does not exist: /ac= cumulo/instances

[zk: localhost:2181(CONN= ECTED) 2]=C2=A0




On Wed, Apr 16, 2014 at 6:31 PM, Josh Elser = <josh.elser@gmail.com> wrote:
Are you sure ZooKeeper is running and wasn't wiped?

See what kind of results come from `ls /accumulo` and `ls /accumulo/instanc= es` in the ZooKeeper shell ($ZOOKEEPER_HOME/bin/zkCli.sh). '/acc= umulo' should be non-empty, and you should see the instance name you pr= ovided when running `accumulo init` inside '/accumulo/instances'


On 4/16/14, 4:15 PM, Geoffry Roberts wrote:
All,

Suddenly, Accumulo will no longer start. =C2=A0Log files are not helpful. = =C2=A0Is
there a way to troubleshoot this?

The back story is I upgraded from OSX 10.7 to 10.9. =C2=A0Everything was working with 10.7. =C2=A0But with 10.9 Accumulo began to complain of
insufficient file limits and recommended setting maxfiles to 65536,
which I did.

Hadoop starts -- version 2.3.0
Zookeeper starts -- version 3.4.6
Java -- version 1.7.55

I've included part of a log file just in case.

Thanks mucho

From: =C2=A0master_abend.home.debug.log

t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting
2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance
d9f3a06a-ef06-4860-a08d-9cff805a9249
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to talk to
zookeeper
2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to
localhost:2181 with timeout 30000 with auth
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:zookeeper.version=3D3.4.6-1569965, built on 02/20/2014 0= 9:09 GMT
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:host.name &l= t;http://host.name>= =3Dabend.home

2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.version=3D1.7.0_55
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.vendor=3DOracle Corporation
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.home=3D/Library/Java/JavaVirtualMachines/jdk= 1.7.0_55.jdk/Contents/Home/jre
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.class.path=3D/usr/local/accumulo/conf:/usr/l= ocal/accumulo/lib/accumulo-start.jar:/usr/local/hadoop/share/= hadoop/common/lib/log4j-1.2.17.jar
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.library.path=3D/usr/local/hadoop/lib/native
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.io.tmpdir=3D/var/folders/sb/g6bpj4cd401c1sw5= 66x2r41m0000gn/T/
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.compiler=3D<NA>
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:os.name <http://os.name>=3DMac OS = X

2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:os.arch=3Dx86_64
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:os.version=3D10.9.2
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:user.name &l= t;http://user.name>= =3Dgcr

2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:user.home=3D/Users/gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:user.dir=3D/Users/gcr
2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating client
connection, connectString=3Dlocalhost:2181 sessionTimeout=3D30000
watcher=3Dorg.apache.accumulo.fate.zookeeper.ZooSession$ZooWa= tcher@14731467
2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket
connection to server localhost/127.0.0.1:2181 <http://127.0.0.1:2181>.

Will not attempt to authenticate using SASL (unknown error)
2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket connection
=
established to localhost/127.0.0.1:2181 <http://127.0.0.1:2181>,

initiating session
2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session
establishment complete on server localhost/127.0.0.1:2181
<http://127.0.0.1:21= 81>, sessionid =3D 0x1456c089e9b0008, negotiated

timeout =3D 30000
2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set watch
for Monitor Log4j watcher
2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for accumulo to be initialized
2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for accumulo to be initialized

--
There are ways and there are ways,

Geoffry Roberts



--
There are ways and there are ways,=C2=A0

Geof= fry Roberts
--047d7ba970bc4db6f104f7312cd3--