Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (athena.apache.org: domain of josh.elser@gmail.com
 designates 209.85.216.176 as permitted sender)
Message-ID: <546CE37E.2000806@gmail.com>
Date: Wed, 19 Nov 2014 13:37:50 -0500
From: Josh Elser <josh.elser@gmail.com>
User-Agent: Postbox 3.0.11 (Macintosh/20140602)
MIME-Version: 1.0
To: user@accumulo.apache.org
Subject: Re: Tablet server and tracer could get started since they cannot
 see initialized accumulo instance
References: 
 <CAF+udknSoM=6ay7wTk=-mS2QOkJZWX7=5ZWtKjjfj=SQiB3TEw@mail.gmail.com>
 <546CB868.1090301@gmail.com>
 <CAF+udk=D5cFdcYGrZ=fXsFsew+RNC=vtU9TzNCRF=WqUprNXjQ@mail.gmail.com>
In-Reply-To: 
 <CAF+udk=D5cFdcYGrZ=fXsFsew+RNC=vtU9TzNCRF=WqUprNXjQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Sure thing. This showed me what I thought might be the case.

In your accumulo-env.sh, you're test && export'ing (likely) an incorrect 
HADOOP_CONF_DIR. Note that you have the Hadoop-2 HADOOP_CONF_DIR 
location uncommented with the Hadoop-1 commented. The example 
configuration files for 1.6.1 expect that you're using a Hadoop-2 release.

I'm guessing that you have the location properly configured in your 
local environment which is why `accumulo init` properly found HDFS. When 
start-all.sh SSH's to the other nodes to start them, it doesn't have 
HADOOP_CONF_DIR in its environment, accumulo-env.sh sets the wrong 
location and then the server processes can't find HDFS and falls back to 
the local filesystem.

Try changing accumulo-env.sh on all the nodes in your system and then 
re-run start-all.sh.

Salih Kardan wrote:
> Hi Josh,
>
> Thanks for the prompt response, it is very good for Accumulo users to
> have such a responsive community like this.
>
> I have reproduced the problem and fetched the logs you want.
>
> Here is the output of *accumulo init *command :
>
>     2014-11-19 20:14:30,583 [fs.VolumeManagerImpl] WARN :
>     dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is
>     possible on system reset or power loss
>     2014-11-19 20:14:30,584 [init.Initialize] INFO : Hadoop Filesystem
>     is hdfs://hadoop35:8020
>     2014-11-19 20:14:30,585 [init.Initialize] INFO : Accumulo data dirs
>     are [hdfs://hadoop35:8020/accumulo]
>     2014-11-19 20:14:30,585 [init.Initialize] INFO : Zookeeper server is
>     hadoop33:2181
>     2014-11-19 20:14:30,585 [init.Initialize] INFO : Checking if
>     Zookeeper is available. If this hangs, then you need to make sure
>     zookeeper is running
>
>     Warning!!! Your instance secret is still set to the default, this is
>     not secure. We highly recommend you change it.
>
>     You can change the instance secret in accumulo by using:
>         bin/accumulo org.apache.accumulo.server.util.ChangeSecret
>     oldPassword newPassword.
>     You will also need to edit your secret in your configuration file by
>     adding the property instance.secret to your conf/accumulo-site.xml.
>     Without this accumulo will not operate correctly
>     Instance name : test
>     Instance name "test" exists. Delete existing entry from zookeeper?
>     [Y/N] : Y
>     Enter initial password for root (this may not be applicable for your
>     security setup): ****
>     Confirm initial password for root: ****
>     2014-11-19 20:14:38,774 [fs.VolumeManagerImpl] WARN :
>     dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is
>     possible on system reset or power loss
>     2014-11-19 20:14:41,128 [util.NativeCodeLoader] WARN : Unable to
>     load native-hadoop library for your platform... using builtin-java
>     classes where applicable
>     2014-11-19 20:14:41,214 [conf.AccumuloConfiguration] INFO : Loaded
>     class : org.apache.accumulo.server.security.handler.ZKAuthorizor
>     2014-11-19 20:14:41,216 [conf.AccumuloConfiguration] INFO : Loaded
>     class : org.apache.accumulo.server.security.handler.ZKAuthenticator
>     2014-11-19 20:14:41,217 [conf.AccumuloConfiguration] INFO : Loaded
>     class : org.apache.accumulo.server.security.handler.ZKPermHandler
>
>
>
> Also here is the *accumulo-env.sh*:
>
>
>     #! /usr/bin/env bash
>
>     if [ -z "$HADOOP_HOME" ]
>     then
>         test -z "$HADOOP_PREFIX" && export HADOOP_PREFIX=/opt/hadoop-1.2.1
>     else
>         HADOOP_PREFIX="$HADOOP_HOME"
>         unset HADOOP_HOME
>     fi
>
>     # hadoop-1.2:
>     # test -z "$HADOOP_CONF_DIR" && export
>     HADOOP_CONF_DIR="$HADOOP_PREFIX/conf"
>     # hadoop-2.0:
>     test -z "$HADOOP_CONF_DIR" && export
>     HADOOP_CONF_DIR="$HADOOP_PREFIX/etc/hadoop"
>
>     test -z "$JAVA_HOME" && export
>     JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64
>     test -z "$ZOOKEEPER_HOME" && export ZOOKEEPER_HOME=/opt/zookeeper-3.4.5
>     test -z "$ACCUMULO_LOG_DIR" && export
>     ACCUMULO_LOG_DIR=/opt/accumulo-1.6.1/log
>     if [ -f ${ACCUMULO_CONF_DIR}/accumulo.policy ]
>     then
>         POLICY="-Djava.security.manager
>     -Djava.security.policy=${ACCUMULO_CONF_DIR}/accumulo.policy"
>     fi
>     test -z "$ACCUMULO_TSERVER_OPTS" && export
>     ACCUMULO_TSERVER_OPTS="${POLICY} -Xmx128m -Xms128m "
>     test -z "$ACCUMULO_MASTER_OPTS" && export
>     ACCUMULO_MASTER_OPTS="${POLICY} -Xmx128m -Xms128m"
>     test -z "$ACCUMULO_MONITOR_OPTS" && export
>     ACCUMULO_MONITOR_OPTS="${POLICY} -Xmx64m -Xms64m"
>     test -z "$ACCUMULO_GC_OPTS" && export ACCUMULO_GC_OPTS="-Xmx64m -Xms64m"
>     test -z "$ACCUMULO_GENERAL_OPTS" && export
>     ACCUMULO_GENERAL_OPTS="-XX:+UseConcMarkSweepGC
>     -XX:CMSInitiatingOccupancyFraction=75 -Djava.net.preferIPv4Stack=true"
>     test -z "$ACCUMULO_OTHER_OPTS" && export
>     ACCUMULO_OTHER_OPTS="-Xmx128m -Xms64m"
>     # what do when the JVM runs out of heap memory
>     export ACCUMULO_KILL_CMD='kill -9 %p'
>
>     ### Optionally look for hadoop and accumulo native libraries for your
>     ### platform in additional directories. (Use DYLD_LIBRARY_PATH on
>     Mac OS X.)
>     ### May not be necessary for Hadoop 2.x or using an RPM that installs to
>     ### the correct system library directory.
>     # export
>     LD_LIBRARY_PATH=${HADOOP_PREFIX}/lib/native/${PLATFORM}:${LD_LIBRARY_PATH}
>
>     # Should the monitor bind to all network interfaces -- default: false
>     # export ACCUMULO_MONITOR_BIND_ALL="true"
>
>
>
> And finally output of *accumulo classpath *command:
>
>     Level 1: Java System Classloader (loads Java system resources) URL
>     classpath items are:
>     file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunpkcs11.jar
>     file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/localedata.jar
>     file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunjce_provider.jar
>     file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar
>     file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/pulse-java.jar
>     file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/zipfs.jar
>     file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/dnsns.jar
>     file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper.so
>
>     Level 2: Java Classloader (loads everything defined by java
>     classpath) URL classpath items are:
>     file:/opt/accumulo-1.6.1/conf/
>     file:/opt/accumulo-1.6.1/lib/accumulo-start.jar
>     file:/opt/hadoop-1.2.1/lib/log4j-1.2.15.jar
>
>     Level 3: Accumulo Classloader (loads everything defined by
>     general.classpaths) URL classpath items are:
>     file:/opt/accumulo-1.6.1/lib/accumulo-core.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-start.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-fate.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-proxy.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-test.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-monitor.jar
>     file:/opt/accumulo-1.6.1/lib/jcommander.jar
>     file:/opt/accumulo-1.6.1/lib/jetty-server.jar
>     file:/opt/accumulo-1.6.1/lib/slf4j-log4j12.jar
>     file:/opt/accumulo-1.6.1/lib/jetty-util.jar
>     file:/opt/accumulo-1.6.1/lib/jetty-servlet.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-server-base.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-minicluster.jar
>     file:/opt/accumulo-1.6.1/lib/javax.servlet-api.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-gc.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-fate.jar
>     file:/opt/accumulo-1.6.1/lib/commons-math.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-core.jar
>     file:/opt/accumulo-1.6.1/lib/jline.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-trace.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-start.jar
>     file:/opt/accumulo-1.6.1/lib/gson.jar
>     file:/opt/accumulo-1.6.1/lib/jetty-http.jar
>     file:/opt/accumulo-1.6.1/lib/commons-vfs2.jar
>     file:/opt/accumulo-1.6.1/lib/libthrift.jar
>     file:/opt/accumulo-1.6.1/lib/jetty-security.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-tserver.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-tracer.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-examples-simple.jar
>     file:/opt/accumulo-1.6.1/lib/slf4j-api.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-master.jar
>     file:/opt/accumulo-1.6.1/lib/guava.jar
>     file:/opt/accumulo-1.6.1/lib/jetty-io.jar
>     file:/opt/accumulo-1.6.1/lib/jetty-continuation.jar
>     file:/opt/accumulo-1.6.1/lib/accumulo-proxy.jar
>     file:/opt/zookeeper-3.4.5/zookeeper-3.4.5.jar
>     file:/etc/hadoop/
>     file:/opt/hadoop-1.2.1/hadoop-test-1.2.1.jar
>     file:/opt/hadoop-1.2.1/hadoop-ant-1.2.1.jar
>     file:/opt/hadoop-1.2.1/hadoop-core-1.2.1.jar
>     file:/opt/hadoop-1.2.1/hadoop-minicluster-1.2.1.jar
>     file:/opt/hadoop-1.2.1/hadoop-tools-1.2.1.jar
>     file:/opt/hadoop-1.2.1/hadoop-examples-1.2.1.jar
>     file:/opt/hadoop-1.2.1/hadoop-client-1.2.1.jar
>     file:/opt/hadoop-1.2.1/lib/hadoop-fairscheduler-1.2.1.jar
>     file:/opt/hadoop-1.2.1/lib/commons-collections-3.2.1.jar
>     file:/opt/hadoop-1.2.1/lib/hadoop-thriftfs-1.2.1.jar
>     file:/opt/hadoop-1.2.1/lib/commons-configuration-1.6.jar
>     file:/opt/hadoop-1.2.1/lib/commons-logging-api-1.0.4.jar
>     file:/opt/hadoop-1.2.1/lib/commons-httpclient-3.0.1.jar
>     file:/opt/hadoop-1.2.1/lib/oro-2.0.8.jar
>     file:/opt/hadoop-1.2.1/lib/log4j-1.2.15.jar
>     file:/opt/hadoop-1.2.1/lib/commons-io-2.1.jar
>     file:/opt/hadoop-1.2.1/lib/commons-net-3.1.jar
>     file:/opt/hadoop-1.2.1/lib/xmlenc-0.52.jar
>     file:/opt/hadoop-1.2.1/lib/commons-el-1.0.jar
>     file:/opt/hadoop-1.2.1/lib/aspectjtools-1.6.11.jar
>     file:/opt/hadoop-1.2.1/lib/jdeb-0.8.jar
>     file:/opt/hadoop-1.2.1/lib/hsqldb-1.8.0.10.jar
>     file:/opt/hadoop-1.2.1/lib/jersey-core-1.8.jar
>     file:/opt/hadoop-1.2.1/lib/commons-logging-1.1.1.jar
>     file:/opt/hadoop-1.2.1/lib/commons-beanutils-core-1.8.0.jar
>     file:/opt/hadoop-1.2.1/lib/jsch-0.1.42.jar
>     file:/opt/hadoop-1.2.1/lib/commons-math-2.1.jar
>     file:/opt/hadoop-1.2.1/lib/commons-daemon-1.0.1.jar
>     file:/opt/hadoop-1.2.1/lib/kfs-0.2.2.jar
>     file:/opt/hadoop-1.2.1/lib/core-3.1.1.jar
>     file:/opt/hadoop-1.2.1/lib/jersey-json-1.8.jar
>     file:/opt/hadoop-1.2.1/lib/commons-lang-2.4.jar
>     file:/opt/hadoop-1.2.1/lib/aspectjrt-1.6.11.jar
>     file:/opt/hadoop-1.2.1/lib/jasper-runtime-5.5.12.jar
>     file:/opt/hadoop-1.2.1/lib/jetty-6.1.26.jar
>     file:/opt/hadoop-1.2.1/lib/servlet-api-2.5-20081211.jar
>     file:/opt/hadoop-1.2.1/lib/jackson-mapper-asl-1.8.8.jar
>     file:/opt/hadoop-1.2.1/lib/commons-digester-1.8.jar
>     file:/opt/hadoop-1.2.1/lib/hadoop-capacity-scheduler-1.2.1.jar
>     file:/opt/hadoop-1.2.1/lib/asm-3.2.jar
>     file:/opt/hadoop-1.2.1/lib/jersey-server-1.8.jar
>     file:/opt/hadoop-1.2.1/lib/jackson-core-asl-1.8.8.jar
>     file:/opt/hadoop-1.2.1/lib/commons-beanutils-1.7.0.jar
>     file:/opt/hadoop-1.2.1/lib/jasper-compiler-5.5.12.jar
>     file:/opt/hadoop-1.2.1/lib/jetty-util-6.1.26.jar
>     file:/opt/hadoop-1.2.1/lib/commons-cli-1.2.jar
>     file:/opt/hadoop-1.2.1/lib/commons-codec-1.4.jar
>     file:/opt/hadoop-1.2.1/lib/jets3t-0.6.1.jar
>     file:/opt/hadoop-1.2.1/lib/junit-4.5.jar
>     file:/opt/hadoop-1.2.1/lib/mockito-all-1.8.5.jar
>
>     Level 4: Accumulo Dynamic Classloader (loads everything defined by
>     general.dynamic.classpaths) VFS classpaths items are:
>
>
> Hope this helps you to understand what is going on my setup.
> Thanks,
> Salih
>
> Salih Kardan
>
> On Wed, Nov 19, 2014 at 5:34 PM, Josh Elser <josh.elser@gmail.com
> <mailto:josh.elser@gmail.com>> wrote:
>
>     Hi Salih,
>
>     It looks like the Accumulo processes are trying to communicate with
>     the local filesystem instead of HDFS. Do you still have the output
>     from `accumulo init`?
>
>     Also, is it possible to share your accumulo-env.sh,
>     accumulo-site.xml and the output from `accumulo classpath`?
>
>     Thanks.
>
>     Salih Kardan wrote:
>
>         Hello everyone,
>
>         Sorry in the previous mail, I forgot to fill out subject line of
>         mail.
>         Please ignore previous mail.
>
>         Currently I am testing Accumulo 1.6.1 with Hadoop 1.2.1 and
>         Zookeeper
>         3.4.5 with 4 hadoop nodes. I gave a role to each node in hadoop
>         cluster;
>         specifically
>         assume my hadoop nodes like this: hadoop1, hadoop2, hadoop3 and
>         hadoop4.
>
>         The accumulo roles are distributed to hadoop nodes;
>         hadoop1 = accumulo master ( single zookeeper instance is also
>         running on
>         this node)
>         hadoop2 = gc
>         hadoop3 = monitor
>         hadoop4 = tablet server + tracer
>
>         After I initialize accumulo with *"accumulo init"* command, I
>         invoked
>
>         start_all.sh script from master machine. All services except
>         tracer and
>         tablet server started working.
>         However on tablet server machine I see these logs :
>
>         2014-11-19 11:20:32,327 [zookeeper.ZooUtil] ERROR: unable obtain
>         instance id at file:/accumulo/instance_id
>         2014-11-19 11:20:32,328 [tserver.TabletServer] ERROR: Uncaught
>         exception
>         in TabletServer.main, exiting
>         java.lang.RuntimeException: Accumulo not initialized, there is no
>         instance id at file:/accumulo/instance_id
>               at
>         org.apache.accumulo.core.__zookeeper.ZooUtil.__getInstanceIDFromHdfs(ZooUtil.__java:62)
>               at
>         org.apache.accumulo.server.__client.HdfsZooInstance.___getInstanceID(HdfsZooInstance.__java:132)
>               at
>         org.apache.accumulo.server.__client.HdfsZooInstance.__getInstanceID(HdfsZooInstance.__java:116)
>               at
>         org.apache.accumulo.server.__conf.__ServerConfigurationFactory.<__init>(__ServerConfigurationFactory.__java:113)
>               at
>         org.apache.accumulo.server.__conf.ServerConfiguration.<__init>(ServerConfiguration.__java:79)
>               at
>         org.apache.accumulo.tserver.__TabletServer.main(__TabletServer.java:3668)
>               at sun.reflect.__NativeMethodAccessorImpl.__invoke0(Native
>         Method)
>               at
>         sun.reflect.__NativeMethodAccessorImpl.__invoke(__NativeMethodAccessorImpl.java:__57)
>               at
>         sun.reflect.__DelegatingMethodAccessorImpl.__invoke(__DelegatingMethodAccessorImpl.__java:43)
>               at java.lang.reflect.Method.__invoke(Method.java:606)
>               at org.apache.accumulo.start.__Main$1.run(Main.java:141)
>               at java.lang.Thread.run(Thread.__java:745)
>
>         And the logs I see in tracer log file:
>
>         Thread "tracer" died Accumulo not initialized, there is no
>         instance id
>         at file:/accumulo/instance_id
>         java.lang.RuntimeException: Accumulo not initialized, there is no
>         instance id at file:/accumulo/instance_id
>               at
>         org.apache.accumulo.core.__zookeeper.ZooUtil.__getInstanceIDFromHdfs(ZooUtil.__java:62)
>               at
>         org.apache.accumulo.server.__client.HdfsZooInstance.___getInstanceID(HdfsZooInstance.__java:132)
>               at
>         org.apache.accumulo.server.__client.HdfsZooInstance.__getInstanceID(HdfsZooInstance.__java:116)
>               at
>         org.apache.accumulo.server.__conf.__ServerConfigurationFactory.<__init>(__ServerConfigurationFactory.__java:113)
>               at
>         org.apache.accumulo.server.__conf.ServerConfiguration.<__init>(ServerConfiguration.__java:79)
>               at
>         org.apache.accumulo.tracer.__TraceServer.main(TraceServer.__java:290)
>               at sun.reflect.__NativeMethodAccessorImpl.__invoke0(Native
>         Method)
>               at
>         sun.reflect.__NativeMethodAccessorImpl.__invoke(__NativeMethodAccessorImpl.java:__57)
>               at
>         sun.reflect.__DelegatingMethodAccessorImpl.__invoke(__DelegatingMethodAccessorImpl.__java:43)
>               at java.lang.reflect.Method.__invoke(Method.java:606)
>               at org.apache.accumulo.start.__Main$1.run(Main.java:141)
>               at java.lang.Thread.run(Thread.__java:745)
>
>
>         I checked hdfs and it seems accumulo is initialized. Here is the
>         output
>         of *"hadoop dfs -ls /accumulo/instance_id" *command*, *
>
>         Found 1 items
>         -rw-r--r--   1 root supergroup          0 2014-11-19 11:19
>         /accumulo/instance_id/__268acc40-e20b-4a35-8d8a-__0e46e7859a0d
>
>         I googled the problem, some comments stating that the problem
>         may occur
>         due to missing hadoop libs in classpath, but I checked classpath of
>         with *"accumulo classpath"*  command, it also seems correct,
>         both hadoop
>         and zookeeper libs listed in classspath.
>
>
>         Then I tried single node accumulo installation, that way all
>         services
>         including tablet server seems working.
>         What can be the problem when I use multiple nodes? Any help is
>         appreciated.
>
>         Thanks
>
>