accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Salih Kardan <karda...@gmail.com>
Subject Re: Tablet server and tracer could get started since they cannot see initialized accumulo instance
Date Wed, 19 Nov 2014 18:29:33 GMT
Hi Josh,

Thanks for the prompt response, it is very good for Accumulo users to have
such a responsive community like this.

I have reproduced the problem and fetched the logs you want.

Here is the output of *accumulo init *command :


2014-11-19 20:14:30,583 [fs.VolumeManagerImpl] WARN :
dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is
possible on system reset or power loss
2014-11-19 20:14:30,584 [init.Initialize] INFO : Hadoop Filesystem is
hdfs://hadoop35:8020
2014-11-19 20:14:30,585 [init.Initialize] INFO : Accumulo data dirs are
[hdfs://hadoop35:8020/accumulo]
2014-11-19 20:14:30,585 [init.Initialize] INFO : Zookeeper server is
hadoop33:2181
2014-11-19 20:14:30,585 [init.Initialize] INFO : Checking if Zookeeper is
available. If this hangs, then you need to make sure zookeeper is running

Warning!!! Your instance secret is still set to the default, this is not
secure. We highly recommend you change it.

You can change the instance secret in accumulo by using:
   bin/accumulo org.apache.accumulo.server.util.ChangeSecret oldPassword
newPassword.
You will also need to edit your secret in your configuration file by adding
the property instance.secret to your conf/accumulo-site.xml. Without this
accumulo will not operate correctly
Instance name : test
Instance name "test" exists. Delete existing entry from zookeeper? [Y/N] : Y
Enter initial password for root (this may not be applicable for your
security setup): ****
Confirm initial password for root: ****
2014-11-19 20:14:38,774 [fs.VolumeManagerImpl] WARN :
dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is
possible on system reset or power loss
2014-11-19 20:14:41,128 [util.NativeCodeLoader] WARN : Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2014-11-19 20:14:41,214 [conf.AccumuloConfiguration] INFO : Loaded class :
org.apache.accumulo.server.security.handler.ZKAuthorizor
2014-11-19 20:14:41,216 [conf.AccumuloConfiguration] INFO : Loaded class :
org.apache.accumulo.server.security.handler.ZKAuthenticator
2014-11-19 20:14:41,217 [conf.AccumuloConfiguration] INFO : Loaded class :
org.apache.accumulo.server.security.handler.ZKPermHandler



Also here is the *accumulo-env.sh*:


#! /usr/bin/env bash

if [ -z "$HADOOP_HOME" ]
then
   test -z "$HADOOP_PREFIX"      && export HADOOP_PREFIX=/opt/hadoop-1.2.1
else
   HADOOP_PREFIX="$HADOOP_HOME"
   unset HADOOP_HOME
fi

# hadoop-1.2:
# test -z "$HADOOP_CONF_DIR"       && export
HADOOP_CONF_DIR="$HADOOP_PREFIX/conf"
# hadoop-2.0:
test -z "$HADOOP_CONF_DIR"     && export
HADOOP_CONF_DIR="$HADOOP_PREFIX/etc/hadoop"

test -z "$JAVA_HOME"             && export
JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64
test -z "$ZOOKEEPER_HOME"        && export
ZOOKEEPER_HOME=/opt/zookeeper-3.4.5
test -z "$ACCUMULO_LOG_DIR"      && export
ACCUMULO_LOG_DIR=/opt/accumulo-1.6.1/log
if [ -f ${ACCUMULO_CONF_DIR}/accumulo.policy ]
then
   POLICY="-Djava.security.manager
-Djava.security.policy=${ACCUMULO_CONF_DIR}/accumulo.policy"
fi
test -z "$ACCUMULO_TSERVER_OPTS" && export ACCUMULO_TSERVER_OPTS="${POLICY}
-Xmx128m -Xms128m "
test -z "$ACCUMULO_MASTER_OPTS"  && export ACCUMULO_MASTER_OPTS="${POLICY}
-Xmx128m -Xms128m"
test -z "$ACCUMULO_MONITOR_OPTS" && export ACCUMULO_MONITOR_OPTS="${POLICY}
-Xmx64m -Xms64m"
test -z "$ACCUMULO_GC_OPTS"      && export ACCUMULO_GC_OPTS="-Xmx64m
-Xms64m"
test -z "$ACCUMULO_GENERAL_OPTS" && export
ACCUMULO_GENERAL_OPTS="-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -Djava.net.preferIPv4Stack=true"
test -z "$ACCUMULO_OTHER_OPTS"   && export ACCUMULO_OTHER_OPTS="-Xmx128m
-Xms64m"
# what do when the JVM runs out of heap memory
export ACCUMULO_KILL_CMD='kill -9 %p'

### Optionally look for hadoop and accumulo native libraries for your
### platform in additional directories. (Use DYLD_LIBRARY_PATH on Mac OS X.)
### May not be necessary for Hadoop 2.x or using an RPM that installs to
### the correct system library directory.
# export
LD_LIBRARY_PATH=${HADOOP_PREFIX}/lib/native/${PLATFORM}:${LD_LIBRARY_PATH}

# Should the monitor bind to all network interfaces -- default: false
# export ACCUMULO_MONITOR_BIND_ALL="true"



And finally output of *accumulo classpath *command:

Level 1: Java System Classloader (loads Java system resources) URL
classpath items are:
file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunpkcs11.jar
file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/localedata.jar
file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunjce_provider.jar
file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar
file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/pulse-java.jar
file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/zipfs.jar
file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/dnsns.jar
file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper.so

Level 2: Java Classloader (loads everything defined by java classpath) URL
classpath items are:
file:/opt/accumulo-1.6.1/conf/
file:/opt/accumulo-1.6.1/lib/accumulo-start.jar
file:/opt/hadoop-1.2.1/lib/log4j-1.2.15.jar

Level 3: Accumulo Classloader (loads everything defined by
general.classpaths) URL classpath items are:
file:/opt/accumulo-1.6.1/lib/accumulo-core.jar
file:/opt/accumulo-1.6.1/lib/accumulo-start.jar
file:/opt/accumulo-1.6.1/lib/accumulo-fate.jar
file:/opt/accumulo-1.6.1/lib/accumulo-proxy.jar
file:/opt/accumulo-1.6.1/lib/accumulo-test.jar
file:/opt/accumulo-1.6.1/lib/accumulo-monitor.jar
file:/opt/accumulo-1.6.1/lib/jcommander.jar
file:/opt/accumulo-1.6.1/lib/jetty-server.jar
file:/opt/accumulo-1.6.1/lib/slf4j-log4j12.jar
file:/opt/accumulo-1.6.1/lib/jetty-util.jar
file:/opt/accumulo-1.6.1/lib/jetty-servlet.jar
file:/opt/accumulo-1.6.1/lib/accumulo-server-base.jar
file:/opt/accumulo-1.6.1/lib/accumulo-minicluster.jar
file:/opt/accumulo-1.6.1/lib/javax.servlet-api.jar
file:/opt/accumulo-1.6.1/lib/accumulo-gc.jar
file:/opt/accumulo-1.6.1/lib/accumulo-fate.jar
file:/opt/accumulo-1.6.1/lib/commons-math.jar
file:/opt/accumulo-1.6.1/lib/accumulo-core.jar
file:/opt/accumulo-1.6.1/lib/jline.jar
file:/opt/accumulo-1.6.1/lib/accumulo-trace.jar
file:/opt/accumulo-1.6.1/lib/accumulo-start.jar
file:/opt/accumulo-1.6.1/lib/gson.jar
file:/opt/accumulo-1.6.1/lib/jetty-http.jar
file:/opt/accumulo-1.6.1/lib/commons-vfs2.jar
file:/opt/accumulo-1.6.1/lib/libthrift.jar
file:/opt/accumulo-1.6.1/lib/jetty-security.jar
file:/opt/accumulo-1.6.1/lib/accumulo-tserver.jar
file:/opt/accumulo-1.6.1/lib/accumulo-tracer.jar
file:/opt/accumulo-1.6.1/lib/accumulo-examples-simple.jar
file:/opt/accumulo-1.6.1/lib/slf4j-api.jar
file:/opt/accumulo-1.6.1/lib/accumulo-master.jar
file:/opt/accumulo-1.6.1/lib/guava.jar
file:/opt/accumulo-1.6.1/lib/jetty-io.jar
file:/opt/accumulo-1.6.1/lib/jetty-continuation.jar
file:/opt/accumulo-1.6.1/lib/accumulo-proxy.jar
file:/opt/zookeeper-3.4.5/zookeeper-3.4.5.jar
file:/etc/hadoop/
file:/opt/hadoop-1.2.1/hadoop-test-1.2.1.jar
file:/opt/hadoop-1.2.1/hadoop-ant-1.2.1.jar
file:/opt/hadoop-1.2.1/hadoop-core-1.2.1.jar
file:/opt/hadoop-1.2.1/hadoop-minicluster-1.2.1.jar
file:/opt/hadoop-1.2.1/hadoop-tools-1.2.1.jar
file:/opt/hadoop-1.2.1/hadoop-examples-1.2.1.jar
file:/opt/hadoop-1.2.1/hadoop-client-1.2.1.jar
file:/opt/hadoop-1.2.1/lib/hadoop-fairscheduler-1.2.1.jar
file:/opt/hadoop-1.2.1/lib/commons-collections-3.2.1.jar
file:/opt/hadoop-1.2.1/lib/hadoop-thriftfs-1.2.1.jar
file:/opt/hadoop-1.2.1/lib/commons-configuration-1.6.jar
file:/opt/hadoop-1.2.1/lib/commons-logging-api-1.0.4.jar
file:/opt/hadoop-1.2.1/lib/commons-httpclient-3.0.1.jar
file:/opt/hadoop-1.2.1/lib/oro-2.0.8.jar
file:/opt/hadoop-1.2.1/lib/log4j-1.2.15.jar
file:/opt/hadoop-1.2.1/lib/commons-io-2.1.jar
file:/opt/hadoop-1.2.1/lib/commons-net-3.1.jar
file:/opt/hadoop-1.2.1/lib/xmlenc-0.52.jar
file:/opt/hadoop-1.2.1/lib/commons-el-1.0.jar
file:/opt/hadoop-1.2.1/lib/aspectjtools-1.6.11.jar
file:/opt/hadoop-1.2.1/lib/jdeb-0.8.jar
file:/opt/hadoop-1.2.1/lib/hsqldb-1.8.0.10.jar
file:/opt/hadoop-1.2.1/lib/jersey-core-1.8.jar
file:/opt/hadoop-1.2.1/lib/commons-logging-1.1.1.jar
file:/opt/hadoop-1.2.1/lib/commons-beanutils-core-1.8.0.jar
file:/opt/hadoop-1.2.1/lib/jsch-0.1.42.jar
file:/opt/hadoop-1.2.1/lib/commons-math-2.1.jar
file:/opt/hadoop-1.2.1/lib/commons-daemon-1.0.1.jar
file:/opt/hadoop-1.2.1/lib/kfs-0.2.2.jar
file:/opt/hadoop-1.2.1/lib/core-3.1.1.jar
file:/opt/hadoop-1.2.1/lib/jersey-json-1.8.jar
file:/opt/hadoop-1.2.1/lib/commons-lang-2.4.jar
file:/opt/hadoop-1.2.1/lib/aspectjrt-1.6.11.jar
file:/opt/hadoop-1.2.1/lib/jasper-runtime-5.5.12.jar
file:/opt/hadoop-1.2.1/lib/jetty-6.1.26.jar
file:/opt/hadoop-1.2.1/lib/servlet-api-2.5-20081211.jar
file:/opt/hadoop-1.2.1/lib/jackson-mapper-asl-1.8.8.jar
file:/opt/hadoop-1.2.1/lib/commons-digester-1.8.jar
file:/opt/hadoop-1.2.1/lib/hadoop-capacity-scheduler-1.2.1.jar
file:/opt/hadoop-1.2.1/lib/asm-3.2.jar
file:/opt/hadoop-1.2.1/lib/jersey-server-1.8.jar
file:/opt/hadoop-1.2.1/lib/jackson-core-asl-1.8.8.jar
file:/opt/hadoop-1.2.1/lib/commons-beanutils-1.7.0.jar
file:/opt/hadoop-1.2.1/lib/jasper-compiler-5.5.12.jar
file:/opt/hadoop-1.2.1/lib/jetty-util-6.1.26.jar
file:/opt/hadoop-1.2.1/lib/commons-cli-1.2.jar
file:/opt/hadoop-1.2.1/lib/commons-codec-1.4.jar
file:/opt/hadoop-1.2.1/lib/jets3t-0.6.1.jar
file:/opt/hadoop-1.2.1/lib/junit-4.5.jar
file:/opt/hadoop-1.2.1/lib/mockito-all-1.8.5.jar

Level 4: Accumulo Dynamic Classloader (loads everything defined by
general.dynamic.classpaths) VFS classpaths items are:


Hope this helps you to understand what is going on my setup.
Thanks,
Salih

Salih Kardan

On Wed, Nov 19, 2014 at 5:34 PM, Josh Elser <josh.elser@gmail.com> wrote:

> Hi Salih,
>
> It looks like the Accumulo processes are trying to communicate with the
> local filesystem instead of HDFS. Do you still have the output from
> `accumulo init`?
>
> Also, is it possible to share your accumulo-env.sh, accumulo-site.xml and
> the output from `accumulo classpath`?
>
> Thanks.
>
> Salih Kardan wrote:
>
>> Hello everyone,
>>
>> Sorry in the previous mail, I forgot to fill out subject line of mail.
>> Please ignore previous mail.
>>
>> Currently I am testing Accumulo 1.6.1 with Hadoop 1.2.1 and Zookeeper
>> 3.4.5 with 4 hadoop nodes. I gave a role to each node in hadoop cluster;
>> specifically
>> assume my hadoop nodes like this: hadoop1, hadoop2, hadoop3 and hadoop4.
>>
>> The accumulo roles are distributed to hadoop nodes;
>> hadoop1 = accumulo master ( single zookeeper instance is also running on
>> this node)
>> hadoop2 = gc
>> hadoop3 = monitor
>> hadoop4 = tablet server + tracer
>>
>> After I initialize accumulo with *"accumulo init"* command, I invoked
>>
>> start_all.sh script from master machine. All services except tracer and
>> tablet server started working.
>> However on tablet server machine I see these logs :
>>
>> 2014-11-19 11:20:32,327 [zookeeper.ZooUtil] ERROR: unable obtain
>> instance id at file:/accumulo/instance_id
>> 2014-11-19 11:20:32,328 [tserver.TabletServer] ERROR: Uncaught exception
>> in TabletServer.main, exiting
>> java.lang.RuntimeException: Accumulo not initialized, there is no
>> instance id at file:/accumulo/instance_id
>>      at
>> org.apache.accumulo.core.zookeeper.ZooUtil.getInstanceIDFromHdfs(ZooUtil.
>> java:62)
>>      at
>> org.apache.accumulo.server.client.HdfsZooInstance._
>> getInstanceID(HdfsZooInstance.java:132)
>>      at
>> org.apache.accumulo.server.client.HdfsZooInstance.
>> getInstanceID(HdfsZooInstance.java:116)
>>      at
>> org.apache.accumulo.server.conf.ServerConfigurationFactory.<init>(
>> ServerConfigurationFactory.java:113)
>>      at
>> org.apache.accumulo.server.conf.ServerConfiguration.<
>> init>(ServerConfiguration.java:79)
>>      at
>> org.apache.accumulo.tserver.TabletServer.main(TabletServer.java:3668)
>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>      at
>> sun.reflect.NativeMethodAccessorImpl.invoke(
>> NativeMethodAccessorImpl.java:57)
>>      at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>>      at java.lang.reflect.Method.invoke(Method.java:606)
>>      at org.apache.accumulo.start.Main$1.run(Main.java:141)
>>      at java.lang.Thread.run(Thread.java:745)
>>
>> And the logs I see in tracer log file:
>>
>> Thread "tracer" died Accumulo not initialized, there is no instance id
>> at file:/accumulo/instance_id
>> java.lang.RuntimeException: Accumulo not initialized, there is no
>> instance id at file:/accumulo/instance_id
>>      at
>> org.apache.accumulo.core.zookeeper.ZooUtil.getInstanceIDFromHdfs(ZooUtil.
>> java:62)
>>      at
>> org.apache.accumulo.server.client.HdfsZooInstance._
>> getInstanceID(HdfsZooInstance.java:132)
>>      at
>> org.apache.accumulo.server.client.HdfsZooInstance.
>> getInstanceID(HdfsZooInstance.java:116)
>>      at
>> org.apache.accumulo.server.conf.ServerConfigurationFactory.<init>(
>> ServerConfigurationFactory.java:113)
>>      at
>> org.apache.accumulo.server.conf.ServerConfiguration.<
>> init>(ServerConfiguration.java:79)
>>      at org.apache.accumulo.tracer.TraceServer.main(TraceServer.java:290)
>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>      at
>> sun.reflect.NativeMethodAccessorImpl.invoke(
>> NativeMethodAccessorImpl.java:57)
>>      at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>>      at java.lang.reflect.Method.invoke(Method.java:606)
>>      at org.apache.accumulo.start.Main$1.run(Main.java:141)
>>      at java.lang.Thread.run(Thread.java:745)
>>
>>
>> I checked hdfs and it seems accumulo is initialized. Here is the output
>> of *"hadoop dfs -ls /accumulo/instance_id" *command*, *
>>
>> Found 1 items
>> -rw-r--r--   1 root supergroup          0 2014-11-19 11:19
>> /accumulo/instance_id/268acc40-e20b-4a35-8d8a-0e46e7859a0d
>>
>> I googled the problem, some comments stating that the problem may occur
>> due to missing hadoop libs in classpath, but I checked classpath of
>> with *"accumulo classpath"*  command, it also seems correct, both hadoop
>> and zookeeper libs listed in classspath.
>>
>>
>> Then I tried single node accumulo installation, that way all services
>> including tablet server seems working.
>> What can be the problem when I use multiple nodes? Any help is
>> appreciated.
>>
>> Thanks
>>
>>

Mime
View raw message