accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Vines <vi...@apache.org>
Subject Re: My Accumulo 1.5.0 instance has no tablet servers
Date Fri, 13 Sep 2013 14:14:21 GMT
Those are from the tracer. In the accumulo-site.xml, change
the trace.token.property.password value to your root password. Or you can
create a new user in Accumulo and update that and trace.user field
appropriately.


On Thu, Sep 12, 2013 at 3:43 PM, John Vines <jvines@gmail.com> wrote:

> Those are from the tracer. In the accumulo-site.xml, change
> the trace.token.property.password value to your root password. Or you can
> create a new user in Accumulo and update that and trace.user field
> appropriately.
>
>
> On Thu, Sep 12, 2013 at 3:01 PM, Pete Carlson <pgcarlson@gmail.com> wrote:
>
>> Thanks Eric, that worked.  I now see in the monitor that I have 1 Tablet
>> Server.
>>
>> Under the monitor log i.e., http://127.0.0.1:50095/log the warn and
>> fatal messages went away, and now I have the following error:
>>
>> tserver:localhost.localdomain 115 ERROR
>> ThriftSecurityException(user:root, code:BAD_CREDENTIALS)
>>
>> and I also see these same errors in the tserver_localhost.localdomain.log
>> (here's a snippet):
>>
>> 2013-09-12 14:56:29,985 [client.ClientServiceHandler] ERROR:
>> ThriftSecurityException(user:root, code:BAD_CREDENTIALS)
>> 2013-09-12 14:56:30,996 [client.ClientServiceHandler] ERROR:
>> ThriftSecurityException(user:root, code:BAD_CREDENTIALS)
>> 2013-09-12 14:56:32,008 [client.ClientServiceHandler] ERROR:
>> ThriftSecurityException(user:root, code:BAD_CREDENTIALS)
>> 2013-09-12 14:56:33,021 [client.ClientServiceHandler] ERROR:
>> ThriftSecurityException(user:root, code:BAD_CREDENTIALS)
>>
>> Do I have to do a similar configuration for Thrift that I just did for
>> Hadoop?
>>
>> Pete
>>
>> On Thu, Sep 12, 2013 at 2:31 PM, Eric Newton <eric.newton@gmail.com>wrote:
>>
>>> Add:
>>>
>>>   <property>
>>>       <name>dfs.support.append</name>
>>>       <value>true</value>
>>>   </property>
>>>   <property>
>>>       <name>dfs.data.synconclose</name>
>>>       <value>true</value>
>>>   </property>
>>>
>>> To hdfs-site.xml in your hadoop configuration.
>>>
>>> -Eric
>>>
>>>
>>>
>>> On Thu, Sep 12, 2013 at 2:27 PM, Pete Carlson <pgcarlson@gmail.com>wrote:
>>>
>>>> Ok, so now that I have an Accumulo monitor I discovered that my
>>>> Accumulo instance doesn't have any tablet servers.
>>>>
>>>> Here is what I tried so far to resolve the issue:
>>>>
>>>> 1) Looked in the tserver_localhost.localdomain.log file, and found
>>>> this FATAL message:
>>>>
>>>> 2013-09-12 08:09:42,273 [tabletserver.TabletServer] FATAL: Must set
>>>> dfs.durable.sync OR dfs.support.append to true.  Which one needs to be set
>>>> depends on your version of HDFS.  See ACCUMULO-623.
>>>> HADOOP RELEASE          VERSION           SYNC NAME             DEFAULT
>>>> Apache Hadoop           0.20.205          dfs.support.append    false
>>>> Apache Hadoop            0.23.x           dfs.support.append    true
>>>> Apache Hadoop             1.0.x           dfs.support.append    false
>>>> Apache Hadoop             1.1.x           dfs.durable.sync      true
>>>> Apache Hadoop          2.0.0-2.0.2        dfs.support.append    true
>>>> Cloudera CDH             3u0-3u3             ????               true
>>>> Cloudera CDH               3u4            dfs.support.append    true
>>>> Hortonworks HDP           `1.0            dfs.support.append    false
>>>> Hortonworks HDP           `1.1            dfs.support.append    false
>>>> 2013-09-12 11:54:00,752 [server.Accumulo] INFO : tserver starting
>>>> 2013-09-12 11:54:00,768 [server.Accumulo] INFO : Instance
>>>> d57cdc38-8ceb-4192-9da3-1ce2664df33b
>>>> 2013-09-12 11:54:00,771 [server.Accumulo] INFO : Data Version 5
>>>> 2013-09-12 11:54:00,771 [server.Accumulo] INFO : Attempting to talk to
>>>> zookeeper
>>>> 2013-09-12 11:54:00,952 [server.Accumulo] INFO : Zookeeper connected
>>>> and initialized, attemping to talk to HDFS
>>>> 2013-09-12 11:54:00,956 [server.Accumulo] INFO : Connected to HDFS
>>>> 2013-09-12 11:54:00,969 [server.Accumulo] INFO : gc.cycle.delay = 5m
>>>> 2013-09-12 11:54:00,969 [server.Accumulo] INFO : gc.cycle.start = 30s
>>>> 2013-09-12 11:54:00,969 [server.Accumulo] INFO : gc.port.client = 50091
>>>> 2013-09-12 11:54:00,969 [server.Accumulo] INFO : gc.threads.delete = 16
>>>> 2013-09-12 11:54:00,969 [server.Accumulo] INFO : gc.trash.ignore = false
>>>>
>>>> I saw this same FATAL message 8 times in the tserver_localhost.localdomain.log
>>>> between blocks of INFO messages, but no other fatal or warn messages.
>>>> Btw, this FATAL message also appears in my
>>>> tserver_localhost.localdomain.debug.log file.
>>>>
>>>> When I googled this Fatal message I found this page:
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/accumulo-user/201304.mbox/%3C515F5518.1090703@gmail.com%3E
with
>>>> the same "WARN: There are no tablet servers: check that zookeeper and
>>>> accumulo are running." message.
>>>>
>>>> I checked http://127.0.0.1:50095/tservers, and it showed that there
>>>> were no tablet servers online. I looked at http://127.0.0.1:50095/log,
>>>> and saw the following messages:
>>>>
>>>> FATAL: Must set dfs.durable.sync or dfs.support.append to true. Which
>>>> one needs to be set depends on your version of HDFS. See Accumulo-623.
>>>>
>>>> WARN: There are no tablet servers: check that zookeeper and accumulo
>>>> are running.
>>>>
>>>> Using the info from the page I referenced above, I checked my
>>>> $ACCUMULO_HOME path and realized that I hadn't set that in the
>>>> conf/accumulo-env.sh
>>>>
>>>> So, I set it to the following:
>>>>
>>>> test -z "$ACCUMULO_HOME" && export
>>>> ACCUMULO_HOME=/home/accumulo/accumulo-1.5.0
>>>>
>>>> When I did an echo of $ACCUMULO_HOME it didn't return anything, so I
>>>> also tried setting it in my bash profile to see if that made any difference
>>>> (it didn't).
>>>>
>>>> I also looked in the lib directory but didn't see any stray jars.
>>>>
>>>> In my tracer_localhost_localdomain.log I saw the following Exception
>>>> with Zookeeper:
>>>>
>>>> 2013-09-11 16:09:48,649 [impl.ServerClient] WARN : There are no tablet
>>>> servers: check that zookeeper and accumulo are running.
>>>> 2013-09-11 18:02:23,385 [zookeeper.ZooCache] WARN : Zookeeper error,
>>>> will retry
>>>> org.apache.zookeeper.KeeperException$SessionExpiredException:
>>>> KeeperErrorCode = Session expired for
>>>> /accumulo/d57cdc38-8ceb-4192-9da3-1ce2664df33b/tservers
>>>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>>>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>>>> at org.apache.accumulo.fate.zookeeper.ZooCache$1.run(ZooCache.java:167)
>>>> at org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:130)
>>>> at
>>>> org.apache.accumulo.fate.zookeeper.ZooCache.getChildren(ZooCache.java:178)
>>>> at
>>>> org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClient.java:140)
>>>> at
>>>> org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClient.java:128)
>>>> at
>>>> org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClient.java:123)
>>>> at
>>>> org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.java:105)
>>>> at
>>>> org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:71)
>>>> at
>>>> org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:64)
>>>> at
>>>> org.apache.accumulo.server.client.HdfsZooInstance.getConnector(HdfsZooInstance.java:154)
>>>> at
>>>> org.apache.accumulo.server.client.HdfsZooInstance.getConnector(HdfsZooInstance.java:149)
>>>> at
>>>> org.apache.accumulo.server.trace.TraceServer.<init>(TraceServer.java:185)
>>>> at
>>>> org.apache.accumulo.server.trace.TraceServer.main(TraceServer.java:260)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at org.apache.accumulo.start.Main$1.run(Main.java:101)
>>>> at java.lang.Thread.run(Thread.java:724)
>>>> 2013-09-12 08:09:44,861 [server.Accumulo] INFO : tracer starting
>>>> 2013-09-12 08:09:44,926 [server.Accumulo] INFO : Instance
>>>> d57cdc38-8ceb-4192-9da3-1ce2664df33b
>>>> 2013-09-12 08:09:44,929 [server.Accumulo] INFO : Data Version 5
>>>> 2013-09-12 08:09:44,929 [server.Accumulo] INFO : Attempting to talk to
>>>> zookeeper
>>>> 2013-09-12 08:09:45,114 [server.Accumulo] INFO : Zookeeper connected
>>>> and initialized, attemping to talk to HDFS
>>>> 2013-09-12 08:09:45,130 [server.Accumulo] INFO : Connected to HDFS
>>>> 2013-09-12 08:09:45,150 [server.Accumulo] INFO : gc.cycle.delay = 5m
>>>> 2013-09-12 08:09:45,150 [server.Accumulo] INFO : gc.cycle.start = 30s
>>>>
>>>> but then it appeared to reconnect with Zookeeper.
>>>>
>>>> 2) I looked at the ACCUMULO-623 Jira ticket from the FATAL message
>>>> above i.e., https://issues.apache.org/jira/browse/ACCUMULO-623 , but
>>>> this Jira ticket indicates this issue is fixed in Accumulo 1.5.0 although
>>>> that ticket references Hadoop 1.0.3, and Zookeeper 3.3.3  (I'm using Hadoop
>>>> 1.2.1, and Zookeeper 3.4.5)  I noticed that a fix was added to Hadoop 1.1
>>>> for a related Hadoop Jira ticket.
>>>>
>>>> 3) Next, I went to the Accumulo Jira page i.e.,
>>>> https://issues.apache.org/jira/browse/accumulo to look for this
>>>> issue.  Besides ACCUMULO-623, the following tickets are similar but not
>>>> quite the same:
>>>>
>>>>    - ACCUMULO-327 ( but I don't have any tablet servers to begin with
>>>>    to be killed)
>>>>    - ACCUMULO-1235 (I only have a the default !METADATA table)
>>>>
>>>> 4) Looked again at the User manual to see if there was information
>>>> about configuring the tablet server, but didn't see anything.
>>>>
>>>> Any suggestions on what I should try next?
>>>>
>>>> Thanks,
>>>>
>>>> Pete
>>>>
>>>
>>>
>>
>
>
> --
> Cheers
> ~John
>

Mime
View raw message