accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-2154) NoNodeException error in master
Date Tue, 21 Jan 2014 16:13:19 GMT


Keith Turner commented on ACCUMULO-2154:

I realize that synchronization will fix this if we assume only one process and one object
in that process access this resource.  But simply catching and ignoring no-node exception
will also fix the problem w/o those assumptions.   Synchronization is great when the resource
being protected is private process memory, however thats not true in this case.  ZooKeeper
is a cluster wide resource and its possible that any other process in the cluster could mutate
zookeeper at any time.  The way I see there are at least three options to solve this problem.

 # use java synchronization with assumptions stated above
 # use zookeeper primitives for dealing with concurrency
 # use java synchronization and zookeeper  primitives for dealing with concurrency

I am in favor of #2.   And its also very simple in this case, just ignore NoNodeException
because it indicates the node was deleted after the call to getChildren() was made.   
bq.  I've made post() also synchronized so that getList() doesn't miss any dead servers that
get added after zoo.getChildren() is called.

It will still miss those servers.  When synchronized, If one thread is in getList() then another
thread calling post() will block.   Detecting changes after getChildren is called is not needed,
just need a consistent snapshot at a point in time.  It could be achieved by checking getChildren
in a loop and waiting for it stabilize.   But the data could still  be outdated by other operations
immediately after getList() returns, so the code still has to treat it as a snapshot.   Anything
more would require some sort of transaction semantics across method calls, which is not needed.

> NoNodeException error in master
> -------------------------------
>                 Key: ACCUMULO-2154
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: master
>         Environment: 1.6.0 sha 417902e218c566333b6ea5ac492186ae305e5e16
>            Reporter: John Vines
>            Assignee: Vikram Srivastava
>              Labels: PatchAvailable
>             Fix For: 1.6.0
>         Attachments: ACCUMULO-2154.v1.patch.txt
> I have a test that brings accumulo down hard after a minute and then brings it back up
again. I was running it overnight and I saw this stack trace once. Not sure if it's a problem
or not though.
> {code}org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
for /accumulo/617ee3a7-98b9-4f5f-af13-8894afe7c33c/dead/tservers/
> 	org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /accumulo/617ee3a7-98b9-4f5f-af13-8894afe7c33c/dead/tservers/
> 		at org.apache.zookeeper.KeeperException.create(
> 		at org.apache.zookeeper.KeeperException.create(
> 		at org.apache.zookeeper.ZooKeeper.getData(
> 		at org.apache.zookeeper.ZooKeeper.getData(
> 		at org.apache.accumulo.fate.zookeeper.ZooReader.getData(
> 		at org.apache.accumulo.server.master.state.DeadServerList.getList(
> 		at org.apache.accumulo.master.MasterClientServiceHandler.getMasterStats(
> 		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 		at sun.reflect.NativeMethodAccessorImpl.invoke(
> 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> 		at java.lang.reflect.Method.invoke(
> 		at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(
> 		at com.sun.proxy.$Proxy11.getMasterStats(Unknown Source)
> 		at org.apache.accumulo.core.master.thrift.MasterClientService$Processor$getMasterStats.getResult(
> 		at org.apache.accumulo.core.master.thrift.MasterClientService$Processor$getMasterStats.getResult(
> 		at org.apache.thrift.ProcessFunction.process(
> 		at org.apache.thrift.TBaseProcessor.process(
> 		at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(
> 		at org.apache.thrift.server.TThreadPoolServer$
> 		at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> 		at java.util.concurrent.ThreadPoolExecutor$
> 		at{code}

This message was sent by Atlassian JIRA

View raw message