accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Installing Accumulo in multinode setup
Date Fri, 07 Oct 2016 15:17:11 GMT
Did you re-initialize Accumulo after nuking the ZK entries?

Yamini Joshi wrote:
> I restarted my zookeeper to get rid of all the accumulo entries in the
> zookeeper. Now I am getting this error in my monitor debug log:
>
> 2016-10-07 10:07:46,548 [impl.ThriftScanner] DEBUG:  Failed to locate
> tablet for table : !0 row : ~err_\00
> 2016-10-07 10:07:49,781 [monitor.Monitor] INFO :  Failed to obtain
> problem reports
> java.lang.RuntimeException:
> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
>      at
> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:161)
>      at
> org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252)
>      at
> org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310)
>      at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:371)
>      at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:511)
>      at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>      at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
>      at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:230)
>      at
> org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:80)
>      at
> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:151)
>      ... 6 more
>
> Best regards,
> Yamini Joshi
>
> On Fri, Oct 7, 2016 at 10:08 AM, Sean Busbey <busbey@cloudera.com
> <mailto:busbey@cloudera.com>> wrote:
>
>     tracers used to be under the instance id in 1.6. I can confirm that in
>     1.7.2 it is definitely at /tracers. I just ran into this issue last
>     week.
>
>     @Yamini where do you get the Node does not exist error? Could you
>     copy/paste the whole stacktrace? When I followed the same workaround
>     Josh mentioned, restarting the Trace servers was sufficient to have
>     the znode recreated.
>
>     On Fri, Oct 7, 2016 at 10:01 AM, Josh Elser <josh.elser@gmail.com
>     <mailto:josh.elser@gmail.com>> wrote:
>      > I had pulled up 1.7.2 and I thought /tracers was at the root of
>     ZooKeeper
>      > (not within the /accumulo/$instance.id <http://instance.id>
>     znode). Maybe I looked at the wrong
>      > code though?
>      >
>      > Michael Wall wrote:
>      >>
>      >> Maybe this is already known, so apologies in advance.  But don't you
>      >> need to do
>      >>
>      >> $ $ZOOKEEPER_HOME/bin/zkCli.sh -server m4:2181,m5:2181,m6:2181
>      >>>
>      >>>  rmr accumulo/WHATEVER-INSTANCE-ID/tracers
>      >>
>      >>
>      >> instead of rmr /tracers
>      >>
>      >> If you don't know the instance id, I you can do the following in
>     zookeeper
>      >>
>      >> get /accumulo/instances/INSTANCE_NAME
>      >>
>      >>
>      >>
>      >> On Thu, Oct 6, 2016 at 6:26 PM, Josh Elser <josh.elser@gmail.com
>     <mailto:josh.elser@gmail.com>
>      >> <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>>
wrote:
>      >>
>      >>     Uh, I am confused now. /tracers/ is the default parent znode
>     that
>      >>     the Accumulo Tracer process registers itself in. We can also see
>      >>     this is what was happening with your snippet.
>      >>
>      >>     I don't understand how that would not exist. Have you properly
>      >>     configured ZooKeeper to act as a quorom? (e.g. if you zkCli.sh
>      >>     -server to your ZK servers individually, do you see a consistent
>      >>     view of ZK?)
>      >>
>      >>     You can try to remove the directory in HDFS you configured
>     Accumulo
>      >>     to use, and re `accumulo init` to get back to an initial
>     state. I'm
>      >>     not sure how you got into this situation to begin with.
>      >>
>      >>     Yamini Joshi wrote:
>      >>
>      >>         1.7.2
>      >>
>      >>
>      >>         Best regards,
>      >>         Yamini Joshi
>      >>
>      >>         On Thu, Oct 6, 2016 at 4:17 PM, Josh Elser
>     <josh.elser@gmail.com <mailto:josh.elser@gmail.com>
>      >> <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
>      >> <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>
>     <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>>>
>      >> wrote:
>      >>
>      >>              Hrm, maybe I am looking at a newer version of
>     Accumulo than
>      >>         what
>      >>              you're using. What version are you on?
>      >>
>      >>              Yamini Joshi wrote:
>      >>
>      >>                  Thank you for replying Josh.
>      >>                  I am getting this error:
>      >>                  Node does not exist: /tracers
>      >>
>      >>                  I did ls and saw accumulo there. Should I
>     delete that?
>      >>                  When I try to delete accumulo, it gives me a
>      >>         Authentication is
>      >>                  not valid
>      >>                  error.
>      >>
>      >>                  Best regards,
>      >>                  Yamini Joshi
>      >>
>      >>                  On Thu, Oct 6, 2016 at 2:57 PM, Josh Elser
>      >> <josh.elser@gmail.com <mailto:josh.elser@gmail.com>
>     <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
>      >> <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>
>     <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>>
>      >> <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>
>     <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
>      >> <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>
>     <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>>>>
>      >> wrote:
>      >>
>      >>                       Hi Yamini,
>      >>
>      >>                       The /tracers node in ZooKeeper was likely
>      >> initialized
>      >>                  previously
>      >>                       using an Accumulo configuration that has a
>      >>         different value for
>      >>                       instance.secret. You can delete this znode
>     to move
>      >>         forward
>      >>                  if you
>      >>                       have no other Accumulo instance you intend
>     to run
>      >>         on this
>      >>                  ZK instance.
>      >>
>      >>                       $ $ZOOKEEPER_HOME/bin/zkCli.sh -server
>      >>         m4:2181,m5:2181,m6:2181
>      >> > rmr /tracers
>      >>
>      >>                       You can also set a different value for
>      >>         trace.zookeeper.path in
>      >>                       accumulo-site.xml. After doing this, you
>     can try
>      >>         running a
>      >>                       start-all.sh again to start the tracer.
>      >>
>      >>                       You also mentioned that the Monitor did
>     not start
>      >>         but did not
>      >>                       include an exception for that service. The
>     error you
>      >>                  included only
>      >>                       shows that the Accumulo
>     Master/TabletServer did not
>      >>                  correctly start
>      >>                       and assign the system tables
>     (accumulo.root and
>      >>                  accumulo.metadata).
>      >>
>      >>
>      >>                       Yamini Joshi wrote:
>      >>
>      >>                           Hello all
>      >>
>      >>                           I am trying to configure a multinode
>     cluster
>      >> with
>      >>                  accumulo. I am
>      >>                           using 3
>      >>                           machines with the config as follows:
>      >>
>      >>                           M4: Hadoop Master/Acc Slave
>      >>                           M5: Accumulo Master/Monitor/Gc
>      >>                           M6: Slave
>      >>
>      >>                           I have configured zookeeper ensemble
>     on the
>      >>         cluster
>      >>                  with all the 3
>      >>                           nodes. The Hadoop part seems to work
>     fine, I
>      >>         can see
>      >>                  the console
>      >>                           and run
>      >>                           MR jobs easily. I can also see the
>     zookeeper
>      >>         cluster
>      >>                  with M2 as the
>      >>                           leader. My problem is with accumulo, after
>      >>         accumulo init:
>      >>                           bin/accumulo init
>      >>                              which seems to run without any
>     error, after
>      >>         I isuue
>      >>                  the command
>      >>                           bin/start-all.sh
>      >>                              the Monitor process can't start. I am
>      >>         getting this
>      >>                  error in
>      >>                           my tracer log:
>      >>                           [start.Main] ERROR: Thread 'tracer' died.
>      >>
>      >>           org.apache.zookeeper.KeeperException$NoAuthException:
>      >>                           KeeperErrorCode =
>      >>                           NoAuth for /tracers/trace-
>      >>                                 at
>      >>
>      >>
>      >>
>      >>
>     org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
>      >>                                 at
>      >>
>      >>
>      >>
>      >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>      >>                                 at
>      >>
>     org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>      >>                                 at
>      >>
>      >>
>      >>
>      >>
>     org.apache.accumulo.fate.zookeeper.ZooUtil.putEphemeralSequential(ZooUtil.java:464)
>      >>                                 at
>      >>
>      >>
>      >>
>      >>
>     org.apache.accumulo.fate.zookeeper.ZooReaderWriter.putEphemeralSequential(ZooReaderWriter.java:99)
>      >>                                 at
>      >>
>      >>
>      >>
>      >>
>     org.apache.accumulo.tracer.TraceServer.registerInZooKeeper(TraceServer.java:304)
>      >>                                 at
>      >>
>      >>
>      >>
>      >> org.apache.accumulo.tracer.TraceServer.<init>(TraceServer.java:241)
>      >>                                 at
>      >>
>      >>
>      >>
>       org.apache.accumulo.tracer.TraceServer.main(TraceServer.java:346)
>      >>                                 at
>      >>
>      >>
>      >>
>      >>
>     org.apache.accumulo.tracer.TracerExecutable.execute(TracerExecutable.java:33)
>      >>                                 at
>      >>                  org.apache.accumulo.start.Main$1.run(Main.java:120)
>      >>                                 at
>     java.lang.Thread.run(Thread.java:745)
>      >>
>      >>                           And this in my monitor debug log:
>      >>                           Failed to locate tablet for table : !0
>     row :
>      >>         ~err_\100
>      >>
>      >>                           I can see that the accumulo dir has been
>      >>         created in the
>      >>                  HDFS.
>      >>                           Can anyone
>      >>                           please guide me set this up? I am
>     attaching my
>      >>         config
>      >>                  files here
>      >>                           (Rest
>      >>                           all the same generated as a result of
>      >>         bin_config file).
>      >>
>      >>                           Best regards,
>      >>                           Yamini Joshi
>      >>
>      >>
>      >>
>      >>
>      >
>
>
>
>     --
>     busbey
>
>

Mime
View raw message