accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adina Crainiceanu <ad...@usna.edu>
Subject Re: Accumulo not initializing
Date Sat, 14 May 2016 19:06:15 GMT
Not sure what is going on, I did not encounter the problem on the VM after
changing the zoo.cfg.
The other file you might need to change is Hadoop's hdfs-site.xml, to
change the value of the dfs.datanode.data.dir.

However, changing the hadoop defaults will not fix the problem with
Accumulo.

My suggestion is to re-start everything:
1. stop Accumulo (which is probably not running anyway)
2. clean up Hadoop's /accumulo directory (see Christopher's message earlier
in the thread)
3. stop zookeeper
4. change hadoop's hdfs-site.xml - make sure permission on the new folder
are permissive enough
5. double check zoo.cfg and permissions for the dataDir
6. restart computer
7. start zookeeper
8. try initialize Accumulo
9. try start Accumulo


On Sat, May 14, 2016 at 1:34 PM, Amila Wijayarathna <
amwijayarathna@gmail.com> wrote:

> Hi Adina,
> Zookeeper is successfully started, i got imok back. But Accumulo is failed
> to initialize.  i got the following,
>
> 2016-05-14 22:56:07,267 [init.Initialize] INFO : Hadoop Filesystem is
> hdfs://localhost:9000
> 2016-05-14 22:56:07,268 [init.Initialize] INFO : Accumulo data dirs are
> [hdfs://localhost:9000/ACCUMULO5]
> 2016-05-14 22:56:07,268 [init.Initialize] INFO : Zookeeper server is
> localhost:2181
> 2016-05-14 22:56:07,268 [init.Initialize] INFO : Checking if Zookeeper is
> available. If this hangs, then you need to make sure zookeeper is running
> 2016-05-14 22:57:07,640 [init.Initialize] FATAL:
> java.lang.RuntimeException: Failed to connect to zookeeper (localhost:2181)
> within 2x zookeeper timeout period 30000
> java.lang.RuntimeException: Failed to connect to zookeeper (localhost:2181)
> within 2x zookeeper timeout period 30000
>     at
> org.apache.accumulo.fate.zookeeper.ZooSession.connect(ZooSession.java:123)
>     at
>
> org.apache.accumulo.fate.zookeeper.ZooSession.getSession(ZooSession.java:167)
>     at
> org.apache.accumulo.fate.zookeeper.ZooReader.getSession(ZooReader.java:39)
>     at
>
> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.getZooKeeper(ZooReaderWriter.java:50)
>     at
> org.apache.accumulo.fate.zookeeper.ZooReader.exists(ZooReader.java:166)
>     at
>
> org.apache.accumulo.server.init.Initialize.zookeeperAvailable(Initialize.java:277)
>     at
> org.apache.accumulo.server.init.Initialize.checkInit(Initialize.java:159)
>     at
> org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:213)
>     at org.apache.accumulo.server.init.Initialize.main(Initialize.java:590)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at org.apache.accumulo.start.Main$1.run(Main.java:141)
>     at java.lang.Thread.run(Thread.java:745)
> Thread "init" died java.lang.RuntimeException: Failed to connect to
> zookeeper (localhost:2181) within 2x zookeeper timeout period 30000
> java.lang.RuntimeException: java.lang.RuntimeException: Failed to connect
> to zookeeper (localhost:2181) within 2x zookeeper timeout period 30000
>     at org.apache.accumulo.server.init.Initialize.main(Initialize.java:594)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at org.apache.accumulo.start.Main$1.run(Main.java:141)
>     at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Failed to connect to zookeeper
> (localhost:2181) within 2x zookeeper timeout period 30000
>     at
> org.apache.accumulo.fate.zookeeper.ZooSession.connect(ZooSession.java:123)
>     at
>
> org.apache.accumulo.fate.zookeeper.ZooSession.getSession(ZooSession.java:167)
>     at
> org.apache.accumulo.fate.zookeeper.ZooReader.getSession(ZooReader.java:39)
>     at
>
> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.getZooKeeper(ZooReaderWriter.java:50)
>     at
> org.apache.accumulo.fate.zookeeper.ZooReader.exists(ZooReader.java:166)
>     at
>
> org.apache.accumulo.server.init.Initialize.zookeeperAvailable(Initialize.java:277)
>     at
> org.apache.accumulo.server.init.Initialize.checkInit(Initialize.java:159)
>     at
> org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:213)
>     at org.apache.accumulo.server.init.Initialize.main(Initialize.java:590)
>     ... 6 more
>
> Are there any configurations need to change other than zoo.cfg?
>
> Thank you!
>
> On Sat, May 14, 2016 at 9:02 PM, Adina Crainiceanu <adina@usna.edu> wrote:
>
> > Amila,
> >
> >
> > 2 ideas:
> >
> > 1) Are you sure that Zookeeper started successfully? Maybe it did not.
> Make
> > sure the permissions on that local dir you used for dirData are correct.
> >
> > Try the
> > sudo echo ruok | nc 127.0.0.1 2181
> >
> > if you don;t get imok back, start Zookeeper
> >
> > 2) Did you re-initialized Accumulo? Since you started Zookeeper in a
> > different directory, Accumulo needs to be initialized before it starts
> >
> > accumulo-${ACCUMULO_VERSION}/bin/accumulo init --instance-name dev
> > --password root
> > accumulo-${ACCUMULO_VERSION}/bin/start-all.sh
> >
> > Adina
> >
> > On Sat, May 14, 2016 at 7:34 AM, Amila Wijayarathna <
> > amwijayarathna@gmail.com> wrote:
> >
> > > Hi Adina,
> > >
> > > After a restart I used *accumulo-1.6.4/bin/stop-all.sh* and after that
> I
> > > start accumulo again with *accumulo-1.6.4/bin/start-all.sh * then I can
> > > successfully start accumulo without initializing problem.
> > >
> > > But after I changed the Datadir value to a local, I can successfully
> > start
> > > Zookeeper. Then I tried to start accumulo using
> > > *accumulo-1.6.4/bin/start-all.sh
> > > *but I got the following,
> > >
> > >
> > > amila@amila-X550LD:~$ ~/Installs/accumulo-1.6.4/bin/start-all.sh
> > >
> > > Starting monitor on localhost
> > >
> > > WARN : Max open files on localhost is 1024, recommend 32768
> > >
> > > Starting tablet servers .... done
> > >
> > > 2016-05-14 16:34:54,379 [server.Accumulo] INFO : Attempting to talk to
> > > zookeeper
> > >
> > > Starting tablet server on localhost
> > >
> > > WARN : Max open files on localhost is 1024, recommend 32768
> > >
> > > Thread "org.apache.accumulo.master.state.SetGoalState" died Failed to
> > > connect to zookeeper (localhost:2181) within 2x zookeeper timeout
> period
> > > 30000
> > >
> > > java.lang.RuntimeException: Failed to connect to zookeeper
> > (localhost:2181)
> > > within 2x zookeeper timeout period 30000
> > >
> > > at
> > >
> >
> org.apache.accumulo.fate.zookeeper.ZooSession.connect(ZooSession.java:123)
> > >
> > > at
> > >
> > >
> >
> org.apache.accumulo.fate.zookeeper.ZooSession.getSession(ZooSession.java:167)
> > >
> > >
> > > at
> > >
> >
> org.apache.accumulo.fate.zookeeper.ZooReader.getSession(ZooReader.java:39)
> > >
> > > at
> > >
> > >
> >
> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.getZooKeeper(ZooReaderWriter.java:50)
> > >
> > >
> > > at
> > >
> > >
> >
> org.apache.accumulo.fate.zookeeper.ZooReader.getChildren(ZooReader.java:128)
> > >
> > >
> > > at
> > >
> > >
> >
> org.apache.accumulo.server.Accumulo.waitForZookeeperAndHdfs(Accumulo.java:269)
> > >
> > >
> > > at
> > org.apache.accumulo.master.state.SetGoalState.main(SetGoalState.java:46)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >
> > > at
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > >
> > >
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >
> > >
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > >
> > > at org.apache.accumulo.start.Main$1.run(Main.java:141)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > Starting master on localhost
> > >
> > > WARN : Max open files on localhost is 1024, recommend 32768
> > >
> > > Starting garbage collector on localhost
> > >
> > > WARN : Max open files on localhost is 1024, recommend 32768
> > >
> > > Starting tracer on localhost
> > >
> > > WARN : Max open files on localhost is 1024, recommend 32768
> > >
> > >
> > > Any idea about this is appreciated.
> > >
> > > Thank you.
> > >
> > > On Fri, May 13, 2016 at 8:28 PM, Adina Crainiceanu <adina@usna.edu>
> > wrote:
> > >
> > > > Amila,
> > > >
> > > > As Christopher said, make sure that Zookeeper and Hadoop store the
> data
> > > > outside the /tmp folder. For Zookeeper, the default configuration
> > > specifies
> > > > the /tmp/zookeeper folder, so make sure to change that (go to
> > > > /pathtoZookeeper/conf/zoo.cfg  and look at the dataDir value; it
> should
> > > not
> > > > start with /tmp. If it does, change that to some local, non-tmp
> folder,
> > > for
> > > > example /data/zookeeper).
> > > >
> > > > That is a problem for the Rya Vagrant VM too - Zookeeper stores the
> > data
> > > in
> > > > /tmp/zookeeper so a restart creates problems with Accumulo, including
> > > > a "Waiting
> > > > for accumulo to be initialized." message
> > > >
> > > > Adina
> > > >
> > > > On Thu, Apr 21, 2016 at 2:17 PM, Amila Wijayarathna <
> > > > amwijayarathna@gmail.com> wrote:
> > > >
> > > > > Hi Christoper,
> > > > > Thank you for your prompt reply. I successfully re-corrected that
> > > warning
> > > > > messages from your guidelines and still working on that
> initializing
> > > > > problem.Any further ideas about that are thoroughly appreciated.
> > > > >
> > > > > Thank you!
> > > > >
> > > > > On Tue, Apr 19, 2016 at 11:06 PM, Christopher <ctubbsii@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > The warning about the ZooKeeper version is probably due to a
> > trailing
> > > > > slash
> > > > > > at the end of your environment variable for ZOOKEEPER_HOME.
> > > > > >
> > > > > > The max open files warning is regarding what is being discussed
> > here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user
> > > > > > (this
> > > > > > is just the first link I found that covered the topic... but
> there
> > > > might
> > > > > be
> > > > > > better discussions of system security limits and ulimit
> elsewhere)
> > > > > >
> > > > > > The synconclose warning can be addressed by setting the
> recommended
> > > > value
> > > > > > to true in your hadoop configuration (hdfs-site.xml)
> > > > > >
> > > > > > As for the initialization problem, it sounds like you're trying
> to
> > > > > > re-initialize Accumulo in the same location that it was
> previously
> > > > > > initialized. You only need to do initialization once, but you
do
> > need
> > > > to
> > > > > > make sure that wherever Hadoop and ZooKeeper are storing their
> data
> > > in
> > > > > the
> > > > > > local filesystem, that they aren't in a temporary directory
or
> > > > something
> > > > > > like that.
> > > > > >
> > > > > > If you actually do wish to re-initialize Accumulo, you can delete
> > the
> > > > > > previous contents of Hadoop's /accumulo directory first (make
> sure
> > > you
> > > > > want
> > > > > > to do this... existing data will be deleted), using something
> like
> > > > > `hadoop
> > > > > > fs -rmr /accumulo` (exact command may depend on your version
of
> > > > Hadoop).
> > > > > >
> > > > > > On Tue, Apr 19, 2016 at 1:10 PM Amila Wijayarathna <
> > > > > > amwijayarathna@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'm trying to work with RYA in my Ubuntu laptop. And it
needs
> > > Hadoop,
> > > > > > > Zookeeper and Accumulo servers As prerequisites.
> > > > > > >
> > > > > > > After a shutdown, I need to restart hadoop, zookeeper and
> > accumulo.
> > > > > When
> > > > > > I
> > > > > > > do, Hadoop and Zookeeper servers are starting successfully
but
> > > when I
> > > > > > start
> > > > > > > Accumulo I get the following,
> > > > > > >
> > > > > > > WARN : Using Zookeeper
> > > > > > > /home/amila/Installs/zookeeper-3.4.8/zookeeper-3.4.8. Use
> version
> > > > 3.3.0
> > > > > > or
> > > > > > > greater to avoid zookeeper deadlock bug.
> > > > > > >
> > > > > > > Starting monitor on localhost
> > > > > > >
> > > > > > > WARN : Max open files on localhost is 1024, recommend 32768
> > > > > > >
> > > > > > > Starting tablet servers .... done
> > > > > > >
> > > > > > > 2016-04-19 17:40:43,303 [fs.VolumeManagerImpl] WARN :
> > > > > > > dfs.datanode.synconclose set to false in hdfs-site.xml:
data
> loss
> > > is
> > > > > > > possible on hard system reset or power loss
> > > > > > >
> > > > > > > 2016-04-19 17:40:43,308 [server.Accumulo] INFO : Attempting
to
> > talk
> > > > to
> > > > > > > zookeeper
> > > > > > >
> > > > > > > 2016-04-19 17:40:43,522 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:44,534 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:45,536 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:46,538 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:47,541 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:48,543 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:49,545 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:50,547 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:51,549 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:52,550 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > Starting tablet server on localhost
> > > > > > >
> > > > > > > WARN : Max open files on localhost is 1024, recommend 32768
> > > > > > >
> > > > > > > 2016-04-19 17:40:53,551 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:54,552 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:55,554 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:56,555 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:57,556 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:58,558 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:40:59,559 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > 2016-04-19 17:41:00,561 [server.Accumulo] INFO : Waiting
for
> > > accumulo
> > > > > to
> > > > > > be
> > > > > > > initialized
> > > > > > >
> > > > > > > Then I tried to initialize Accumulo and I got following,
> > > > > > >
> > > > > > > 2016-04-19 16:56:06,984 [fs.VolumeManagerImpl] WARN :
> > > > > > > dfs.datanode.synconclose set to false in hdfs-site.xml:
data
> loss
> > > is
> > > > > > > possible on hard system reset or power loss
> > > > > > >
> > > > > > > 2016-04-19 16:56:06,985 [init.Initialize] INFO : Hadoop
> > Filesystem
> > > is
> > > > > > > hdfs://localhost:9000
> > > > > > >
> > > > > > > 2016-04-19 16:56:06,986 [init.Initialize] INFO : Accumulo
data
> > dirs
> > > > are
> > > > > > > [hdfs://localhost:9000/accumulo]
> > > > > > >
> > > > > > > 2016-04-19 16:56:06,986 [init.Initialize] INFO : Zookeeper
> server
> > > is
> > > > > > > localhost:2181
> > > > > > >
> > > > > > > 2016-04-19 16:56:06,986 [init.Initialize] INFO : Checking
if
> > > > Zookeeper
> > > > > is
> > > > > > > available. If this hangs, then you need to make sure zookeeper
> is
> > > > > running
> > > > > > >
> > > > > > > 2016-04-19 16:56:07,250 [init.Initialize] FATAL: It appears
the
> > > > > > directories
> > > > > > > [hdfs://localhost:9000/accumulo] were previously initialized.
> > > > > > >
> > > > > > > 2016-04-19 16:56:07,250 [init.Initialize] FATAL: Change
the
> > > property
> > > > > > > instance.volumes to use different filesystems,
> > > > > > >
> > > > > > > 2016-04-19 16:56:07,250 [init.Initialize] FATAL: or change
the
> > > > property
> > > > > > > instance.dfs.dir to use a different directory.
> > > > > > >
> > > > > > > 2016-04-19 16:56:07,250 [init.Initialize] FATAL: The current
> > value
> > > of
> > > > > > > instance.dfs.uri is ||
> > > > > > >
> > > > > > > 2016-04-19 16:56:07,250 [init.Initialize] FATAL: The current
> > value
> > > of
> > > > > > > instance.dfs.dir is |/accumulo|
> > > > > > >
> > > > > > > 2016-04-19 16:56:07,250 [init.Initialize] FATAL: The current
> > value
> > > of
> > > > > > > instance.volumes is |hdfs://localhost:9000/accumulo|
> > > > > > >
> > > > > > >
> > > > > > > Can anyone point me how to do this correctly?
> > > > > > >
> > > > > > > Thank you!
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > *Amila Wijayarathna*
> > > > > > > Undergraduate,
> > > > > > > Faculty of Information Technology,
> > > > > > > University of Moratuwa,
> > > > > > > Sri Lanka.
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Amila Wijayarathna*
> > > > > Undergraduate,
> > > > > Faculty of Information Technology,
> > > > > University of Moratuwa,
> > > > > Sri Lanka.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Dr. Adina Crainiceanu
> > > > Associate Professor, Computer Science Department
> > > > United States Naval Academy
> > > > 410-293-6822
> > > > adina@usna.edu
> > > > http://www.usna.edu/Users/cs/adina/
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > *Amila Wijayarathna*
> > > Undergraduate,
> > > Faculty of Information Technology,
> > > University of Moratuwa,
> > > Sri Lanka.
> > >
> >
> >
> >
> > --
> > Dr. Adina Crainiceanu
> > Associate Professor, Computer Science Department
> > United States Naval Academy
> > 410-293-6822
> > adina@usna.edu
> > http://www.usna.edu/Users/cs/adina/
> >
>
>
>
> --
>
> *Amila Wijayarathna*
> Undergraduate,
> Faculty of Information Technology,
> University of Moratuwa,
> Sri Lanka.
>



-- 
Dr. Adina Crainiceanu
Associate Professor, Computer Science Department
United States Naval Academy
410-293-6822
adina@usna.edu
http://www.usna.edu/Users/cs/adina/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message