Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 888981054C for ; Tue, 6 Jan 2015 15:42:34 +0000 (UTC) Received: (qmail 19029 invoked by uid 500); 6 Jan 2015 15:42:35 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 18981 invoked by uid 500); 6 Jan 2015 15:42:35 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 18970 invoked by uid 99); 6 Jan 2015 15:42:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jan 2015 15:42:35 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (nike.apache.org: local policy) Received: from [209.85.216.47] (HELO mail-qa0-f47.google.com) (209.85.216.47) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jan 2015 15:42:10 +0000 Received: by mail-qa0-f47.google.com with SMTP id n4so16373106qaq.6 for ; Tue, 06 Jan 2015 07:40:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=g9yoPLVhHYhLwPL670T1V8d8ZVf+dWef5mdfmkkO9jI=; b=JrzBpl5Ga7Yf4plQnbv72L3rU+gHiL9smMNo0G4hDQPOXAmmoxG2R6yCDGlUGFv0HY RrXF6gXuOIzTShMs3FkpdrexwHO2BITC7tzie7EB9yB6BqmQxeK20J+J4rgjOQRH5n7g hEViwrWfTbvlC3SAek5p/e1vLXWxIdZo31GvGygG3DPpmPF7oODZN8vAb1OTiHJtwnpM vtqbmzh/IVwlPufBLGUJTsYoGrBxWawv9GVqg5QBo1XwMxMoqFEcMhcaPLs3kvseJuhq KN1s0yQYsqooguix74WVwQW2JlNGZ0Hd19QTHDiYjd2ncR9IybRoDXDoDGHCBklZmRa9 1ZMw== X-Gm-Message-State: ALoCoQnD13/HFQN3ipz7UIWF7TNsrPZLOxnXOg7lSaqJdGLT3bd9VlYE/nh5tJcTw18gSOE3xGUX MIME-Version: 1.0 X-Received: by 10.224.69.200 with SMTP id a8mr153722491qaj.40.1420558817207; Tue, 06 Jan 2015 07:40:17 -0800 (PST) Received: by 10.229.9.138 with HTTP; Tue, 6 Jan 2015 07:40:17 -0800 (PST) In-Reply-To: References: Date: Tue, 6 Jan 2015 10:40:17 -0500 Message-ID: Subject: Re: first time setup: Mkdirs failed to create hdfs directory /accumulo/recovery/ From: Keith Turner To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary=001a11c29d0caebf3b050bfd9f2d X-Virus-Checked: Checked by ClamAV on apache.org --001a11c29d0caebf3b050bfd9f2d Content-Type: text/plain; charset=UTF-8 On Mon, Jan 5, 2015 at 6:50 PM, Mike Atlas wrote: > Hello, > > I'm running Accumulo 1.5.2, trying to test out the GeoMesa > family of > spatio-temporal iterators using their quickstart demonstration tool. I > think I'm not making progress due to my Accumulo setup, though, so can > someone validate that all looks good from here? > > start-all.sh output: > > hduser@accumulo:~$ $ACCUMULO_HOME/bin/start-all.sh > Starting monitor on localhost > Starting tablet servers .... done > Starting tablet server on localhost > 2015-01-05 21:37:18,523 [server.Accumulo] INFO : Attempting to talk to zookeeper > 2015-01-05 21:37:18,772 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS > 2015-01-05 21:37:19,028 [server.Accumulo] INFO : Connected to HDFS > Starting master on localhost > Starting garbage collector on localhost > Starting tracer on localhost > > hduser@accumulo:~$ > > > I do believe my HDFS is set up correctly: > > hduser@accumulo:/home/ubuntu/geomesa-quickstart$ hadoop fs -ls /accumulo > Found 5 items > drwxrwxrwx - hduser supergroup 0 2014-12-10 01:04 /accumulo/instance_id > drwxrwxrwx - hduser supergroup 0 2015-01-05 21:22 /accumulo/recovery > drwxrwxrwx - hduser supergroup 0 2015-01-05 20:14 /accumulo/tables > drwxrwxrwx - hduser supergroup 0 2014-12-10 01:04 /accumulo/version > drwxrwxrwx - hduser supergroup 0 2014-12-10 01:05 /accumulo/wal > > > However, when I check the Accumulo monitor logs, I see these errors > post-startup: > > java.io.IOException: Mkdirs failed to create directory /accumulo/recovery/15664488-bd10-4d8d-9584-f88d8595a07c/part-r-00000 > java.io.IOException: Mkdirs failed to create directory /accumulo/recovery/15664488-bd10-4d8d-9584-f88d8595a07c/part-r-00000 > at org.apache.hadoop.io.MapFile$Writer.(MapFile.java:264) > at org.apache.hadoop.io.MapFile$Writer.(MapFile.java:103) > at org.apache.accumulo.server.tabletserver.log.LogSorter$LogProcessor.writeBuffer(LogSorter.java:196) > at org.apache.accumulo.server.tabletserver.log.LogSorter$LogProcessor.sort(LogSorter.java:166) > at org.apache.accumulo.server.tabletserver.log.LogSorter$LogProcessor.process(LogSorter.java:89) > at org.apache.accumulo.server.zookeeper.DistributedWorkQueue$1.run(DistributedWorkQueue.java:101) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47) > at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) > at java.lang.Thread.run(Thread.java:745) > > > I don't really understand - I started accumulo as the hduser, which is > the same user that has access to the HDFS directory /accumulo/recovery, > and it looks like the directory was created actually, except for the last > directory (part-r-0000): > > hduser@accumulo:~$ hadoop fs -ls /accumulo0/recovery/ > Found 1 items > drwxr-xr-x - hduser supergroup 0 2015-01-05 22:11 /accumulo/recovery/87fb7aac-0274-4aea-8014-9d53dbbdfbbc > > > I'm not out of physical disk space: > > hduser@accumulo:~$ df -h > Filesystem Size Used Avail Use% Mounted on > /dev/xvda1 1008G 8.5G 959G 1% / > > > What could be going on here? Any ideas on something simple I could have > missed? > One possibility is that tserver where the exception occurred had bad or missing config for hdfs. In this case the hadoop code may try to create /accumulo/recovery/.../part-r-00000 in local fs, which would fail. > > Thanks, > Mike > --001a11c29d0caebf3b050bfd9f2d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On Mon, Jan 5, 2015 at 6:50 PM, Mike Atlas <mike@weft.io> wro= te:
H= ello,

I'm running Accumulo 1.5.2, trying to test out= the GeoMesa family of spatio-temporal iterators using their q= uickstart demonstration tool. I think I'm not making progress due to my= Accumulo setup, though, so can someone validate that all looks good from h= ere?

start-all.sh output:

hduser@accumulo:~$ $ACCUMULO_HOME/bin/start=
-all.sh
Starting monitor on localhost
Starting tablet servers .... done
Starting tablet server on localhost
2015-01-05 21:37:18,523 [server.Accumulo] INFO : Attempting to talk to zook=
eeper
2015-01-05 21:37:18,772 [server.Accumulo] INFO : Zookeeper connected and in=
itialized, attemping to talk to HDFS
2015-01-05 21:37:19,028 [server.Accumulo] INFO : Connected to HDFS
Starting master on localhost
Starting garbage collector on localhost
Starting tracer on localhost
hduser@accumulo:~$

I do believe my = HDFS is set up correctly:

hduser@accumulo:/home/ubuntu/geomesa-quickstart$ hadoop fs -ls /accu=
mulo
Found 5 items
drwxrwxrwx   - hduser supergroup          0 2014-12-10 01:04 /accumulo/inst=
ance_id
drwxrwxrwx   - hduser supergroup          0 2015-01-05 21:22 /accumulo/reco=
very
drwxrwxrwx   - hduser supergroup          0 2015-01-05 20:14 /accumulo/tabl=
es
drwxrwxrwx   - hduser supergroup          0 2014-12-10 01:04 /accumulo/vers=
ion
drwxrwxrwx   - hduser supergroup          0 2014-12-10 01:05 /accumulo/wal<=
/font>

However, when I check the Accumu= lo monitor logs, I see these errors post-startup:

= 
java.io.IOException: Mkdirs faile=
d to create directory /accumulo/recovery/15664488-bd10-4d8d-9584-f88d8595a0=
7c/part-r-00000
	java.io.IOException: Mkdirs failed to create directory /accumulo/recovery/=
15664488-bd10-4d8d-9584-f88d8595a07c/part-r-00000
		at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:264)
		at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:103)
		at org.apache.accumulo.server.tabletserver.log.LogSorter$LogProcessor.wri=
teBuffer(LogSorter.java:196)
		at org.apache.accumulo.server.tabletserver.log.LogSorter$LogProcessor.sor=
t(LogSorter.java:166)
		at org.apache.accumulo.server.tabletserver.log.LogSorter$LogProcessor.pro=
cess(LogSorter.java:89)
		at org.apache.accumulo.server.zookeeper.DistributedWorkQueue$1.run(Distri=
butedWorkQueue.java:101)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j=
ava:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.=
java:615)
		at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.j=
ava:47)
		at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java=
:34)
		at java.lang.Thread.run(Thread.java:745)

=
I don't really understand - I started accumulo as the=C2=A0hduser, which is the same user that has access to the HDFS directory=C2=A0/accumulo/r= ecovery, and it looks like the directory was created actually, excep= t for the last directory (part-r-0000):

hduser@accumulo:~$ hadoop fs -ls /accumulo0=
/recovery/
Found 1 items
drwxr-xr-x   - hduser supergroup          0 2015-01-05 22:11 /accumulo/reco=
very/87fb7aac-0274-4aea-8014-9d53dbbdfbbc

I&= #39;m not out of physical disk space:

hduser@accumulo:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1     1008G  8.5G  959G   1% /

What= could be going on here? Any ideas on something simple I could have missed?=

One possibility is that tserve= r where the exception occurred had bad or missing config for hdfs.=C2=A0 In= this case the hadoop code may try to create /accumulo/recovery/.../part-r-= 00000 in local fs, which would fail.
=C2=A0

= Thanks,
Mike=C2=A0

--001a11c29d0caebf3b050bfd9f2d--