Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 990B217A22 for ; Mon, 2 Mar 2015 16:50:19 +0000 (UTC) Received: (qmail 530 invoked by uid 500); 2 Mar 2015 16:49:53 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 415 invoked by uid 500); 2 Mar 2015 16:49:53 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 404 invoked by uid 99); 2 Mar 2015 16:49:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2015 16:49:52 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=AC_DIV_BONANZA,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of busbey@cloudera.com designates 209.85.216.172 as permitted sender) Received: from [209.85.216.172] (HELO mail-qc0-f172.google.com) (209.85.216.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2015 16:49:27 +0000 Received: by qcxn11 with SMTP id n11so3466073qcx.3 for ; Mon, 02 Mar 2015 08:47:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=78u0WdTnS5BXWzrqM0xnvQo9hJhMIxATCZI2cGy+Umc=; b=kS04PrCG8iwi0ridw3i2nbTO7K7Gui3ytOZmQdxKil/5gNwxgi+GH0KxctJ1dD1Jxl FbmyYOOoeqOM+Bxz+A1zqz5Qbb1XidSzF6boFjMrvpp5jjkUBvjR30uLasNa1XJb+Hdh 0skP+8sSLnl10Sjv9louMedp5liWI8fM7c+FQnR4M76uQ/fVPVKu4i+keTNsXwp+phyh Z5/3uOZPK7atV1LaYLY1CD+CfTrRCCiqZas43JDPkacYGH0yBTIa9n2QyfqZB1GA+SDM UapYVBkeTsYJKHVe+VmZZQUay8/duaroLCfV/rjY01RPAqf1JBxHMOqUcLdrITvrFV1o xJKA== X-Gm-Message-State: ALoCoQmZy1zJbovjHD3bSUHgdJyxQQXixB76D1TpNMT0lMEdiYm4pbz+dMOAJtJVKTrkYL8h8yhR X-Received: by 10.55.52.200 with SMTP id b191mr6924982qka.46.1425314875290; Mon, 02 Mar 2015 08:47:55 -0800 (PST) MIME-Version: 1.0 Received: by 10.229.252.73 with HTTP; Mon, 2 Mar 2015 08:47:35 -0800 (PST) In-Reply-To: References: From: Sean Busbey Date: Mon, 2 Mar 2015 10:47:35 -0600 Message-ID: Subject: Re: Permission Denied To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a1146ff82d5e9b6051050faa8 X-Virus-Checked: Checked by ClamAV on apache.org --001a1146ff82d5e9b6051050faa8 Content-Type: text/plain; charset=UTF-8 Splitting into three unix users is a good idea. Generally, none of the linux users should need access to any of the local resources owned by the others. (that is, the user running the accumulo processes shouldn't be able to interfere with the backing files used by the HDFS processes). By default, the linux user that drives a particular process will be resolved to a Hadoop user by the NameNode process. Presuming your Accumulo services are running under the linux user "accumulo", you should ensure that user exists on the linux node that runs the NameNode. The main issue with running init as the hadoop user is that by default it's likely going to write the accumulo directories as owned by the user that created them. Presuming you are using Accumulo because you have security requirements, the common practice is to make sure only the user that runs Accumulo processes can write to /accumulo and that only that user can read /accumulo/tables and /accumulo/wal. This ensures that other users with access to the HDFS cluster won't be able to bypass the cell-level access controls provided by Accumulo. While you are setting up HDFS directories, you should also create a home directory for the user that runs Accumulo processes. If your HDFS instance is set to use the trash feature (either in server configs or the client configs made available to Accumulo), then by default Accumulo will attempt to use it. Without a home directory, this will result in failures. Alternatively, you can ensure Accumulo doesn't rely on the trash feature by setting gc.trash.ignore in your accumulo-site.xml. One other note: > I edited the accumulo-site.xml so it now has > > instance.volumes > hdfs://localhost:9000/accumulo > comma separated list of URIs for volumes. example: hdfs://localhost:9000/accumulo > You will save yourself headache later if you stick with fully qualified domain names for all HDFS, ZooKeeper, and Accumulo connections. -- Sean On Mon, Mar 2, 2015 at 8:13 AM, David Patterson wrote: > David, > > Thanks for the information. I've issued those two commands in my hadoop > shell and still get the same error when I try to initialize accumulo in > *its* shell. : > > 2015-03-02 13:30:41,175 [init.Initialize] FATAL: Failed to initialize > filesystem > org.apache.hadoop.security.AccessControlException: Permission denied: > user=accumulo, access=WRITE, inode="/accumulo": > accumulo.supergroup:supergroup:drwxr-xr-x > > My comment that I had 3 users was meant in a linux sense, not in a hadoop > sense. So (to borrow terminoloy from RDF or XML) is there something I have > to do in my hadoop setup (running under linix:hadoop) or my accumulo setup > (running under linux:accumulo) so that the accumuulo I/O gets processed as > from someone in the hadoop:supergroup? > > > I tried running the accumulo init from the linux:hadoop user and it > worked. I'm not sure if any permissions/etc were hosed by doing it there. > I'll see. > > Thanks for you help. > > (By the way, is it wrong or a bad idea to split the work into three > linux:users, or should it all be done in one linux:user space?) > > Dave Patterson > > On Sun, Mar 1, 2015 at 8:35 PM, dlmarion wrote: > >> hadoop fs -mkdir /accumulo >> hadoop fs -chown accumulo:supergroup /accumulo >> >> >> >> -------- Original message -------- >> From: David Patterson >> Date:03/01/2015 7:04 PM (GMT-05:00) >> To: user@hadoop.apache.org >> Cc: >> Subject: Re: Permission Denied >> >> David, >> >> Thanks for the reply. >> >> Taking the questions in the opposite order, my accumulo-site.xml does not >> have volumes specified. >> >> I edited the accumulo-site.xml so it now has >> >> instance.volumes >> hdfs://localhost:9000/accumulo >> comma separated list of URIs for volumes. example: >> hdfs://localhost:9000/accumulo >> >> >> and got the same error. >> >> How can I precreate /accumulo ? >> >> Dave Patterson >> >> On Sun, Mar 1, 2015 at 3:50 PM, david marion >> wrote: >> >>> It looks like / is owned by hadoop.supergroup and the perms are 755. >>> You could precreate /accumulo and chown it appropriately, or set the perms >>> for / to 775. Init is trying to create /accumulo in hdfs as the accumulo >>> user and your perms dont allow it. >>> >>> Do you have instance.volumes set in accumulo-site.xml? >>> >>> >>> -------- Original message -------- >>> From: David Patterson >>> Date:03/01/2015 3:36 PM (GMT-05:00) >>> To: user@hadoop.apache.org >>> Cc: >>> Subject: Permission Denied >>> >>> I'm trying to create an Accumulo/Hadoop/Zookeeper configuration >>> on a single (Ubuntu) machine, with Hadoop 2.6.0, Zookeeper 3.4.6 and >>> Accumulo 1.6.1. >>> >>> I've got 3 userids for these components that are in the same group and >>> no other users are in that group. >>> >>> I have zookeeper running, and hadoop as well. >>> >>> Hadoop's core-site.xml file has the hadoop.tmp.dir set to >>> /app/hadoop/tmp.The /app/hadoop/tmp directory is owned by the hadoop user >>> and has permissions that allow other members of the group to write >>> (drwxrwxr-x). >>> >>> When I try to initialize Accumulo, with bin/accumulo init, I get FATAL: >>> Failed to initialize filesystem. >>> org.apache.hadoop.security.AccessControlException: Permission denied: >>> user=accumulo, access=WRITE, inode="/":hadoop:supergroup:drwxr-xr-x >>> >>> So, my main question is which directory do I need to give group-write >>> permission so the accumulo user can write as needed so it can initialize? >>> >>> The second problem is that the Accumulo init reports >>> [Configuration.deprecation] INFO : fs.default.name is deprecated. >>> Instead use fs.defaultFS. However, the hadoop core-site.xml file contains: >>> fs.defaultFS >>> hdfs://localhost:9000 >>> >>> Is there somewhere else that this value (fs.default.name) is >>> specified? Could it be due to Accumulo having a default value and not >>> getting the override from hadoop because of the problem listed above? >>> >>> Thanks >>> >>> Dave Patterson >>> patterd@gmail.com >>> >> >> > -- Sean --001a1146ff82d5e9b6051050faa8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Splitting into three unix users is a good idea. Generally,= none of the linux users should need access to any of the local resources o= wned by the others. (that is, the user running the accumulo processes shoul= dn't be able to interfere with the backing files used by the HDFS proce= sses).

By default, the linux user that drives a particul= ar process will be resolved to a Hadoop user by the NameNode process. Presu= ming your Accumulo services are running under the linux user "accumulo= ", you should ensure that user exists on the linux node that runs the = NameNode.

The main issue with running init as the = hadoop user is that by default it's likely going to write the accumulo = directories as owned by the user that created them. Presuming you are using= Accumulo because you have security requirements, the common practice is to= make sure only the user that runs Accumulo processes can write to /accumul= o and that only that user can read /accumulo/tables and /accumulo/wal. This= ensures that other users with access to the HDFS cluster won't be able= to bypass the cell-level access controls provided by Accumulo.
<= br>
While you are setting up HDFS directories, you should also cr= eate a home directory for the user that runs Accumulo processes. If your HD= FS instance is set to use the trash feature (either in server configs or th= e client configs made available to Accumulo), then by default Accumulo will= attempt to use it. Without a home directory, this will result in failures.= Alternatively, you can ensure Accumulo doesn't rely on the trash featu= re by setting gc.trash.ignore in your accumulo-site.xml.

One other note:

>=C2=A0I edited the accumulo-site.xml so it now has
> =C2=A0<prope= rty>
> =C2=A0 =C2=A0<name>instance.volumes&l= t;/name>
> =C2=A0 =C2=A0<value>hdfs://localh= ost:9000/accumulo</v= alue>
> =C2=A0 =C2=A0<description>comma sepa= rated list of URIs for volumes. example: hdfs://localhost:9000/accumulo</description>=
> =C2=A0</property>

You wil= l save yourself headache later if you stick with fully qualified domain nam= es for all HDFS, ZooKeeper, and Accumulo connections.

<= div>--=C2=A0
Sean

On Mon, Mar 2, 2015 at 8:13 AM, David Patterson <patterd= @gmail.com> wrote:
David,

Thanks for the information. I&#= 39;ve issued those two commands in my hadoop shell and still get the same e= rror when I try to initialize accumulo in its shell. :

2015-0= 3-02 13:30:41,175 [init.Initialize] FATAL: Failed to initialize filesystem<= br>=C2=A0=C2=A0 org.apache.hadoop.security.AccessControlException: Permissi= on denied: user=3Daccumulo, access=3DWRITE, inode=3D"/accumulo":<= br>=C2=A0=C2=A0 accumulo.supergroup:supergroup:drwxr-xr-x

My c= omment that I had 3 users was meant in a linux sense, not in a hadoop sense= . So (to borrow terminoloy from RDF or XML) is there something I have to do= in my hadoop setup (running under linix:hadoop) or my accumulo setup (runn= ing under linux:accumulo) so that the accumuulo I/O gets processed as from = someone in the hadoop:supergroup?


I tried running the= accumulo init from the linux:hadoop user and it worked. I'm not sure i= f any permissions/etc were hosed by doing it there. I'll see.

Thanks for you help.

(By the way, is it wrong or= a bad idea to split the work into three linux:users, or should it all be d= one in one linux:user space?)

Dave Patterson

On Sun, Mar 1, 2015 at 8:35 PM, dlmarion <dlmarion@com= cast.net> wrote:
hadoop fs -mkdir /accumulo
hadoop fs -chown accumulo:supergroup= /accumulo



-------- Origi= nal message --------
From: David Patterson <patterd@gmail.com>
Date:03/01/2015 7:04 PM (GMT-05:00)


On Sun, Mar 1, 2015 at = 3:50 PM, david marion <dlmarion@hotmail.com> wrote:
It looks like / is owned by hadoop.supergroup and the perms are 755. You co= uld precreate /accumulo and chown it appropriately, or set the perms for / = to 775. Init is trying to create /accumulo in hdfs as the accumulo user and= your perms dont allow it.

Do you have instance.volumes set in accumulo-site.xml?


-------- Original message --------
From: David Patterson <patterd@gmail.com>
Date:03/01/2015 3:36 PM (GMT-05:00)
Cc:
Subject: Permission Denied

I'm trying to create an Accumulo/Hadoop/Zookeeper configuration on= a single (Ubuntu) machine, with Hadoop 2.6.0, Zookeeper 3.4.6 and Accumulo= 1.6.1.

I've got 3 userids for these components that are in the same group and = no other users are in that group.

I have zookeeper running, and hadoop as well.

Hadoop's core-site.xml file has the hadoop.tmp.dir set to /app/hadoop/t= mp.The /app/hadoop/tmp directory is owned by the hadoop user and has permis= sions that allow other members of the group to write (drwxrwxr-x).

When I try to initialize Accumulo, with bin/accumulo init, I get FATAL: Fai= led to initialize filesystem.
org.apache.hadoop.security.AccessControlException: Permission denied: user= =3Daccumulo, access=3DWRITE, inode=3D"/":hadoop:supergroup:drwxr-= xr-x

So, my main question is which directory do I need to give group-write permi= ssion so the accumulo user can write as needed so it can initialize?

The second problem is that the Accumulo init reports [Configuration.depreca= tion] INFO : fs.default.name is= deprecated. Instead use fs.defaultFS. However, the hadoop core-site.xml fi= le contains:
=C2=A0=C2=A0=C2=A0 <name>fs.defaultFS</name>
=C2=A0=C2=A0=C2=A0 <value>hdfs://localhost:9000</value>

Is there somewhere else that this value (fs.default.name) is specified? Could it be due to Ac= cumulo having a default value and not getting the override from hadoop beca= use of the problem listed above?

Thanks

Dave Patterson
patterd@gmail.com





--
=
Sean
--001a1146ff82d5e9b6051050faa8--