hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: Permission Denied
Date Mon, 02 Mar 2015 16:47:35 GMT
Splitting into three unix users is a good idea. Generally, none of the
linux users should need access to any of the local resources owned by the
others. (that is, the user running the accumulo processes shouldn't be able
to interfere with the backing files used by the HDFS processes).

By default, the linux user that drives a particular process will be
resolved to a Hadoop user by the NameNode process. Presuming your Accumulo
services are running under the linux user "accumulo", you should ensure
that user exists on the linux node that runs the NameNode.

The main issue with running init as the hadoop user is that by default it's
likely going to write the accumulo directories as owned by the user that
created them. Presuming you are using Accumulo because you have security
requirements, the common practice is to make sure only the user that runs
Accumulo processes can write to /accumulo and that only that user can read
/accumulo/tables and /accumulo/wal. This ensures that other users with
access to the HDFS cluster won't be able to bypass the cell-level access
controls provided by Accumulo.

While you are setting up HDFS directories, you should also create a home
directory for the user that runs Accumulo processes. If your HDFS instance
is set to use the trash feature (either in server configs or the client
configs made available to Accumulo), then by default Accumulo will attempt
to use it. Without a home directory, this will result in failures.
Alternatively, you can ensure Accumulo doesn't rely on the trash feature by
setting gc.trash.ignore in your accumulo-site.xml.

One other note:

> I edited the accumulo-site.xml so it now has
>  <property>
>    <name>instance.volumes</name>
>    <value>hdfs://localhost:9000/accumulo</value>
>    <description>comma separated list of URIs for volumes. example:
hdfs://localhost:9000/accumulo</description>
>  </property>

You will save yourself headache later if you stick with fully qualified
domain names for all HDFS, ZooKeeper, and Accumulo connections.

-- 
Sean

On Mon, Mar 2, 2015 at 8:13 AM, David Patterson <patterd@gmail.com> wrote:

> David,
>
> Thanks for the information. I've issued those two commands in my hadoop
> shell and still get the same error when I try to initialize accumulo in
> *its* shell. :
>
> 2015-03-02 13:30:41,175 [init.Initialize] FATAL: Failed to initialize
> filesystem
>    org.apache.hadoop.security.AccessControlException: Permission denied:
> user=accumulo, access=WRITE, inode="/accumulo":
>    accumulo.supergroup:supergroup:drwxr-xr-x
>
> My comment that I had 3 users was meant in a linux sense, not in a hadoop
> sense. So (to borrow terminoloy from RDF or XML) is there something I have
> to do in my hadoop setup (running under linix:hadoop) or my accumulo setup
> (running under linux:accumulo) so that the accumuulo I/O gets processed as
> from someone in the hadoop:supergroup?
>
>
> I tried running the accumulo init from the linux:hadoop user and it
> worked. I'm not sure if any permissions/etc were hosed by doing it there.
> I'll see.
>
> Thanks for you help.
>
> (By the way, is it wrong or a bad idea to split the work into three
> linux:users, or should it all be done in one linux:user space?)
>
> Dave Patterson
>
> On Sun, Mar 1, 2015 at 8:35 PM, dlmarion <dlmarion@comcast.net> wrote:
>
>> hadoop fs -mkdir /accumulo
>> hadoop fs -chown accumulo:supergroup /accumulo
>>
>>
>>
>> -------- Original message --------
>> From: David Patterson <patterd@gmail.com>
>> Date:03/01/2015 7:04 PM (GMT-05:00)
>> To: user@hadoop.apache.org
>> Cc:
>> Subject: Re: Permission Denied
>>
>> David,
>>
>> Thanks for the reply.
>>
>> Taking the questions in the opposite order, my accumulo-site.xml does not
>> have volumes specified.
>>
>> I edited the accumulo-site.xml so it now has
>>   <property>
>>     <name>instance.volumes</name>
>>     <value>hdfs://localhost:9000/accumulo</value>
>>     <description>comma separated list of URIs for volumes. example:
>> hdfs://localhost:9000/accumulo</description>
>>   </property>
>>
>> and got the same error.
>>
>> How can I precreate /accumulo ?
>>
>> Dave Patterson
>>
>> On Sun, Mar 1, 2015 at 3:50 PM, david marion <dlmarion@hotmail.com>
>> wrote:
>>
>>>  It looks like / is owned by hadoop.supergroup and the perms are 755.
>>> You could precreate /accumulo and chown it appropriately, or set the perms
>>> for / to 775. Init is trying to create /accumulo in hdfs as the accumulo
>>> user and your perms dont allow it.
>>>
>>>  Do you have instance.volumes set in accumulo-site.xml?
>>>
>>>
>>> -------- Original message --------
>>> From: David Patterson <patterd@gmail.com>
>>> Date:03/01/2015 3:36 PM (GMT-05:00)
>>> To: user@hadoop.apache.org
>>> Cc:
>>> Subject: Permission Denied
>>>
>>>        I'm trying to create an Accumulo/Hadoop/Zookeeper configuration
>>> on a single (Ubuntu) machine, with Hadoop 2.6.0, Zookeeper 3.4.6 and
>>> Accumulo 1.6.1.
>>>
>>>  I've got 3 userids for these components that are in the same group and
>>> no other users are in that group.
>>>
>>>  I have zookeeper running, and hadoop as well.
>>>
>>>  Hadoop's core-site.xml file has the hadoop.tmp.dir set to
>>> /app/hadoop/tmp.The /app/hadoop/tmp directory is owned by the hadoop user
>>> and has permissions that allow other members of the group to write
>>> (drwxrwxr-x).
>>>
>>>  When I try to initialize Accumulo, with bin/accumulo init, I get FATAL:
>>> Failed to initialize filesystem.
>>>  org.apache.hadoop.security.AccessControlException: Permission denied:
>>> user=accumulo, access=WRITE, inode="/":hadoop:supergroup:drwxr-xr-x
>>>
>>>  So, my main question is which directory do I need to give group-write
>>> permission so the accumulo user can write as needed so it can initialize?
>>>
>>>  The second problem is that the Accumulo init reports
>>> [Configuration.deprecation] INFO : fs.default.name is deprecated.
>>> Instead use fs.defaultFS. However, the hadoop core-site.xml file contains:
>>>     <name>fs.defaultFS</name>
>>>     <value>hdfs://localhost:9000</value>
>>>
>>>  Is there somewhere else that this value (fs.default.name) is
>>> specified? Could it be due to Accumulo having a default value and not
>>> getting the override from hadoop because of the problem listed above?
>>>
>>>  Thanks
>>>
>>>  Dave Patterson
>>>  patterd@gmail.com
>>>
>>
>>
>


-- 
Sean

Mime
View raw message