Return-Path: 
 <user-return-6501-apmail-hadoop-common-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-user-archive@www.apache.org
Delivered-To: apmail-hadoop-common-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 964BCF30D
	for <apmail-hadoop-common-user-archive@www.apache.org>;
 Wed, 27 Mar 2013 20:03:00 +0000 (UTC)
Received: (qmail 14151 invoked by uid 500); 27 Mar 2013 20:02:55 -0000
Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org
Received: (qmail 14036 invoked by uid 500); 27 Mar 2013 20:02:55 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 14025 invoked by uid 99); 27 Mar 2013 20:02:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Mar 2013 20:02:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.215.52] (HELO mail-la0-f52.google.com) (209.85.215.52)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Mar 2013 20:02:50 +0000
Received: by mail-la0-f52.google.com with SMTP id fs12so16472060lab.11
        for <user@hadoop.apache.org>; Wed, 27 Mar 2013 13:02:28 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20120113;
        h=x-received:mime-version:from:date:message-id:subject:to
         :content-type:x-gm-message-state;
        bh=2V2vCEMX2sKOihWRXNRKxjl2GXsOtcT8z/ye8HUgM1o=;
        b=KFCD75VsiWs0Y9xlW60gY/RYPmEgHfrhXMjXsntkW1PiNzfbwJbVJp4xEXTZ7E/mem
         vdag+fMYd3cBx+QxpzYvdrHA9DoP1iDVuEShg8dprWllXLD3ZJpXiV4OE3Zh/q/x/moq
         oCua3qHxXiivmwsUO0K5sMiztH8uVLacr4QEx/RJQmJAinIUa26cyIng74lNDE7eQnyI
         YJR3YOERXSgNTw+NDwgjYAXF3FqOcW3gW4r6XhKJHWRsMv245eT7CeiLQDOovR5oANYL
         ursnK0ZR6cQWJx6GNI20B+4D50x83YvsKf+olrQ73Q7lYo2t48N62N9f4uhqiwg7Hajd
         UqZg==
X-Received: by 10.112.5.6 with SMTP id o6mr10680655lbo.57.1364414548396; Wed,
 27 Mar 2013 13:02:28 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.152.6.131 with HTTP; Wed, 27 Mar 2013 13:02:08 -0700 (PDT)
From: Marcos Sousa <falecomigo@marcossousa.com>
Date: Wed, 27 Mar 2013 17:02:08 -0300
Message-ID: 
 <CAB1vRXRWnGhtDmWdR9WQ+soHGNM-pZkykAftBzSVRaLY8RdDGA@mail.gmail.com>
Subject: Hadoop Mapreduce fails with permission management enabled
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=14dae94ee2537be61804d8ed8498
X-Gm-Message-State: 
 ALoCoQk8qQbUn3oBT7RSCu50dikDJeJ1RRrOlD8xCJmRIBS6I8hp5ddiguJAGcoNutWOlpiHIGJ5
X-Virus-Checked: Checked by ClamAV on apache.org

--14dae94ee2537be61804d8ed8498
Content-Type: text/plain; charset=ISO-8859-1

I enabled the permission management in my hadoop cluster, but I'm facing a
problem sending jobs with pig. This is the scenario:

1 - I have hadoop/hadoop user

2 - I have myuserapp/myuserapp user that runs PIG script.

3 - We setup the path /myapp to be owned by myuserapp

4 - We set pig.temp.dir to /myapp/pig/tmp

But when we pig try to run the jobs we got the following error:

job_201303221059_0009    all_actions,filtered,raw_data    DISTINCT
Message: Job failed! Error - Job initialization failed:
org.apache.hadoop.security.AccessControlException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=realtime, access=EXECUTE,
inode="system":hadoop:supergroup:rwx------

 Hadoop jobtracker requires this permission to statup it's server.

My hadoop policy looks like:

<property>
<name>security.client.datanode.protocol.acl</name>
<value>hadoop,myuserapp supergroup,myuserapp</value>
</property>
<property>
<name>security.inter.tracker.protocol.acl</name>
<value>hadoop,myuserapp supergroup,myuserapp</value>
</property>
<property>
<name>security.job.submission.protocol.acl</name>
<value>hadoop,myuserapp supergroup,myuserapp</value>
<property>

My hdfs-site.xml:

<property>
<name>dfs.permissions</name>
<value>true</value>
</property>

<property>
 <name>dfs.datanode.data.dir.perm</name>
 <value>755</value>
</property>

<property>
 <name>dfs.web.ugi</name>
 <value>hadoop,supergroup</value>
</property>

My core site:

...
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
...

And finally my mapred-site.xml

...
<property>
 <name>mapred.local.dir</name>
 <value>/tmp/mapred</value>
</property>

<property>
 <name>mapreduce.jobtracker.jobhistory.location</name>
 <value>/opt/logs/hadoop/history</value>
</property>
<property>
<name>mapreduce.jobtracker.staging.root.dir</name>
<value>/user</value>
</property>

Is there a missing configuration? How can I deal with multiples users
running jobs in a restrict HDFS cluster?

--14dae94ee2537be61804d8ed8498
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><p style=3D"margin:0px 0px 1em;padding:0px;border:0px;font=
-size:14px;vertical-align:baseline;clear:both;word-wrap:break-word;color:rg=
b(0,0,0);font-family:Arial,&#39;Liberation Sans&#39;,&#39;DejaVu Sans&#39;,=
sans-serif;line-height:18px">

I enabled the permission management in my hadoop cluster, but I&#39;m facin=
g a problem sending jobs with pig. This is the scenario:</p><p style=3D"mar=
gin:0px 0px 1em;padding:0px;border:0px;font-size:14px;vertical-align:baseli=
ne;clear:both;word-wrap:break-word;color:rgb(0,0,0);font-family:Arial,&#39;=
Liberation Sans&#39;,&#39;DejaVu Sans&#39;,sans-serif;line-height:18px">

1 - I have hadoop/hadoop user</p><p style=3D"margin:0px 0px 1em;padding:0px=
;border:0px;font-size:14px;vertical-align:baseline;clear:both;word-wrap:bre=
ak-word;color:rgb(0,0,0);font-family:Arial,&#39;Liberation Sans&#39;,&#39;D=
ejaVu Sans&#39;,sans-serif;line-height:18px">

2 - I have myuserapp/myuserapp user that runs PIG script.</p><p style=3D"ma=
rgin:0px 0px 1em;padding:0px;border:0px;font-size:14px;vertical-align:basel=
ine;clear:both;word-wrap:break-word;color:rgb(0,0,0);font-family:Arial,&#39=
;Liberation Sans&#39;,&#39;DejaVu Sans&#39;,sans-serif;line-height:18px">

3 - We setup the path /myapp to be owned by myuserapp</p><p style=3D"margin=
:0px 0px 1em;padding:0px;border:0px;font-size:14px;vertical-align:baseline;=
clear:both;word-wrap:break-word;color:rgb(0,0,0);font-family:Arial,&#39;Lib=
eration Sans&#39;,&#39;DejaVu Sans&#39;,sans-serif;line-height:18px">

4 - We set pig.temp.dir to /myapp/pig/tmp</p><p style=3D"margin:0px 0px 1em=
;padding:0px;border:0px;font-size:14px;vertical-align:baseline;clear:both;w=
ord-wrap:break-word;color:rgb(0,0,0);font-family:Arial,&#39;Liberation Sans=
&#39;,&#39;DejaVu Sans&#39;,sans-serif;line-height:18px">

But when we pig try to run the jobs we got the following error:</p><p style=
=3D"margin:0px 0px 1em;padding:0px;border:0px;font-size:14px;vertical-align=
:baseline;clear:both;word-wrap:break-word;color:rgb(0,0,0);font-family:Aria=
l,&#39;Liberation Sans&#39;,&#39;DejaVu Sans&#39;,sans-serif;line-height:18=
px">

</p><pre style=3D"margin-top:0px;margin-bottom:10px;padding:5px;border:0px;=
vertical-align:baseline;background-color:rgb(238,238,238);font-family:Conso=
las,Menlo,Monaco,&#39;Lucida Console&#39;,&#39;Liberation Mono&#39;,&#39;De=
jaVu Sans Mono&#39;,&#39;Bitstream Vera Sans Mono&#39;,&#39;Courier New&#39=
;,monospace,serif;overflow:auto;width:auto;max-height:600px">

<code style=3D"margin:0px;padding:0px;border:0px;vertical-align:baseline;fo=
nt-family:Consolas,Menlo,Monaco,&#39;Lucida Console&#39;,&#39;Liberation Mo=
no&#39;,&#39;DejaVu Sans Mono&#39;,&#39;Bitstream Vera Sans Mono&#39;,&#39;=
Courier New&#39;,monospace,serif">job_201303221059_0009    all_actions,filt=
ered,raw_data    DISTINCT    Message: Job failed! Error - Job initializatio=
n failed: org.apache.hadoop.security.AccessControlException: org.apache.had=
oop.security.AccessControlException: Permission denied: user=3Drealtime, ac=
cess=3DEXECUTE, inode=3D&quot;system&quot;:hadoop:supergroup:rwx------</cod=
e></pre>

<p></p><p style=3D"margin:0px 0px 1em;padding:0px;border:0px;font-size:14px=
;vertical-align:baseline;clear:both;word-wrap:break-word;color:rgb(0,0,0);f=
ont-family:Arial,&#39;Liberation Sans&#39;,&#39;DejaVu Sans&#39;,sans-serif=
;line-height:18px">

</p><p style=3D"margin:0px 0px 1em;padding:0px;border:0px;vertical-align:ba=
seline;clear:both;word-wrap:break-word">Hadoop jobtracker requires this per=
mission to statup it&#39;s server.</p><p style=3D"margin:0px 0px 1em;paddin=
g:0px;border:0px;vertical-align:baseline;clear:both;word-wrap:break-word">

My hadoop policy looks like:</p><pre style=3D"margin-top:0px;margin-bottom:=
10px;padding:5px;border:0px;vertical-align:baseline;background-color:rgb(23=
8,238,238);font-family:Consolas,Menlo,Monaco,&#39;Lucida Console&#39;,&#39;=
Liberation Mono&#39;,&#39;DejaVu Sans Mono&#39;,&#39;Bitstream Vera Sans Mo=
no&#39;,&#39;Courier New&#39;,monospace,serif;overflow:auto;width:auto;max-=
height:600px">

<code style=3D"margin:0px;padding:0px;border:0px;vertical-align:baseline;fo=
nt-family:Consolas,Menlo,Monaco,&#39;Lucida Console&#39;,&#39;Liberation Mo=
no&#39;,&#39;DejaVu Sans Mono&#39;,&#39;Bitstream Vera Sans Mono&#39;,&#39;=
Courier New&#39;,monospace,serif">&lt;property&gt;
&lt;name&gt;security.client.datanode.protocol.acl&lt;/name&gt;
&lt;value&gt;hadoop,myuserapp supergroup,myuserapp&lt;/value&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;security.inter.tracker.protocol.acl&lt;/name&gt;
&lt;value&gt;hadoop,myuserapp supergroup,myuserapp&lt;/value&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;security.job.submission.protocol.acl&lt;/name&gt;
&lt;value&gt;hadoop,myuserapp supergroup,myuserapp&lt;/value&gt;
&lt;property&gt;
</code></pre><p style=3D"margin:0px 0px 1em;padding:0px;border:0px;vertical=
-align:baseline;clear:both;word-wrap:break-word">My hdfs-site.xml:</p><pre =
style=3D"margin-top:0px;margin-bottom:10px;padding:5px;border:0px;vertical-=
align:baseline;background-color:rgb(238,238,238);font-family:Consolas,Menlo=
,Monaco,&#39;Lucida Console&#39;,&#39;Liberation Mono&#39;,&#39;DejaVu Sans=
 Mono&#39;,&#39;Bitstream Vera Sans Mono&#39;,&#39;Courier New&#39;,monospa=
ce,serif;overflow:auto;width:auto;max-height:600px">

<code style=3D"margin:0px;padding:0px;border:0px;vertical-align:baseline;fo=
nt-family:Consolas,Menlo,Monaco,&#39;Lucida Console&#39;,&#39;Liberation Mo=
no&#39;,&#39;DejaVu Sans Mono&#39;,&#39;Bitstream Vera Sans Mono&#39;,&#39;=
Courier New&#39;,monospace,serif">&lt;property&gt;
&lt;name&gt;dfs.permissions&lt;/name&gt;
&lt;value&gt;true&lt;/value&gt;
&lt;/property&gt;

&lt;property&gt;
 &lt;name&gt;dfs.datanode.data.dir.perm&lt;/name&gt;
 &lt;value&gt;755&lt;/value&gt;
&lt;/property&gt;

&lt;property&gt;
 &lt;name&gt;dfs.web.ugi&lt;/name&gt;
 &lt;value&gt;hadoop,supergroup&lt;/value&gt;
&lt;/property&gt;
</code></pre><p style=3D"margin:0px 0px 1em;padding:0px;border:0px;vertical=
-align:baseline;clear:both;word-wrap:break-word">My core site:</p><pre styl=
e=3D"margin-top:0px;margin-bottom:10px;padding:5px;border:0px;vertical-alig=
n:baseline;background-color:rgb(238,238,238);font-family:Consolas,Menlo,Mon=
aco,&#39;Lucida Console&#39;,&#39;Liberation Mono&#39;,&#39;DejaVu Sans Mon=
o&#39;,&#39;Bitstream Vera Sans Mono&#39;,&#39;Courier New&#39;,monospace,s=
erif;overflow:auto;width:auto;max-height:600px">

<code style=3D"margin:0px;padding:0px;border:0px;vertical-align:baseline;fo=
nt-family:Consolas,Menlo,Monaco,&#39;Lucida Console&#39;,&#39;Liberation Mo=
no&#39;,&#39;DejaVu Sans Mono&#39;,&#39;Bitstream Vera Sans Mono&#39;,&#39;=
Courier New&#39;,monospace,serif">...
&lt;property&gt;
&lt;name&gt;hadoop.security.authorization&lt;/name&gt;
&lt;value&gt;true&lt;/value&gt;
&lt;/property&gt;
...
</code></pre><p style=3D"margin:0px 0px 1em;padding:0px;border:0px;vertical=
-align:baseline;clear:both;word-wrap:break-word">And finally my mapred-site=
.xml</p><pre style=3D"margin-top:0px;margin-bottom:10px;padding:5px;border:=
0px;vertical-align:baseline;background-color:rgb(238,238,238);font-family:C=
onsolas,Menlo,Monaco,&#39;Lucida Console&#39;,&#39;Liberation Mono&#39;,&#3=
9;DejaVu Sans Mono&#39;,&#39;Bitstream Vera Sans Mono&#39;,&#39;Courier New=
&#39;,monospace,serif;overflow:auto;width:auto;max-height:600px">

<code style=3D"margin:0px;padding:0px;border:0px;vertical-align:baseline;fo=
nt-family:Consolas,Menlo,Monaco,&#39;Lucida Console&#39;,&#39;Liberation Mo=
no&#39;,&#39;DejaVu Sans Mono&#39;,&#39;Bitstream Vera Sans Mono&#39;,&#39;=
Courier New&#39;,monospace,serif">...
&lt;property&gt;
 &lt;name&gt;mapred.local.dir&lt;/name&gt;
 &lt;value&gt;/tmp/mapred&lt;/value&gt;
&lt;/property&gt;

&lt;property&gt;
 &lt;name&gt;mapreduce.jobtracker.jobhistory.location&lt;/name&gt;
 &lt;value&gt;/opt/logs/hadoop/history&lt;/value&gt;
&lt;/property&gt;

<span style=3D"color:rgb(0,0,0);font-family:Arial,FreeSans,Helvetica,sans-s=
erif;font-size:13px;line-height:17px;white-space:normal;background-color:rg=
b(240,240,240)">&lt;property&gt;</span><br style=3D"color:rgb(0,0,0);font-f=
amily:Arial,FreeSans,Helvetica,sans-serif;font-size:13px;line-height:17px;w=
hite-space:normal;background-color:rgb(240,240,240)">

<span style=3D"color:rgb(0,0,0);font-family:Arial,FreeSans,Helvetica,sans-s=
erif;font-size:13px;line-height:17px;white-space:normal;background-color:rg=
b(240,240,240)">&lt;name&gt;mapreduce.jobtracker.staging.root.dir&lt;/name&=
gt;</span><br style=3D"color:rgb(0,0,0);font-family:Arial,FreeSans,Helvetic=
a,sans-serif;font-size:13px;line-height:17px;white-space:normal;background-=
color:rgb(240,240,240)">

<span style=3D"color:rgb(0,0,0);font-family:Arial,FreeSans,Helvetica,sans-s=
erif;font-size:13px;line-height:17px;white-space:normal;background-color:rg=
b(240,240,240)">&lt;value&gt;/user&lt;/value&gt;</span><br style=3D"color:r=
gb(0,0,0);font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:13px;li=
ne-height:17px;white-space:normal;background-color:rgb(240,240,240)">

<span style=3D"color:rgb(0,0,0);font-family:Arial,FreeSans,Helvetica,sans-s=
erif;font-size:13px;line-height:17px;white-space:normal;background-color:rg=
b(240,240,240)">&lt;/property&gt;</span>
</code></pre><p style=3D"margin:0px 0px 1em;padding:0px;border:0px;vertical=
-align:baseline;clear:both;word-wrap:break-word">Is there a missing configu=
ration? How can I deal with multiples users running jobs in a restrict HDFS=
 cluster?</p>

<p></p></div>

--14dae94ee2537be61804d8ed8498--