Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of chen.song.82@gmail.com
 designates 209.85.213.176 as permitted sender)
MIME-Version: 1.0
Date: Mon, 13 Aug 2012 12:21:00 -0400
Message-ID: 
 <CAGF+3rZ15j0bJFw-DjXV0Yp0dcy9VHKum=sguHS8jSa9G1rrhQ@mail.gmail.com>
Subject: group assignment on HDFS from Hadoop and Hive
From: Chen Song <chen.song.82@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba6e8c5a5624a904c72814b0

--90e6ba6e8c5a5624a904c72814b0
Content-Type: text/plain; charset=ISO-8859-1

I am wondering how Hadoop assign groups when dirs/files are being created
by a user and below are some tests I have done. In my cluster, group hadoop
is configured as the supergroup.

> hadoop fs -ls /tmp
drwxrwxrwx   - abc hadoop          0 2012-08-10 23:02 /tmp/abc
drwxrwxrwx   - def other_group          0 2012-08-10 23:02 /tmp/def

> groups apache
apache: apache wheel

> sudo -u apache hadoop fs -put somefile /tmp/abc
> hadoop fs -ls /tmp/abc
-rw-rw-r--   3 apache hadoop     120962 2012-08-13 16:03 /tmp/abc/somefile

> sudo -u apache hadoop fs -put somefile /tmp/def
> hadoop fs -lsr /tmp/def
-rw-rw-r--   3 apache other_group     120962 2012-08-13
16:03 /tmp/abc/somefile

*Based on the experiments above, it looks like the file got pushed on hdfs
is always inheriting its group from the parent including folder. Is that
always the case?*

A follow-up question on one finding in Hive is: when executing a query to
overwrite a table (or a partition within a table), the newly written
overriding directory always end up as belong to HDFS's supergroup, no
matter what context it is running from
1. The user who is executing the hive query
2. The group where the user belongs to
3. The group the parent table directory is belonging to.
*Is it always expected in Hive?*

For example, table A is stored on /path/A and is partitioned on column
dh. /path/A is with group *other_group*.
After running *insert overwrite A partition (dh = "12") select column list
from ... where ...*

/path/A/12 ends up with *hadoop* as always. This has contradicts to the
assumption of inheritance I have drawn out above. Any thoughts would be
appreciated.

Thanks
Chen

--90e6ba6e8c5a5624a904c72814b0
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I am wondering how Hadoop assign groups when dirs/files are being created b=
y a user and below are some tests I have done. In my cluster, group hadoop =
is configured as the supergroup.<div><br></div><div>&gt; hadoop fs -ls /tmp=
</div>
<div>drwxrwxrwx =A0 - abc hadoop =A0 =A0 =A0 =A0 =A00 2012-08-10 23:02 /tmp=
/abc</div><div>drwxrwxrwx =A0 - def other_group =A0 =A0 =A0 =A0 =A00 2012-0=
8-10 23:02 /tmp/def</div><div><br></div><div><div>&gt; groups apache</div><=
div>apache: apache wheel</div>
</div><div><br></div><div>&gt; sudo -u apache hadoop fs -put somefile /tmp/=
abc</div><div>&gt; hadoop fs -ls /tmp/abc</div><div>-rw-rw-r-- =A0 3 apache=
 hadoop =A0 =A0 120962 2012-08-13 16:03 /tmp/abc/somefile</div><div><br></d=
iv>
<div>&gt; sudo -u apache hadoop fs -put somefile /tmp/def</div><div>&gt;=A0=
hadoop fs -lsr /tmp/def</div><div>-rw-rw-r-- =A0 3 apache=A0other_group=A0 =
=A0 =A0120962 2012-08-13 16:03=A0/tmp/abc/somefile</div><div><br></div><div=
><b>Based on the experiments above, it looks like the file got pushed on hd=
fs is always inheriting its group from the parent including folder. Is that=
 always the case?</b></div>
<div><br></div><div>A follow-up question on one finding in Hive is: when ex=
ecuting a query to overwrite a table (or a partition within a table), the n=
ewly written overriding directory always end up as belong to HDFS&#39;s sup=
ergroup, no matter what context it is running from</div>
<div>1. The user who is executing the hive query</div><div>2. The group whe=
re the user belongs to</div><div>3. The group the parent table directory is=
 belonging to.</div><div><b>Is it always expected in Hive?</b></div><div>
<br></div><div>For example, table A is stored on /path/A and is partitioned=
 on column dh.=A0/path/A is with group=A0<b>other_group</b>.</div><div>Afte=
r running <i>insert overwrite A partition (dh =3D &quot;12&quot;) select co=
lumn list from ... where ...</i></div>
<div><br></div><div>/path/A/12 ends up with <b>hadoop</b>=A0as always. This=
 has contradicts to the assumption of inheritance I have drawn out above. A=
ny thoughts would be appreciated.</div><div><br></div><div>Thanks<br>Chen</=
div>
<div><br></div><div><br><div><br></div><div><br></div></div>

--90e6ba6e8c5a5624a904c72814b0--