Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of rarecactus@gmail.com designates
 74.125.82.176 as permitted sender)
MIME-Version: 1.0
Sender: rarecactus@gmail.com
In-Reply-To: 
 <CAJs-t7MMsjsP=sA+a-7bQLvD05OiSqj4ic1L3AYSZP7zyHGoQw@mail.gmail.com>
References: 
 <CA+qbEUOzDJEzCnrW2fU2uPLm__a-_xqCJzEqLcQjCbQ6GXWi3w@mail.gmail.com>
	<CAJs-t7MMsjsP=sA+a-7bQLvD05OiSqj4ic1L3AYSZP7zyHGoQw@mail.gmail.com>
Date: Mon, 11 Jun 2012 16:20:35 -0700
Message-ID: 
 <CA+qbEUOjuca2ZMv9ohTiwJLhgqNTOOu10EG4--qMdPR-3O2i7g@mail.gmail.com>
Subject: Re: validating user IDs
From: Colin McCabe <cmccabe@alumni.cmu.edu>
To: hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Sure.  We could also find the current user ID and bake that into the
test as an "acceptable" UID.  If that makes sense.

Colin


On Mon, Jun 11, 2012 at 4:12 PM, Alejandro Abdelnur <tucu@cloudera.com> wro=
te:
> Colin,
>
> Would be possible using some kind of cmake config magic to set a macro to
> the current OS limit? Even if this means detecting the OS version and
> assuming its default limit.
>
> thx
>
> On Mon, Jun 11, 2012 at 3:57 PM, Colin McCabe <cmccabe@alumni.cmu.edu>wro=
te:
>
>> Hi all,
>>
>> I recently pulled the latest source, and ran a full build. =A0The
>> command line was this:
>> mvn compile -Pnative
>>
>> I was confronted with this:
>>
>> [INFO] Requested user cmccabe has id 500, which is below the minimum
>> allowed 1000
>> [INFO] FAIL: test-container-executor
>> [INFO] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
>> [INFO] 1 of 1 test failed
>> [INFO] Please report to mapreduce-dev@hadoop.apache.org
>> [INFO] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
>> [INFO] make[1]: *** [check-TESTS] Error 1
>> [INFO] make[1]: Leaving directory
>>
>> `/home/cmccabe/hadoop4/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-=
server/hadoop-yarn-server-nodemanager/target/native/container-executor'
>>
>> Needless to say, it didn't do much to improve my mood. =A0I was even
>> less happy when I discovered that -DskipTests has no effect on native
>> tests (they always run.) =A0See HADOOP-8480.
>>
>> Unfortunately, it seems like this problem is popping up more and more
>> in our native code. =A0It first appeared in test-task-controller (see
>> MAPREDUCE-2376) and then later in test-container-executor
>> (HADOOP-8499). =A0The basic problem seems to be the hardcoded assumption
>> that all user IDs below 1000 are system IDs.
>>
>> It is true that there are configuration files that can be changed to
>> alter the minimum user ID, but unfortunately these configuration files
>> are not used by the unit tests. =A0So anyone developing on a platform
>> where the user IDs start at 500 is now a second-class citizen, unable
>> to run unit tests. =A0This includes anyone running Red Hat, MacOS,
>> Fedora, etc.
>>
>> Personally, I can change my user ID. =A0It's a time-consuming process,
>> because I need to re-uid all files, but I can do it. =A0This luxury may
>> not be available to everyone, though-- developers who don't have root
>> on their machines, or are using a pre-assigned user ID to connect to
>> NFS come to mind.
>>
>> It's true that we could hack around this with environment variables.
>> It might even be possible to have Maven set these environment
>> variables automatically from the current user ID. =A0However, the larger
>> question I have here is whether this UID validation scheme even makes
>> any sense. =A0I have a user named "nobody" whose user ID is 65534.
>> Surely I should not be able to run map-reduce jobs as this user? =A0Yet,
>> under the current system, I can do exactly that. =A0The root of the
>> problem seems to be that there is both a default minimum and a default
>> maximum for "automatic" user IDs. =A0This configuration seems to be
>> stored in /etc/login.defs.
>>
>> On my system, it has:
>> SYSTEM_UID_MIN =A0 =A0 =A0 =A0 =A0 =A0100
>> SYSTEM_UID_MAX =A0 =A0 =A0 =A0 =A0 =A0499
>> UID_MIN =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0500
>> UID_MAX =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 60000
>>
>> So that means that anything over 60000 (like nobody) is not considered
>> a valid user ID for regular users.
>> We could potentially read this file (at least on Linux) and get more
>> sensible defaults.
>>
>> I am also curious if we could simply check whether the user we're
>> trying to run the job as has a valid login shell. =A0System users are
>> almost always set to have a login shell of /bin/false or
>> /sbin/nologin.
>>
>> Thoughts?
>> Colin
>>
>
>
>
> --
> Alejandro