Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of atm@cloudera.com designates
 209.85.220.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAJs-t7MMsjsP=sA+a-7bQLvD05OiSqj4ic1L3AYSZP7zyHGoQw@mail.gmail.com>
References: 
 <CA+qbEUOzDJEzCnrW2fU2uPLm__a-_xqCJzEqLcQjCbQ6GXWi3w@mail.gmail.com>
 <CAJs-t7MMsjsP=sA+a-7bQLvD05OiSqj4ic1L3AYSZP7zyHGoQw@mail.gmail.com>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Mon, 11 Jun 2012 16:15:32 -0700
Message-ID: 
 <CA+4052nYsBfyuiHeJEQ5ofqF=HqzDMMGgoixs4GP--R2cmrVFg@mail.gmail.com>
Subject: Re: validating user IDs
To: mapreduce-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3071c7109e4aa504c23a8826

--20cf3071c7109e4aa504c23a8826
Content-Type: text/plain; charset=ISO-8859-1

-hdfs-dev@
+mapreduce-dev@

Moving to the more-relevant mapreduce-dev.

--
Aaron T. Myers
Software Engineer, Cloudera


On Mon, Jun 11, 2012 at 4:12 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:

> Colin,
>
> Would be possible using some kind of cmake config magic to set a macro to
> the current OS limit? Even if this means detecting the OS version and
> assuming its default limit.
>
> thx
>
> On Mon, Jun 11, 2012 at 3:57 PM, Colin McCabe <cmccabe@alumni.cmu.edu
> >wrote:
>
> > Hi all,
> >
> > I recently pulled the latest source, and ran a full build.  The
> > command line was this:
> > mvn compile -Pnative
> >
> > I was confronted with this:
> >
> > [INFO] Requested user cmccabe has id 500, which is below the minimum
> > allowed 1000
> > [INFO] FAIL: test-container-executor
> > [INFO] ================================================
> > [INFO] 1 of 1 test failed
> > [INFO] Please report to mapreduce-dev@hadoop.apache.org
> > [INFO] ================================================
> > [INFO] make[1]: *** [check-TESTS] Error 1
> > [INFO] make[1]: Leaving directory
> >
> >
> `/home/cmccabe/hadoop4/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/container-executor'
> >
> > Needless to say, it didn't do much to improve my mood.  I was even
> > less happy when I discovered that -DskipTests has no effect on native
> > tests (they always run.)  See HADOOP-8480.
> >
> > Unfortunately, it seems like this problem is popping up more and more
> > in our native code.  It first appeared in test-task-controller (see
> > MAPREDUCE-2376) and then later in test-container-executor
> > (HADOOP-8499).  The basic problem seems to be the hardcoded assumption
> > that all user IDs below 1000 are system IDs.
> >
> > It is true that there are configuration files that can be changed to
> > alter the minimum user ID, but unfortunately these configuration files
> > are not used by the unit tests.  So anyone developing on a platform
> > where the user IDs start at 500 is now a second-class citizen, unable
> > to run unit tests.  This includes anyone running Red Hat, MacOS,
> > Fedora, etc.
> >
> > Personally, I can change my user ID.  It's a time-consuming process,
> > because I need to re-uid all files, but I can do it.  This luxury may
> > not be available to everyone, though-- developers who don't have root
> > on their machines, or are using a pre-assigned user ID to connect to
> > NFS come to mind.
> >
> > It's true that we could hack around this with environment variables.
> > It might even be possible to have Maven set these environment
> > variables automatically from the current user ID.  However, the larger
> > question I have here is whether this UID validation scheme even makes
> > any sense.  I have a user named "nobody" whose user ID is 65534.
> > Surely I should not be able to run map-reduce jobs as this user?  Yet,
> > under the current system, I can do exactly that.  The root of the
> > problem seems to be that there is both a default minimum and a default
> > maximum for "automatic" user IDs.  This configuration seems to be
> > stored in /etc/login.defs.
> >
> > On my system, it has:
> > SYSTEM_UID_MIN            100
> > SYSTEM_UID_MAX            499
> > UID_MIN                  500
> > UID_MAX                 60000
> >
> > So that means that anything over 60000 (like nobody) is not considered
> > a valid user ID for regular users.
> > We could potentially read this file (at least on Linux) and get more
> > sensible defaults.
> >
> > I am also curious if we could simply check whether the user we're
> > trying to run the job as has a valid login shell.  System users are
> > almost always set to have a login shell of /bin/false or
> > /sbin/nologin.
> >
> > Thoughts?
> > Colin
> >
>
>
>
> --
> Alejandro
>

--20cf3071c7109e4aa504c23a8826--