hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: hadoop.job.ugi backwards compatibility
Date Mon, 13 Sep 2010 23:23:02 GMT
On Mon, Sep 13, 2010 at 1:04 PM, Owen O'Malley <omalley@apache.org> wrote:

> On Mon, Sep 13, 2010 at 11:10 AM, Todd Lipcon <todd@cloudera.com> wrote:
> > Yep, but there are plenty of 10 node clusters out there that do important
> > work at small startups or single-use-case installations, too. We need to
> > provide scalability and security features that work for the 100+ node
> > clusters but also not leave the beginners in the dust.
> 10 node clusters are an important use case, but creating the user
> accounts on those clusters is very easy because of the few users.
> Futhermore, if the accounts aren't there it just means the users have
> no groups. Which for a single use system with security turned off
> isn't the end of the world.
> > But I think there are plenty of people out there who have built small
> > webapps, shell scripts, cron jobs, etc that use hadoop.job.ugi on some
> > shared account to impersonate other users.
> I'd be surprised. At Yahoo, the primary problem came with people
> screen scraping the jobtracker http. With security turned off that
> isn't an issue. Again, it isn't hard, just the evolving interface of
> UserGroupInformation changed. With security, we tried really hard to
> maintain backwards compatibility and succeeded for the vast (99%+)
> majority of the users.
> > Perhaps I am estimating
> > incorrectly - that's why I wanted this discussion on a user-facing list
> > rather than a dev-facing list.
> Obviously the pointer is there for them to follow into the rabbit hole
> of the dev lists. *grin*
> > Another example use case that I do a lot on non-secure clusters is:
> hadoop
> > fs -Dhadoop.job.ugi=hadoop,hadoop <something I want to do as a
> superuser>.
> > The permissions model we have in 0.20 obviously isn't secure, but it's
> nice
> > to avoid accidental mistakes, and making it easy to "sudo" like that is
> > handy.
> It might make sense to add a new switch ( -user ?) to hadoop fs that
> does a doAs before doing the
> shell command. You could even make it fancy and try to be a proxy user
> if security is turned on.

Yep, I agree - I think either (ab)using proxyuser functionality or adding
some new "sudoers" like configuration would be very handy and we should do

> > Regardless of our particular opinions, isn't our policy that we cannot
> break
> > API compatibility between versions without a one-version deprecation
> period?
> There wasn't a way to keep UGI stable. It was a broken design before
> the security work. It is marked evolving so we try to minimize
> breakage, but it isn't prohibited.
I agree that keeping API compatibility for UGI was probably impossible, and
respect that. But it would certainly be very easy to do a patch like the

JobClient(Configuration conf) {
  if (conf.get("hadoop.job.ugi") != null &&
UserGroupInformation.isSecurityEnabled()) {
    LOG.warn("Stop being evil. Don't use hadoop.job.ugi! RAAWR");
    UserGroupInformation.createRemoteUser(...).doAs() { create proxy }
  } else {
    create normal RPC proxy;

... and the same on the HDFS side.

Would you -1 such a compatibility layer?


Todd Lipcon
Software Engineer, Cloudera

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message