Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B94F27E0 for ; Fri, 24 Aug 2012 13:50:22 +0000 (UTC) Received: (qmail 99785 invoked by uid 500); 24 Aug 2012 13:50:17 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 99657 invoked by uid 500); 24 Aug 2012 13:50:17 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 99650 invoked by uid 99); 24 Aug 2012 13:50:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Aug 2012 13:50:17 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jfwalton@us.ibm.com designates 32.97.182.145 as permitted sender) Received: from [32.97.182.145] (HELO e5.ny.us.ibm.com) (32.97.182.145) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Aug 2012 13:50:06 +0000 Received: from /spool/local by e5.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 24 Aug 2012 09:49:45 -0400 Received: from d01dlp01.pok.ibm.com (9.56.250.166) by e5.ny.us.ibm.com (192.168.1.105) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 24 Aug 2012 09:49:36 -0400 Received: from d01relay02.pok.ibm.com (d01relay02.pok.ibm.com [9.56.227.234]) by d01dlp01.pok.ibm.com (Postfix) with ESMTP id 67ED938C803F for ; Fri, 24 Aug 2012 09:49:35 -0400 (EDT) Received: from d01av03.pok.ibm.com (d01av03.pok.ibm.com [9.56.224.217]) by d01relay02.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q7ODnYOh140868 for ; Fri, 24 Aug 2012 09:49:35 -0400 Received: from d01av03.pok.ibm.com (loopback [127.0.0.1]) by d01av03.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q7ODnYrX025114 for ; Fri, 24 Aug 2012 10:49:34 -0300 Received: from d01ml007.pok.ibm.com (d01ml007.pok.ibm.com [9.63.8.142]) by d01av03.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q7ODnYek025105 for ; Fri, 24 Aug 2012 10:49:34 -0300 In-Reply-To: References: To: user@hadoop.apache.org MIME-Version: 1.0 Subject: Re: Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1 X-KeepSent: 2E935E40:56BFE6E6-85257A64:004BB391; type=4; name=$KeepSent X-Mailer: Lotus Notes Release 8.5.1FP5 SHF29 November 12, 2010 From: James F Walton Message-ID: Date: Fri, 24 Aug 2012 09:49:37 -0400 X-MIMETrack: Serialize by Router on D01ML007/01/M/IBM(Release 8.5.3FP2IF1|July 25, 2012) at 08/24/2012 09:49:33, Serialize complete at 08/24/2012 09:49:33 Content-Type: multipart/alternative; boundary="=_alternative 004BF19885257A64_=" X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12082413-5930-0000-0000-00000B3830EC This is a multipart message in MIME format. --=_alternative 004BF19885257A64_= Content-Type: text/plain; charset="US-ASCII" Good point. I did some reading over there and it looks like even the IBM packaging of Hadoop (BigInsights) is geared strictly towards Linux. Both the Enterprise and Basic editions only list support for Red Hat or SuSE Enterprise Linux. So, thanks for those that chimed in. Off to platform migration planning I go. From: Mike Spreitzer/Watson/IBM@IBMUS To: user@hadoop.apache.org Date: 08/24/2012 09:28 AM Subject: Re: Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1 While I am not involved with it, I am aware that IBM has a Hadoop distribution of its own; I suspect you can expect better coverage from it than from the base distribution. Here is a pointer: http://www-01.ibm.com/software/data/infosphere/biginsights/ Regards, Mike From: James F Walton/Southbury/IBM@IBMUS To: user@hadoop.apache.org Date: 08/24/2012 09:18 AM Subject: Re: Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1 I'm not entirely sure it's fair to say it's a bug in the IBM JVM. It's a current implementation difference. They are still using platform-specific authentication modules for Windows, AIX, and Linux. Even Sun/Oracle Java has a specific SolarisLoginModule, which is deprecated but still available. Essentially, depending on what OS/architecture you are on, one of the following will exist: com.ibm.security.auth.module.NTLoginModule com.ibm.security.auth.module.LinuxLoginModule com.ibm.security.auth.module.AIXLoginModule com.ibm.security.auth.module.AIX64LoginModule Basically, instead of two possible outcomes, there are four. From: Steve Loughran To: user@hadoop.apache.org Date: 08/22/2012 03:09 PM Subject: Re: Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1 This is something you ought to raise with the IBM JVM team, as it does appear to be a bug in their JVM. On 21 August 2012 10:11, James F Walton wrote: Found part of the reason while digging through UserGroupInformation.java /* Return the OS login module class name */ private static String getOSLoginModuleName() { if (System.getProperty("java.vendor").contains("IBM")) { return windows ? "com.ibm.security.auth.module.NTLoginModule" : "com.ibm.security.auth.module.LinuxLoginModule"; } else { return windows ? "com.sun.security.auth.module.NTLoginModule" : "com.sun.security.auth.module.UnixLoginModule"; } } So basically, if you use IBM java, then you must be on either Windows or Linux. IBM's java appears to have platform specific LoginModules, there's AIXLoginModule for 32-bit java on AIX, and AIX64LoginModule for 64-bit java on AIX; however, the IBM Linux module appears not to have any 32-bit vs 64-bit differentiation. So, unless anyone has a means to disable this whole security setup (I'm not using a hadoop cluster or anything), or wants to dive headlong into making the necessary code changes (which I presume from my cursory scanning would include a little more than just the above snippet, like the getOSPrincipalClass as well), I guess I'll need to look at moving our crawler from AIX to Linux. My java coding skills are not top notch, though I could probably fix it if the necessary updates didn't get too convoluted beyond what I think it might require. I would hope there are not too many other instances where AIX incompatibility would rear it's head since prior to the security features overhaul it all worked fine. Thoughts? James From: James F Walton/Southbury/IBM@IBMUS To: user@hadoop.apache.org Date: 08/17/2012 02:47 PM Subject: Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1 When implementing Nutch 1.0 awhile back, we had to point our scripts to /opt/freeware/bin to allow Nutch's Hadoop code to utilize the more Linux-like versions of various system commands (like df) in order for Hadoop to function properly on AIX. After doing that, we've had no issues with Nutch's hadoop implementation from 1.0 through 1.4. Now I'm attempting up migrate my existing, functional configurations from my Nutch 1.4 installation to Nutch 1.5.1, which is now using Hadoop 1.0.3. Now when I attempt to run a crawl, I'm getting these errors that seem to be coming from Hadoop, and seem to want to use a LinuxLoginModule class (see below) Is there some configuration setting, plugin or jar file, etc that is missing from the new version to make this all work on AIX again? 2012-08-02 20:03:21,271 ERROR crawl.Injector - Injector: java.lang.RuntimeException: java.io.IOException: failure to login at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:546) at org.apache.hadoop.mapred.FileInputFormat.addInputPath(FileInputFormat.java:336) at org.apache.nutch.crawl.Injector.inject(Injector.java:209) at org.apache.nutch.crawl.Injector.run(Injector.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.nutch.crawl.Injector.main(Injector.java:238) Caused by: java.io.IOException: failure to login at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452) at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:1494) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1395) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123) at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:542) ... 5 more Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: com.ibm.security.auth.module.LinuxLoginModule at javax.security.auth.login.LoginContext.invoke(LoginContext.java:834) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:209) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:732) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:729) at javax.security.auth.login.LoginContext.login(LoginContext.java:599) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:471) ... 11 more James --=_alternative 004BF19885257A64_= Content-Type: text/html; charset="US-ASCII" Good point.  I did some reading over there and it looks like even the IBM packaging of Hadoop (BigInsights) is geared strictly towards Linux.  Both the Enterprise and Basic editions only list support for Red Hat or SuSE Enterprise Linux.

So, thanks for those that chimed in.  Off to platform migration planning I go.




From:        Mike Spreitzer/Watson/IBM@IBMUS
To:        user@hadoop.apache.org
Date:        08/24/2012 09:28 AM
Subject:        Re: Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1



While I am not involved with it, I am aware that IBM has a Hadoop distribution of its own; I suspect you can expect better coverage from it than from the base distribution.  Here is a pointer: http://www-01.ibm.com/software/data/infosphere/biginsights/

Regards,
Mike



From:        James F Walton/Southbury/IBM@IBMUS
To:        user@hadoop.apache.org
Date:        08/24/2012 09:18 AM
Subject:        Re: Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1



I'm not entirely sure it's fair to say it's a bug in the IBM JVM.  It's a current implementation difference.  They are still using platform-specific authentication modules for Windows, AIX, and Linux.  Even Sun/Oracle Java has a specific SolarisLoginModule, which is deprecated but still available.

Essentially, depending on what OS/architecture you are on, one of the following will exist:
com.ibm.security.auth.module.NTLoginModule
com.ibm.security.auth.module.LinuxLoginModule
com.ibm.security.auth.module.AIXLoginModule
com.ibm.security.auth.module.AIX64LoginModule

Basically, instead of two possible outcomes, there are four.



From:        Steve Loughran <stevel@hortonworks.com>
To:        user@hadoop.apache.org
Date:        08/22/2012 03:09 PM
Subject:        Re: Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1



This is something you ought to raise with the IBM JVM team, as it does appear to be a bug in their JVM.

On 21 August 2012 10:11, James F Walton <jfwalton@us.ibm.com> wrote:
Found part of the reason while digging through UserGroupInformation.java

/* Return the OS login module class name */
private static String getOSLoginModuleName() {
  if (System.getProperty("java.vendor").contains("IBM")) {
    return windows ? "com.ibm.security.auth.module.NTLoginModule"
     : "com.ibm.security.auth.module.LinuxLoginModule";
  } else {
    return windows ? "com.sun.security.auth.module.NTLoginModule"
      : "com.sun.security.auth.module.UnixLoginModule";
  }
}


So basically, if you use IBM java, then you must be on either Windows or Linux.  IBM's java appears to have platform specific LoginModules, there's AIXLoginModule for 32-bit java on AIX, and AIX64LoginModule for 64-bit java on AIX; however, the IBM Linux module appears not to have any 32-bit vs 64-bit differentiation.

So, unless anyone has a means to disable this whole security setup (I'm not using a hadoop cluster or anything), or wants to dive headlong into making the necessary code changes (which I presume from my cursory scanning would include a little more than just the above snippet, like the getOSPrincipalClass as well), I guess I'll need to look at moving our crawler from AIX to Linux.

My java coding skills are not top notch, though I could probably fix it if the necessary updates didn't get too convoluted beyond what I think it might require.  I would hope there are not too many other instances where AIX incompatibility would rear it's head since prior to the security features overhaul it all worked fine.

Thoughts?

James




From:        James F Walton/Southbury/IBM@IBMUS
To:        user@hadoop.apache.org
Date:        08/17/2012 02:47 PM
Subject:        Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1




When implementing Nutch 1.0 awhile back, we had to point our scripts to /opt/freeware/bin to allow Nutch's Hadoop code to utilize the more Linux-like versions of various system commands (like df) in order for Hadoop to function properly on AIX.  After doing that, we've had no issues with Nutch's hadoop implementation from 1.0 through 1.4.

Now I'm attempting up migrate my existing, functional configurations from my Nutch 1.4 installation to Nutch 1.5.1, which is now using Hadoop 1.0.3.
Now when I attempt to run a crawl, I'm getting these errors that seem to be coming from Hadoop, and seem to want to use a LinuxLoginModule class (see below)

Is there some configuration setting, plugin or jar file, etc that is missing from the new version to make this all work on AIX again?

2012-08-02 20:03:21,271 ERROR crawl.Injector - Injector: java.lang.RuntimeException: java.io.IOException: failure to login
     at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:546)
     at org.apache.hadoop.mapred.FileInputFormat.addInputPath(FileInputFormat.java:336)
     at org.apache.nutch.crawl.Injector.inject(Injector.java:209)
     at org.apache.nutch.crawl.Injector.run(Injector.java:248)
     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
     at org.apache.nutch.crawl.Injector.main(Injector.java:238)
Caused by: java.io.IOException: failure to login
     at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490)
     at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452)
     at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1494)
     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1395)
     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
     at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:542)
     ... 5 more
Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: com.ibm.security.auth.module.LinuxLoginModule
     at javax.security.auth.login.LoginContext.invoke(LoginContext.java:834)
     at javax.security.auth.login.LoginContext.access$000(LoginContext.java:209)
     at javax.security.auth.login.LoginContext$5.run(LoginContext.java:732)
     at java.security.AccessController.doPrivileged(AccessController.java:284)
     at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:729)
     at javax.security.auth.login.LoginContext.login(LoginContext.java:599)
     at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:471)
     ... 11 more



James

--=_alternative 004BF19885257A64_=--