hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Krogen <ekro...@linkedin.com.INVALID>
Subject Re: Kerberised JobHistory Server not starting: User jhs trying to create the /mr-history/done directory
Date Thu, 20 Jul 2017 16:04:27 GMT
Hi Kevin,

Since you are using the "jhs" keytab with principal "jhs/_HOST@REALM.TLD",
the JHS is authenticating itself as the jhs user (which is the actual
important part, rather than the user the process is running as). If you
want it to be the "mapred" user, you should change the keytab/principal you
use (mapred.jobhistory.{principal,keytab}).

HTH,
Erik

On Wed, Jul 19, 2017 at 11:34 PM, Kevin Buckley <
kevin.buckley.ecs.vuw.ac.nz@gmail.com> wrote:

> My Hadoop 2.8.0's
>
> /mr-history/done
>
> directory is owned by the mapred user, who is in the hadoop group,
> and the directory has the pemissions
>
> /mr-history":mapred:hadoop:drwxrwx---
>
> If I run the Hadoop instance without any Kerberos config, and
> fire up the JobHistory server as the mapred user, everything
> works.
>
> If I flip over to a Kerberised environment, the NameNode and DataNodes,
> running as the 'hdfs' user, and the Resource and and Node Managers, running
> as the 'yarn' user, all start up OK and their respective web exposure can
> be
> used.
>
>
> When I try to start up the JobHistory server however
>
> /bin/su mapred -c
> '/local/Hadoop/hadoop-2.8.0/sbin/mr-jobhistory-daemon.sh --config
> /local/Hadoop/hadoop-2.8.0/etc/hadoop/ start historyserver
>
> I get a message in the logs telling me that, rather than the mapred
> user doing things,
> a user 'jhs' is trying to do stuff, vis
>
> 2017-07-20 18:15:09,667 INFO
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: registered UNIX
> signal handlers for [TERM, HUP, INT]
> 2017-07-20 18:15:10,062 INFO
> org.apache.hadoop.security.UserGroupInformation: Login successful for
> user jhs/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ using keytab file
> /local/Hadoop/krb/jhs.service.keytab
> 2017-07-20 18:15:10,107 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2017-07-20 18:15:10,142 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric
> snapshot period at 10 second(s).
> 2017-07-20 18:15:10,142 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobHistoryServer
> metrics system started
> 2017-07-20 18:15:10,145 INFO
> org.apache.hadoop.mapreduce.v2.hs.JobHistory: JobHistory Init
> 2017-07-20 18:15:10,411 INFO
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default
> file system [hdfs://co246a-a.ecs.vuw.ac.nz:9000]
> 2017-07-20 18:15:10,518 INFO
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager failed in state
> INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> Error creating done directory:
> [hdfs://co246a-a.ecs.vuw.ac.nz:9000/mr-history/done]
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating
> done directory: [hdfs://co246a-a.ecs.vuw.ac.nz:9000/mr-history/done]
>         at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.
> tryCreatingHistoryDirs(HistoryFileManager.java:639)
>         at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.
> createHistoryDirs(HistoryFileManager.java:585)
>         at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.
> serviceInit(HistoryFileManager.java:550)
>         at org.apache.hadoop.service.AbstractService.init(
> AbstractService.java:163)
>         at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(
> JobHistory.java:95)
>         at org.apache.hadoop.service.AbstractService.init(
> AbstractService.java:163)
>         at org.apache.hadoop.service.CompositeService.serviceInit(
> CompositeService.java:107)
>         at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.
> serviceInit(JobHistoryServer.java:151)
>         at org.apache.hadoop.service.AbstractService.init(
> AbstractService.java:163)
>         at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.
> launchJobHistoryServer(JobHistoryServer.java:231)
>         at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(
> JobHistoryServer.java:241)
> Caused by: org.apache.hadoop.security.AccessControlException:
> Permission denied: user=jhs, access=EXECUTE,
> inode="/mr-history":mapred:hadoop:drwxrwx---
>
>
> But where has the jhs user come from ?
>
> Doesn't appear to be set anywhere in any of the config files.
>
> According to the hadoop-2.8.0  docs SecureMode page,
>
>    https://hadoop.apache.org/docs/r2.8.0/hadoop-project-
> dist/hadoop-common/SecureMode.html
>
> =============================================
> MapReduce JobHistory Server
>
> The MapReduce JobHistory Server keytab file, on that host, should look
> like the following:
>
> $ klist -e -k -t /etc/security/keytab/jhs.service.keytab
> Keytab name: FILE:/etc/security/keytab/jhs.service.keytab
> KVNO Timestamp         Principal
>    4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD
> (AES-256 CTS mode with 96-bit SHA-1 HMAC)
>    4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD
> (AES-128 CTS mode with 96-bit SHA-1 HMAC)
>    4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD
> (ArcFour with HMAC/md5)
>    4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD
> (AES-256 CTS mode with 96-bit SHA-1 HMAC)
>    4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD
> (AES-128 CTS mode with 96-bit SHA-1 HMAC)
>    4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD
> (ArcFour with HMAC/md5)
> =============================================
>
>
> and mine does.
>
> The hadoop-2.8.0  docs SecureMode page also suggests that one would need to
> play around with the
>
> hadoop.security.auth_to_local
>
> config value, but I haven't had to do that for the nn, dn, rm or nm
> keytabs.
>
> So is there something special about the jhs user ?
>
> Or perhaps something special about the other keytab values ?
>
> Any clues/insight welcome,
> Kevin
>
> ---
> Kevin M. Buckley
>
> eScience Consultant
> School of Engineering and Computer Science
> Victoria University of Wellington
> New Zealand
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>

Mime
View raw message