hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Beaulaurier -X (jbeaulau - ADVANCED NETWORK INFORMATION INC at Cisco)" <jbeau...@cisco.com>
Subject hdfs over http error
Date Thu, 12 Jun 2014 23:32:23 GMT

Apache Hadoop

A colleague is using a SPARK shell on a remote host using HDFS protocol attempting to run
a job

on our Hadoop cluster, but the job errors out before finishing with the following noted in
the namenode log.

2014-06-11 16:13:24,958 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception
trying to get groups for user webuser org.apache.hadoop.util.Shell$ExitCodeException: id:
webuser: No such user

The username submitting the job does not exists on the localhost running as namenode, but
a three layered directory tree is created in HDFS owned by

the username submitting the job, but stops at a certain point with the error noted above,
and the last directory empty.

hadoop fs -ls  /existing_dir/sparkcounts3/_temporary/_attempt_201406101047_0000_m_000000_5

I found online that the default for web access is user webuser  group webgroup, neither are
configured on the namenode host OS. If I do the following

should it resolve the issue.

1.        create a local group named webgroup

2.       Create local user with the same name as is submitting the job,

3.       Add that local user to the group webgroup

4.       Add the following property to hdfs-site.xml



  <value> webgroup</value>


Is a restart of dfs, or mapred, or both necessary after the adding the property?



View raw message