Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5903E11E30 for ; Thu, 12 Jun 2014 23:33:20 +0000 (UTC) Received: (qmail 1628 invoked by uid 500); 12 Jun 2014 23:33:16 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 1527 invoked by uid 500); 12 Jun 2014 23:33:16 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 1520 invoked by uid 99); 12 Jun 2014 23:33:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Jun 2014 23:33:16 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbeaulau@cisco.com designates 173.37.86.79 as permitted sender) Received: from [173.37.86.79] (HELO rcdn-iport-8.cisco.com) (173.37.86.79) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Jun 2014 23:33:11 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=9099; q=dns/txt; s=iport; t=1402615988; x=1403825588; h=from:to:subject:date:message-id:mime-version; bh=JZIisKfo+t18U1LB60EKUyiDgoK61ayOcEUwVmzA+4Y=; b=mK1BRCTqHdNAVKldNO3fM1TCauAK7bjB6atwCgr5Byi4q/13Kt6wJFKM TbZQpskXN6ZzhF44k+hQmWzQTyS+RoWHzV/oUp0uOMG3U6sJIAx3cOtaG TooYDlY+h2tOmmA3Ev4ismr6ej69+f/rtdcdEuISZYY1vcOiky9G2DM2f M=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhYFAEQ4mlOtJV2Q/2dsb2JhbABagkZHUlm/bYMRAYEHFnWEBQEELV4BKkMTHwcBBBuIOqMSrzcXjiqDY4EWBJARnXSDPIIv X-IronPort-AV: E=Sophos;i="5.01,468,1400025600"; d="scan'208,217";a="332759875" Received: from rcdn-core-8.cisco.com ([173.37.93.144]) by rcdn-iport-8.cisco.com with ESMTP; 12 Jun 2014 23:32:24 +0000 Received: from xhc-rcd-x03.cisco.com (xhc-rcd-x03.cisco.com [173.37.183.77]) by rcdn-core-8.cisco.com (8.14.5/8.14.5) with ESMTP id s5CNWO2j024906 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Thu, 12 Jun 2014 23:32:24 GMT Received: from xmb-aln-x14.cisco.com ([169.254.8.214]) by xhc-rcd-x03.cisco.com ([173.37.183.77]) with mapi id 14.03.0123.003; Thu, 12 Jun 2014 18:32:24 -0500 From: "John Beaulaurier -X (jbeaulau - ADVANCED NETWORK INFORMATION INC at Cisco)" To: "user@hadoop.apache.org" Subject: hdfs over http error Thread-Topic: hdfs over http error Thread-Index: Ac+Glo7+Bu3lcfHkSJmGodu5m9jeXQ== Date: Thu, 12 Jun 2014 23:32:23 +0000 Message-ID: <565B75ACD9C9964EA087653C7D7A65B70B2CB487@xmb-aln-x14.cisco.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.154.36.216] Content-Type: multipart/alternative; boundary="_000_565B75ACD9C9964EA087653C7D7A65B70B2CB487xmbalnx14ciscoc_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_565B75ACD9C9964EA087653C7D7A65B70B2CB487xmbalnx14ciscoc_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello, Apache Hadoop 0.20.203.0 A colleague is using a SPARK shell on a remote host using HDFS protocol att= empting to run a job on our Hadoop cluster, but the job errors out before finishing with the fol= lowing noted in the namenode log. 2014-06-11 16:13:24,958 WARN org.apache.hadoop.security.ShellBasedUnixGroup= sMapping: got exception trying to get groups for user webuser org.apache.ha= doop.util.Shell$ExitCodeException: id: webuser: No such user The username submitting the job does not exists on the localhost running as= namenode, but a three layered directory tree is created in HDFS owned by the username submitting the job, but stops at a certain point with the erro= r noted above, and the last directory empty. hadoop fs -ls /existing_dir/sparkcounts3/_temporary/_attempt_201406101047_= 0000_m_000000_5 I found online that the default for web access is user webuser group webgr= oup, neither are configured on the namenode host OS. If I do the following should it resolve the issue. 1. create a local group named webgroup 2. Create local user with the same name as is submitting the job, 3. Add that local user to the group webgroup 4. Add the following property to hdfs-site.xml dfs.web.ugi webgroup Is a restart of dfs, or mapred, or both necessary after the adding the prop= erty? Thanks -John --_000_565B75ACD9C9964EA087653C7D7A65B70B2CB487xmbalnx14ciscoc_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello,

 

Apache Hadoop 0.20.203.0

 

A colleague is using a SPARK shell on a remote ho= st using HDFS protocol attempting to run a job

on our Hadoop cluster, but the job errors out bef= ore finishing with the following noted in the namenode log.

 

2014-06-11 16:13:24,958 WARN org.apache.hadoop.se= curity.ShellBasedUnixGroupsMapping: got exception trying to get groups for = user webuser org.apache.hadoop.util.Shell$ExitCodeException: id: webuser: N= o such user

 

The username submitting the job does not exists o= n the localhost running as namenode, but a three layered directory tree is = created in HDFS owned by

the username submitting the job, but stops at a c= ertain point with the error noted above, and the last directory empty.=

 

hadoop fs –ls  /existing_dir/sparkcoun= ts3/_temporary/_attempt_201406101047_0000_m_000000_5

 

I found online that the default for web access is= user webuser  group webgroup, neither are configured on the namenode = host OS. If I do the following

should it resolve the issue.

 

1.        create a local group named webgroup

2.       Create local user with the same name as is submitti= ng the job,

3.       Add that local user to the group webgroup

4.       Add the following property to hdfs-site.xml

 

<property><= /o:p>

  <name>dfs= .web.ugi</name>

  <value> w= ebgroup</value>

</property>=

 

Is a restart of dfs, or mapred, or both necessary= after the adding the property?

 

Thanks

-John

 

--_000_565B75ACD9C9964EA087653C7D7A65B70B2CB487xmbalnx14ciscoc_--