Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 345E4113FB for ; Tue, 8 Apr 2014 17:14:17 +0000 (UTC) Received: (qmail 10463 invoked by uid 500); 8 Apr 2014 17:12:49 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 9643 invoked by uid 500); 8 Apr 2014 17:12:08 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 8447 invoked by uid 99); 8 Apr 2014 17:11:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Apr 2014 17:11:19 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of devaraj.kavali@intel.com designates 134.134.136.24 as permitted sender) Received: from [134.134.136.24] (HELO mga09.intel.com) (134.134.136.24) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Apr 2014 17:11:14 +0000 Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP; 08 Apr 2014 08:34:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.97,818,1389772800"; d="scan'208,217";a="489321565" Received: from fmsmsx104.amr.corp.intel.com ([10.19.9.35]) by orsmga001.jf.intel.com with ESMTP; 08 Apr 2014 08:39:29 -0700 Received: from fmsmsx112.amr.corp.intel.com (10.18.116.6) by FMSMSX104.amr.corp.intel.com (10.19.9.35) with Microsoft SMTP Server (TLS) id 14.3.123.3; Tue, 8 Apr 2014 08:39:22 -0700 Received: from bgsmsx103.gar.corp.intel.com (10.223.4.130) by FMSMSX112.amr.corp.intel.com (10.18.116.6) with Microsoft SMTP Server (TLS) id 14.3.123.3; Tue, 8 Apr 2014 08:39:21 -0700 Received: from bgsmsx101.gar.corp.intel.com ([169.254.1.132]) by BGSMSX103.gar.corp.intel.com ([169.254.4.75]) with mapi id 14.03.0123.003; Tue, 8 Apr 2014 21:09:19 +0530 From: "Kavali, Devaraj" To: "user@hadoop.apache.org" Subject: RE: Running MRV1 code on YARN Thread-Topic: Running MRV1 code on YARN Thread-Index: AQHPUyNYyUX6zjhAgUamrEHzNduNWZsHqYpQ//+9ggCAAA1sAIAAZccw Date: Tue, 8 Apr 2014 15:39:19 +0000 Message-ID: <03B8E515E27DB74CA4A22A2D8053514577D000@BGSMSX101.gar.corp.intel.com> References: <03B8E515E27DB74CA4A22A2D8053514577C843@BGSMSX101.gar.corp.intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.223.10.10] Content-Type: multipart/alternative; boundary="_000_03B8E515E27DB74CA4A22A2D8053514577D000BGSMSX101garcorpi_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_03B8E515E27DB74CA4A22A2D8053514577D000BGSMSX101garcorpi_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable If you have set fs.defaultFS configuration with hdfs, it should use the HDF= S. And also please make sure that you have updated the Hadoop and the dependen= cy jar files in the client side with the Hadoop 2.2.0 jars. Thanks Devaraj K From: divye sheth [mailto:divs.sheth@gmail.com] Sent: Tuesday, April 08, 2014 8:32 PM To: user@hadoop.apache.org Subject: Re: Running MRV1 code on YARN Hi Deveraj, I went through multiple links all asking me to check if the mapreduce.frame= work.name is set to yarn, it is along with= proper pointing to the Namenode i.e. fs.defaultFS. But still it tries to connect to the local. I am not sure what to do, pleas= e help me out with some pointers as I am fairly new to the coding aspect of= map-reduce. Thanks Divye Sheth On Tue, Apr 8, 2014 at 7:43 PM, divye sheth > wrote: Hi, I saw that pretty much after sending the email. I verified the properties f= ile and it has all the correct properties even the mapred.framework.name is set to yarn. I am unable to figure out what = is the cause and why it is connecting to local FS. Using the same configuration file I am able to run my WordCount MRV1 exampl= e but not the code that I have written for a usecase. Thanks Divye Sheth On Tue, Apr 8, 2014 at 6:12 PM, Kavali, Devaraj > wrote: As per the given exception stack trace, it is trying to use local file syst= em. Can you check whether you have configured the file system configuration= s with HDFS? Thanks Devaraj K From: divye sheth [mailto:divs.sheth@gmail.com= ] Sent: Tuesday, April 08, 2014 5:37 PM To: user@hadoop.apache.org Subject: Running MRV1 code on YARN Hi, I have installed Hadoop 2.2.0 along with YARN and am trying to submit a MRV= 1 job already written to YARN. The job does not even submit and it prints the following stack trace on con= sole: 2014-04-08 16:56:11 UserGroupInformation [ERROR] PriviledgedActionException= as:eureka (auth:SIMPLE) cause:org.apache.hadoop.util.Shell$ExitCodeExcepti= on: chmod: cannot access `/user/eureka54695942/.staging/job_local54695942_0= 001': No such file or directory Exception in thread "main" org.apache.hadoop.util.Shell$ExitCodeException: = chmod: cannot access `/user/eureka54695942/.staging/job_local54695942_0001'= : No such file or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at org.apache.hadoop.util.Shell.run(Shell.java:188) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.= java:381) at org.apache.hadoop.util.Shell.execCommand(Shell.java:467) at org.apache.hadoop.util.Shell.execCommand(Shell.java:450) at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFile= System.java:593) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFi= leSystem.java:584) at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSy= stem.java:427) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:579) at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClie= nt.java:786) at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClie= nt.java:746) at org.apache.hadoop.mapred.JobClient.access$400(JobClient.java:177= ) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:963) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:948) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupIn= formation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.j= ava:948) at org.apache.hadoop.mapreduce.Job.submit(Job.java:566) My question here is if you notice the staging location which it is trying t= o clean I do not have any such user in the /user directory in hdfs. It some= how appends the jobId to the username and creates staging area there. Any r= eason for this? Please let me know what am I doing wrong. How can I make su= re it goes to the user that I have created i.e. eureka and not eureka$JOBID= . I am using CDH4. Thanks Divye Sheth --_000_03B8E515E27DB74CA4A22A2D8053514577D000BGSMSX101garcorpi_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

If you have set fs.defaul= tFS configuration with hdfs, it should use the HDFS.

 <= /p>

And also please make sure= that you have updated the Hadoop and the dependency jar files in the clien= t side with the Hadoop 2.2.0 jars.

 <= /p>

 <= /p>

From: divye sh= eth [mailto:divs.sheth@gmail.com]
Sent: Tuesday, April 08, 2014 8:32 PM
To: user@hadoop.apache.org
Subject: Re: Running MRV1 code on YARN

 

Hi Deveraj,

 

I went through multiple links all asking me to check= if the mapreduce.framework.name is set to yarn, it is along with proper pointi= ng to the Namenode i.e. fs.defaultFS.

 

But still it tries to connect to the local. I am not= sure what to do, please help me out with some pointers as I am fairly new = to the coding aspect of map-reduce.

 

Thanks

Divye Sheth

 

On Tue, Apr 8, 2014 at 7:43 PM, divye sheth <divs.sheth@gmail.com<= /a>> wrote:

Hi,

 

 

Using the same configuration file I am able to run m= y WordCount MRV1 example but not the code that I have written for a usecase= .

 

Thanks

Divye Sheth=

 

On Tue, Apr 8, 2014 at 6:12 PM, Kavali, Devaraj <= devaraj.kaval= i@intel.com> wrote:

As per the given exception stack trace,= it is trying to use local file system. Can you check whether you have configured the file system configurations with HDFS?<= /o:p>

 

Thanks

Devaraj K

 

From: divye sheth [mailto:divs.sheth@gmail.co= m]
Sent: Tuesday, April 08, 2014 5:37 PM
To: user= @hadoop.apache.org
Subject: Running MRV1 code on YARN

 

Hi,

 

I have installed Hadoop 2.2.0 along with YARN and am trying to sub= mit a MRV1 job already written to YARN.

 

The job does not even submit and it prints the following stack tra= ce on console:

 

2014-04-08 16:56:11 UserGroupInformation [ERROR] PriviledgedAction= Exception as:eureka (auth:SIMPLE) cause:org.apache.hadoop.util.Shell$ExitCo= deException: chmod: cannot access `/user/eureka54695942/.staging/job_local5= 4695942_0001': No such file or directory

 

Exception in thread "main" org.apache.hadoop.util.Shell$= ExitCodeException: chmod: cannot access `/user/eureka54695942/.staging/job_= local54695942_0001': No such file or directory

 

        at org.apache.hadoop.util.Shell.runCom= mand(Shell.java:261)

        at org.apache.hadoop.util.Shell.run(Sh= ell.java:188)

        at org.apache.hadoop.util.Shell$ShellC= ommandExecutor.execute(Shell.java:381)

        at org.apache.hadoop.util.Shell.execCo= mmand(Shell.java:467)

        at org.apache.hadoop.util.Shell.execCo= mmand(Shell.java:450)

        at org.apache.hadoop.fs.RawLocalFileSy= stem.execCommand(RawLocalFileSystem.java:593)

        at org.apache.hadoop.fs.RawLocalFileSy= stem.setPermission(RawLocalFileSystem.java:584)

        at org.apache.hadoop.fs.FilterFileSyst= em.setPermission(FilterFileSystem.java:427)

        at org.apache.hadoop.fs.FileSystem.mkd= irs(FileSystem.java:579)

        at org.apache.hadoop.mapred.JobClient.= copyAndConfigureFiles(JobClient.java:786)

        at org.apache.hadoop.mapred.JobClient.= copyAndConfigureFiles(JobClient.java:746)

        at org.apache.hadoop.mapred.JobClient.= access$400(JobClient.java:177)

        at org.apache.hadoop.mapred.JobClient$= 2.run(JobClient.java:963)

        at org.apache.hadoop.mapred.JobClient$= 2.run(JobClient.java:948)

        at java.security.AccessController.doPr= ivileged(Native Method)

        at javax.security.auth.Subject.doAs(Su= bject.java:396)

        at org.apache.hadoop.security.UserGrou= pInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.JobClient.= submitJobInternal(JobClient.java:948)

        at org.apache.hadoop.mapreduce.Job.sub= mit(Job.java:566)

 

My question here is if you notice the staging location which it is= trying to clean I do not have any such user in the /user directory in hdfs= . It somehow appends the jobId to the username and creates staging area there. Any reason for this? Please let m= e know what am I doing wrong. How can I make sure it goes to the user that = I have created i.e. eureka and not eureka$JOBID.

 

I am using CDH4.

 

Thanks

Divye Sheth

 

 

--_000_03B8E515E27DB74CA4A22A2D8053514577D000BGSMSX101garcorpi_--