Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 35D32DC96 for ; Fri, 24 Aug 2012 05:58:19 +0000 (UTC) Received: (qmail 65518 invoked by uid 500); 24 Aug 2012 05:58:13 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 65443 invoked by uid 500); 24 Aug 2012 05:58:13 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 65420 invoked by uid 99); 24 Aug 2012 05:58:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Aug 2012 05:58:13 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dechouxb@gmail.com designates 209.85.216.176 as permitted sender) Received: from [209.85.216.176] (HELO mail-qc0-f176.google.com) (209.85.216.176) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Aug 2012 05:58:06 +0000 Received: by qcsc21 with SMTP id c21so1189009qcs.35 for ; Thu, 23 Aug 2012 22:57:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=0cFby1UeBv1PKbnXW7QIc7bjLVt/pqRlfnp+7GHn8uY=; b=OKS5Xg5mpaoijcq5nROZHKD0tMHGctvqsj0nlXtdG8R6col6uB8zPS3/Xy0Iz1HjKg qg+aKaaCYEP5HtP9DyjlQm9FkcSmMM5FK3edLqGm3YnwrnwvMW5nMrECEZGB5bml3vQS fPjViLyInVvipb+KOQjC8sy93ax5l/+GMQRRMLlKoo05n90VUW1/5u7H/dBENQJhCOB1 lPz4Kbb3RcN5bnAktcPt2DeXHJbTQWdGeR8YHJiReZr3/nvBUrHJej+sBRtJyG2EmR3B 3fVFWPTJJis55zR5t4mGXajI1DRWsr+KafkhFa9YxNH2TOtMW3LUcj6vI8DvErjPq+WR y1Ag== MIME-Version: 1.0 Received: by 10.224.188.76 with SMTP id cz12mr7079761qab.12.1345787865627; Thu, 23 Aug 2012 22:57:45 -0700 (PDT) Received: by 10.49.76.10 with HTTP; Thu, 23 Aug 2012 22:57:45 -0700 (PDT) In-Reply-To: References: Date: Fri, 24 Aug 2012 07:57:45 +0200 Message-ID: Subject: Re: Running map reduce job from a web application From: Bertrand Dechoux To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf303640cfac7dd604c7fca79a --20cf303640cfac7dd604c7fca79a Content-Type: text/plain; charset=ISO-8859-1 I haven't been confronted to your use case but your error message explicitly says that you are not passing correctly the user "hadoop". You should use the doAs() method on UserGroupInformation in order to execute the job with this particular user. http://mail-archives.apache.org/mod_mbox/hadoop-general/201009.mbox/%3CAANLkTiknJ_SzRux7KhjhxVfUU9FBkNgvYnkpbz3G_+a4@mail.gmail.com%3E Regards Bertrand On Fri, Aug 24, 2012 at 6:01 AM, Chandra Mohan, Ananda Vel Murugan < Ananda.Murugan@honeywell.com> wrote: > Hi, **** > > ** ** > > I am developing a web application in Java which would be hosted in a > machine outside of my Hadoop cluster. User account configuration would be > different between the machines. **** > > ** ** > > I am using the following code in my servlet to run the map-reduce job.**** > > ** ** > > ** ** > > *final* Configuration config = *new* Configuration();**** > > config.set("mapred.job.tracker", ":9001" > );**** > > config.set("fs.default.name", "hdfs://:9000");**** > > config.set("hadoop.tmp.dir", > "/var/lib/hadoop/data-store/hadoop-hadoop");**** > > config.set("hadoop.job.ugi", "hadoop,hadoop");**** > > UserGroupInformation *ugi1* = UserGroupInformation.* > createRemoteUser*("hadoop");**** > > Job job = *new* Job(config);**** > > job.setJarByClass(ImportTsv.*class*);**** > > job.setJobName("ImportTsv");**** > > job.getConfiguration().set("user", "hadoop");**** > > job.waitForCompletion(*true*);**** > > job.submit();**** > > *Iterator* iter = config.iterator();**** > > *while*(iter.hasNext())**** > > {**** > > Object obj = iter.next();**** > > System.*out*.println(obj);**** > > }**** > > ** ** > > It is not working and failing with following authentication error**** > > ** ** > > Exception in thread "main" * > org.apache.hadoop.security.AccessControlException*: * > org.apache.hadoop.security.AccessControlException*: Permission denied: > user=E317376, access=WRITE, inode="staging":hadoop:supergroup:rwxr-xr-x*** > * > > ** ** > > Appreciate your inputs for fixing this.**** > > ** ** > > Regards,**** > > Anand.C **** > > ** ** > > ** ** > -- Bertrand Dechoux --20cf303640cfac7dd604c7fca79a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I haven't been confronted to your use case but your error message expli= citly says that you are not passing correctly the user "hadoop".<= div>You should use the doAs() method on=A0UserGroupInformation in order to exe= cute the job with this particular user.


Regards

Bertrand

On Fri, Aug 24, 2012 at 6:01 AM, Chandra Mohan, Ananda= Vel Murugan <Ananda.Murugan@honeywell.com> wrote= :

Hi,

=A0<= /font>

I am developing a web app= lication in Java which would be hosted in a machine outside of my Hadoop cl= uster. User account configuration would be different between the machines.

=A0<= /font>

I am using the following = code in my servlet to run the map-reduce job.

=A0<= /font>

=A0<= /font>

fi= nal Configuration config =3D new Configuration();

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 config.set(<= font color=3D"#2a00ff" face=3D"Courier New">"mapred.job.tracke= r", "&l= t;jobtracker.hostname>:9001");

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 config.set(<= font color=3D"#2a00ff" face=3D"Courier New">"fs.default.name"= , "hd= fs://<namenode>:9000");

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 config.set(<= font color=3D"#2a00ff" face=3D"Courier New">"hadoop.tmp.dir&qu= ot;, "/v= ar/lib/hadoop/data-store/hadoop-hadoop");

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 config.set(<= font color=3D"#2a00ff" face=3D"Courier New">"hadoop.job.ugi&qu= ot;, "ha= doop,hadoop"<= span style=3D"font-size:10.0pt;font-family:"Courier New"">);

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 UserGroupInformation ugi1 =3D UserGroupInformation.c= reateRemoteUser("hadoop");

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Job job =3D new Job(config);

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 job.setJarByClass(ImportTs= v.class)= ;

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 job.setJobName("ImportTsv&quo= t;);

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 job.getConfiguration().set= ("u= ser", "ha= doop");

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 job.waitForCompletion(true);

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 job.submit();

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Iterator iter =3D config.iterator();

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 while(= iter.hasNext())

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 {

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Object o= bj =3D iter.next();=

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 System.<= /span>out.prin= tln(obj);

=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 }

=A0

It is not working and fai= ling with following authentication error

=A0

Excepti= on in thread "main" org.apache.= hadoop.security.AccessControlException: org.apache.= hadoop.security.AccessControlException: Permission denied: user=3DE317376, access=3DWRITE, inode=3D"staging&q= uot;:hadoop:supergroup:rwxr-xr-x

= =A0

Appreciate your inputs fo= r fixing this.

=A0<= /font>

Regards,

Anand.C

=A0<= /font>

=A0<= /font>




--
Bertrand Dec= houx
--20cf303640cfac7dd604c7fca79a--