Return-Path: X-Original-To: apmail-crunch-user-archive@www.apache.org Delivered-To: apmail-crunch-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B6C571008F for ; Tue, 3 Sep 2013 13:41:11 +0000 (UTC) Received: (qmail 7568 invoked by uid 500); 3 Sep 2013 13:41:11 -0000 Delivered-To: apmail-crunch-user-archive@crunch.apache.org Received: (qmail 7161 invoked by uid 500); 3 Sep 2013 13:41:06 -0000 Mailing-List: contact user-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@crunch.apache.org Delivered-To: mailing list user@crunch.apache.org Received: (qmail 6804 invoked by uid 99); 3 Sep 2013 13:41:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Sep 2013 13:41:04 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hpnole@gmail.com designates 209.85.192.181 as permitted sender) Received: from [209.85.192.181] (HELO mail-pd0-f181.google.com) (209.85.192.181) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Sep 2013 13:41:00 +0000 Received: by mail-pd0-f181.google.com with SMTP id g10so5990766pdj.26 for ; Tue, 03 Sep 2013 06:40:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Cy81OYx+187pcgdM+XEJLfQi6azoYGM2GjhETVJoJ0E=; b=zUVJ++Amol0Xh85ZMpnMKe+RyKyRY5kWG5iLiNFEyKCvT1UlryFC6sZs87Qphz79H9 07S9l8OhSMEfHH5EKBqpvrZYoH2pWSFvhmo4nr/xHj8v1BQf3ZZfA82bUBE+bknzueQy rVC2s2HuM+hgsM3AU3CfUUPtWQWUza3vAGUjtj/U348M3Tfa7/pXlXQ4VsQNQ4qTzqgc v7arGyYpz6sI6T2o/7GIn/3MKkuDAQyQkCpBd427hTuijsu3RcGXNPZ5NvApLw4IqHjb WKfWW4dbD1yQIisBbuJsxgem+UnL2ogXnFsxnK5pf8CoaSZo+SeT1SoFej6sk1MFbAHo AsPQ== MIME-Version: 1.0 X-Received: by 10.66.88.3 with SMTP id bc3mr8174650pab.48.1378215639471; Tue, 03 Sep 2013 06:40:39 -0700 (PDT) Received: by 10.68.184.197 with HTTP; Tue, 3 Sep 2013 06:40:39 -0700 (PDT) In-Reply-To: References: <138801cea5f5$139e6000$3adb2000$@gmail.com> Date: Tue, 3 Sep 2013 08:40:39 -0500 Message-ID: Subject: Re: UnknownHostException while submitting job to remote cluster From: Narlin M To: Crunch users Content-Type: multipart/alternative; boundary=f46d042f9cfc9d5c0b04e57ad5ec X-Virus-Checked: Checked by ClamAV on apache.org --f46d042f9cfc9d5c0b04e57ad5ec Content-Type: text/plain; charset=ISO-8859-1 Hello Micah, thanks for replying. I am not sure but I am probably targeting YARN, because I got this warning to use fs.defaultFS instead of fs.default.name when I ran my application. But I will try to confirm this. On Sat, Aug 31, 2013 at 11:48 AM, Micah Whitacre wrote: > It sounds like you are reading configuration files that are setup for HDFS > HA. This done by the HBaseConfiguration.create() reading files such as > hbase-site.xml, core-site.xml, or hdfs-site.xml. > > Are you targeting YARN or MRv1? If MRv1, according to the > documentation[1] you should be setting "fs.default.name" instead. > > [1] - > http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-High-Availability-Guide/cdh4hag_topic_2_3.html > > > On Fri, Aug 30, 2013 at 9:52 PM, Narlin M wrote: > >> Hi,**** >> >> ** ** >> >> I am getting following exception while trying to submit a crunch job to a >> remote hadoop cluster:**** >> >> ** ** >> >> 2880 [Thread-15] INFO >> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob - >> java.lang.IllegalArgumentException: java.net.UnknownHostException: bdatadev >> **** >> >> at >> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414) >> **** >> >> at >> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164) >> **** >> >> at >> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129) >> **** >> >> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:389) >> **** >> >> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:356) >> **** >> >> at >> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:124) >> **** >> >> at >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)*** >> * >> >> at >> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)**** >> >> at >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)** >> ** >> >> at >> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)**** >> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)** >> ** >> >> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)**** >> >> at >> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103) >> **** >> >> at >> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)**** >> >> at >> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)**** >> >> at java.security.AccessController.doPrivileged(Native Method) >> **** >> >> at javax.security.auth.Subject.doAs(Subject.java:396)**** >> >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) >> **** >> >> at >> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)* >> *** >> >> at org.apache.hadoop.mapreduce.Job.submit(Job.java:531)**** >> >> at >> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:305) >> **** >> >> at >> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJobs(CrunchJobControl.java:180) >> **** >> >> at >> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:209) >> **** >> >> at >> org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:100) >> **** >> >> at >> org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:51)* >> *** >> >> at >> org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:75)**** >> >> at java.lang.Thread.run(Thread.java:680)**** >> >> Caused by: java.net.UnknownHostException: bdatadev**** >> >> ... 27 more**** >> >> ** ** >> >> However nowhere in my code a host named "bdatadev" is mentioned and also, >> I cannot ping this host.**** >> >> ** ** >> >> The section of the code where I am setting up the MRPipeline is as >> follows:**** >> >> ** ** >> >> Configuration conf = HBaseConfiguration.create();**** >> >> ** ** >> >> conf.set("fs.defaultFS", "hdfs://:8020");**** >> >> conf.set("mapred.job.tracker", ":8021");**** >> >> ** ** >> >> System.out.println("Hadoop configuration created.");**** >> >> System.out.println("Initializing crunch pipeline ...");**** >> >> ** ** >> >> conf.set("mapred.jar", "");**** >> >> ** ** >> >> pipeline = new MRPipeline(getClass(), "crunchjobtest", conf);**** >> >> ** ** >> >> Has anyone faced this issue before and knows how to resolve it/point out >> if I am missing anything?**** >> >> ** ** >> >> Thanks,**** >> >> Narlin.**** >> > > --f46d042f9cfc9d5c0b04e57ad5ec Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hello Micah, thanks for replying.

I am not sure but I am probably targeting YARN, because I got this warning = to use fs.defaultFS instead of fs.defaul= t.name when I ran my application. But I will try to confirm this.=A0


On Sat,= Aug 31, 2013 at 11:48 AM, Micah Whitacre <mkwhitacre@gmail.com>= wrote:
It sounds like you are read= ing configuration files that are setup for HDFS HA. =A0This done by the HBa= seConfiguration.create() reading files such as hbase-site.xml, core-site.xm= l, or hdfs-site.xml. =A0

Are you targeting YARN or MRv1? =A0If MRv1, according to the documenta= tion[1] you should be setting "fs.default.name" instead.


On Fri, Aug 30, 2013 at 9:52 PM, Narlin M = <hpnole@gmail.com> wrote:

Hi,

=A0

I am getting following exception while tr= ying to submit a crunch job to a remote hadoop cluster:

=A0

2880 [Thread-15] INFO =A0org.apache.crunch.hadoop.mapr= educe.lib.jobcontrol.CrunchControlledJob =A0- java.lang.IllegalArgumentExce= ption: java.net.UnknownHostException: bdatadev

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414= )

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:16= 4)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)=

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:389)=

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.DFSC= lient.<init>(DFSClient.java:356)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoo= p.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:124)<= /u>

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.fs.FileSy= stem.access$200(FileSystem.java:80)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.f= s.FileSystem$Cache.getInternal(FileSystem.java:2252)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.fs.FileSy= stem.get(FileSystem.java:300)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.fs.Path= .getFileSystem(Path.java:194)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles= .java:103)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.Jo= bClient$2.run(JobClient.java:896)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java.security.AccessC= ontroller.doPrivileged(Native Method)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java= x.security.auth.Subject.doAs(Subject.java:396)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apac= he.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332= )

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)=

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapreduce= .Job.submit(Job.java:531)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.crunch.hadoop.mapr= educe.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:30= 5)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJo= bs(CrunchJobControl.java:180)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatu= sAndStartNewOnes(CrunchJobControl.java:209)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.= apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:100)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apache.crunch.impl.mr.e= xec.MRExecutor.access$000(MRExecutor.java:51)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at org.apach= e.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:75)

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at java= .lang.Thread.run(Thread.java:680)

Caused by: java.net.UnknownHostException: bdatadev<= u>

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ... 27 = more

=A0

However nowhere in my code a host named &= quot;bdatadev" is mentioned and also, I cannot ping this host.<= u>

=A0

The section of the code where I am setting up the MRPi= peline is as follows:

=A0

Configuration conf =3D HBaseConfiguration.create();

=A0

conf.set(&quo= t;fs.defaultFS", "hdfs://<server_address>:8020");

conf.set("mapred.job.tracker", "<server_= address>:8021");

= =A0

System.out.println("Hadoop configuration created.&quo= t;);

System.out.println("Init= ializing crunch pipeline ...");

=A0

conf.set(&quo= t;mapred.jar", "<path_to_jar_file>");

=A0

pipeline =3D = new MRPipeline(getClass(), "crunchjobtest", conf);<= /span>

=A0

Has anyone faced this issue before and knows how to re= solve it/point out if I am missing anything?

=A0

Thanks,

Narlin.



--f46d042f9cfc9d5c0b04e57ad5ec--