crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Whitacre <mkwhita...@gmail.com>
Subject Re: UnknownHostException while submitting job to remote cluster
Date Sat, 31 Aug 2013 16:48:27 GMT
It sounds like you are reading configuration files that are setup for HDFS
HA.  This done by the HBaseConfiguration.create() reading files such as
hbase-site.xml, core-site.xml, or hdfs-site.xml.

Are you targeting YARN or MRv1?  If MRv1, according to the documentation[1]
you should be setting "fs.default.name" instead.

[1] -
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-High-Availability-Guide/cdh4hag_topic_2_3.html


On Fri, Aug 30, 2013 at 9:52 PM, Narlin M <hpnole@gmail.com> wrote:

> Hi,****
>
> ** **
>
> I am getting following exception while trying to submit a crunch job to a
> remote hadoop cluster:****
>
> ** **
>
> 2880 [Thread-15] INFO
>  org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob  -
> java.lang.IllegalArgumentException: java.net.UnknownHostException: bdatadev
> ****
>
>             at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
> ****
>
>             at
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)
> ****
>
>             at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
> ****
>
>             at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:389)
> ****
>
>             at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:356)
> ****
>
>             at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:124)
> ****
>
>             at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)****
>
>             at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)****
>
>             at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)***
> *
>
>             at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)****
>
>             at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)***
> *
>
>             at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)****
>
>             at
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103)
> ****
>
>             at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
> ****
>
>             at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
> ****
>
>             at java.security.AccessController.doPrivileged(Native Method)*
> ***
>
>             at javax.security.auth.Subject.doAs(Subject.java:396)****
>
>             at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> ****
>
>             at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)**
> **
>
>             at org.apache.hadoop.mapreduce.Job.submit(Job.java:531)****
>
>             at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:305)
> ****
>
>             at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJobs(CrunchJobControl.java:180)
> ****
>
>             at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:209)
> ****
>
>             at
> org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:100)
> ****
>
>             at
> org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:51)**
> **
>
>             at
> org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:75)****
>
>             at java.lang.Thread.run(Thread.java:680)****
>
> Caused by: java.net.UnknownHostException: bdatadev****
>
>             ... 27 more****
>
> ** **
>
> However nowhere in my code a host named "bdatadev" is mentioned and also,
> I cannot ping this host.****
>
> ** **
>
> The section of the code where I am setting up the MRPipeline is as follows:
> ****
>
> ** **
>
> Configuration conf = HBaseConfiguration.create();****
>
> ** **
>
> conf.set("fs.defaultFS", "hdfs://<server_address>:8020");****
>
> conf.set("mapred.job.tracker", "<server_address>:8021");****
>
> ** **
>
> System.out.println("Hadoop configuration created.");****
>
> System.out.println("Initializing crunch pipeline ...");****
>
> ** **
>
> conf.set("mapred.jar", "<path_to_jar_file>");****
>
> ** **
>
> pipeline = new MRPipeline(getClass(), "crunchjobtest", conf);****
>
> ** **
>
> Has anyone faced this issue before and knows how to resolve it/point out
> if I am missing anything?****
>
> ** **
>
> Thanks,****
>
> Narlin.****
>

Mime
View raw message