crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Narlin M" <hpn...@gmail.com>
Subject UnknownHostException while submitting job to remote cluster
Date Sat, 31 Aug 2013 02:52:04 GMT
Hi,

 

I am getting following exception while trying to submit a crunch job to a
remote hadoop cluster:

 

2880 [Thread-15] INFO
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob  -
java.lang.IllegalArgumentException: java.net.UnknownHostException: bdatadev

            at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:
414)

            at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java
:164)

            at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)

            at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:389)

            at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:356)

            at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSyste
m.java:124)

            at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)

            at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)

            at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)

            at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)

            at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)

            at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)

            at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFi
les.java:103)

            at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)

            at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)

            at java.security.AccessController.doPrivileged(Native Method)

            at javax.security.auth.Subject.doAs(Subject.java:396)

            at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1332)

            at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)

            at org.apache.hadoop.mapreduce.Job.submit(Job.java:531)

            at
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit
(CrunchControlledJob.java:305)

            at
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startRead
yJobs(CrunchJobControl.java:180)

            at
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobSt
atusAndStartNewOnes(CrunchJobControl.java:209)

            at
org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:100)

            at
org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:51)

            at
org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:75)

            at java.lang.Thread.run(Thread.java:680)

Caused by: java.net.UnknownHostException: bdatadev

            ... 27 more

 

However nowhere in my code a host named "bdatadev" is mentioned and also, I
cannot ping this host.

 

The section of the code where I am setting up the MRPipeline is as follows:

 

Configuration conf = HBaseConfiguration.create();

 

conf.set("fs.defaultFS", "hdfs://<server_address>:8020");

conf.set("mapred.job.tracker", "<server_address>:8021");

 

System.out.println("Hadoop configuration created.");

System.out.println("Initializing crunch pipeline ...");

 

conf.set("mapred.jar", "<path_to_jar_file>");

 

pipeline = new MRPipeline(getClass(), "crunchjobtest", conf);

 

Has anyone faced this issue before and knows how to resolve it/point out if
I am missing anything?

 

Thanks,

Narlin.


Mime
View raw message