kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hongbin ma <mahong...@apache.org>
Subject upgrading kylin's hadoop depedencies version
Date Wed, 14 Jan 2015 12:20:28 GMT
hi, yang

I tried our deploy script on hdp 2.2 (with current kylin dependencies in
pom untouched), it turns out that MR job submission will fail. the log is
attached at the end.
I found that "mapreduce.application.framework.path" is something new in hdp
2.2's mapred-site.xml. I guess the obsolete MR client might not be able to
handle it properly.

I suggest we upgrade the hadoop dependencies versions in our pom, and see
if it will work out in both hdp 2.1 and 2.2.
I remember you tried upgrading the versions recently, Did you checked them
in?

hongbin.

=================================

[QuartzScheduler_Worker-1]:[2015-01-14
09:24:45,832][ERROR][com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:83)]
- Unable to parse
'/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
URI, check the setting for mapreduce.application.framework.path

java.lang.IllegalArgumentException: Unable to parse
'/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
URI, check the setting for mapreduce.application.framework.path

at
org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:739)

at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:435)

at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)

at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)

at
com.kylinolap.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:114)

at
com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:79)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)

at com.kylinolap.job.cmd.JavaHadoopCmd.execute(JavaHadoopCmd.java:53)

at com.kylinolap.job.flow.AsyncJobFlowNode.execute(AsyncJobFlowNode.java:70)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Caused by: java.net.URISyntaxException: Illegal character in path at index
11: /hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework

at java.net.URI$Parser.fail(URI.java:2829)

at java.net.URI$Parser.checkChars(URI.java:3002)

at java.net.URI$Parser.parseHierarchical(URI.java:3086)

at java.net.URI$Parser.parse(URI.java:3044)

at java.net.URI.<init>(URI.java:595)

at
org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:737)

... 15 more

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message