kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luke Han <luke...@apache.org>
Subject Re: upgrading kylin's hadoop depedencies version
Date Wed, 14 Jan 2015 13:58:32 GMT
Hi Hongbin, please send mail direct to dev@ list, not cc :-)

Thanks.

Luke

2015-01-14 20:20 GMT+08:00 hongbin ma <mahongbin@apache.org>:

> hi, yang
>
> I tried our deploy script on hdp 2.2 (with current kylin dependencies in
> pom untouched), it turns out that MR job submission will fail. the log is
> attached at the end.
> I found that "mapreduce.application.framework.path" is something new in hdp
> 2.2's mapred-site.xml. I guess the obsolete MR client might not be able to
> handle it properly.
>
> I suggest we upgrade the hadoop dependencies versions in our pom, and see
> if it will work out in both hdp 2.1 and 2.2.
> I remember you tried upgrading the versions recently, Did you checked them
> in?
>
> hongbin.
>
> =================================
>
> [QuartzScheduler_Worker-1]:[2015-01-14
>
> 09:24:45,832][ERROR][com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:83)]
> - Unable to parse
> '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
> URI, check the setting for mapreduce.application.framework.path
>
> java.lang.IllegalArgumentException: Unable to parse
> '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
> URI, check the setting for mapreduce.application.framework.path
>
> at
>
> org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:739)
>
> at
>
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:435)
>
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
>
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
>
> at
>
> com.kylinolap.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:114)
>
> at
>
> com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:79)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>
> at com.kylinolap.job.cmd.JavaHadoopCmd.execute(JavaHadoopCmd.java:53)
>
> at
> com.kylinolap.job.flow.AsyncJobFlowNode.execute(AsyncJobFlowNode.java:70)
>
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
>
> at
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
>
> Caused by: java.net.URISyntaxException: Illegal character in path at index
> 11: /hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework
>
> at java.net.URI$Parser.fail(URI.java:2829)
>
> at java.net.URI$Parser.checkChars(URI.java:3002)
>
> at java.net.URI$Parser.parseHierarchical(URI.java:3086)
>
> at java.net.URI$Parser.parse(URI.java:3044)
>
> at java.net.URI.<init>(URI.java:595)
>
> at
>
> org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:737)
>
> ... 15 more
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message