flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 徐涛 <happydexu...@gmail.com>
Subject Re: How to tune Hadoop version in flink shaded jar to Hadoop version actually used?
Date Wed, 31 Oct 2018 01:24:52 GMT
Hi Hequn & Vino,
	Finally I rebuild the Flink by change the “hadoop.version” in the pom file. 
	Because Flink use maven shaded plugin to shade the Hadoop dependency, this also means I need
to rebuild the hadoop shaded jar each time I upgrade Flink version.

Best
Henry

> 在 2018年10月30日,下午12:35,Hequn Cheng <chenghequn@gmail.com> 写道:
> 
> Hi Henry,
> 
> You can specify a specific Hadoop version to build against:
> mvn clean install -DskipTests -Dhadoop.version=2.6.1
>  More details here[1].
> 
> Best, Hequn
> 
> [1] https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/building.html#hadoop-versions
<https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/building.html#hadoop-versions>
> On Tue, Oct 30, 2018 at 10:02 AM vino yang <yanghua1127@gmail.com <mailto:yanghua1127@gmail.com>>
wrote:
> Hi Henry,
> 
> You just need to change the node of "hadoop.version" in the parent pom file.
> 
> Thanks, vino.
> 
> 徐涛 <happydexutao@gmail.com <mailto:happydexutao@gmail.com>> 于2018年10月29日周一
下午11:23写道:
> Hi Vino,
> 	Because I build the project with Maven, maybe I can not use the jars directly download
from the web.
> 	If built with Maven, how can I adjust the Hadoop version with the Hadoop version really
used?
> 	Thanks a lot!!
> 
> Best 
> Henry
> 
>> 在 2018年10月26日,上午10:02,vino yang <yanghua1127@gmail.com <mailto:yanghua1127@gmail.com>>
写道:
>> 
>> Hi Henry,
>> 
>> When running flink on YARN, from ClusterEntrypoint the system environment info is
print out.
>> One of the info is "Hadoop version: 2.4.1”, I think it is from the flink-shaded-hadoop2
jar. But actually the system Hadoop version is 2.7.2.
>> 
>> I want to know is it OK if the version is different? 
>> 
>> > I don't think it is OK, because you will use a lower version of the client to
access the higher version of the server.
>> 
>> Is it a best practice to adjust flink Hadoop version to the Hadoop version actually
used?
>> 
>> > I personally recommend that you keep the two versions consistent to eliminate
the possibility of causing various potential problems. 
>> In fact, Flink provides a bundle of Hadoop 2.7.x bundles for you to download.[1]
>> 
>> [1]: https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz
<https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz>
>> 
>> Thanks, vino.
>> 
>> 徐涛 <happydexutao@gmail.com <mailto:happydexutao@gmail.com>> 于2018年10月26日周五
上午9:13写道:
>> Hi Experts
>>         When running flink on YARN, from ClusterEntrypoint the system environment
info is print out.
>>         One of the info is "Hadoop version: 2.4.1”, I think it is from the flink-shaded-hadoop2
jar. But actually the system Hadoop version is 2.7.2.
>>         I want to know is it OK if the version is different? Is it a best practice
to adjust flink Hadoop version to the Hadoop version actually used?
>> 
>>         Thanks a lot.
>> 
>> Best
>> Henry
> 


Mime
View raw message