flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: How to compile Flink against vendor-specific Hadoop libs?
Date Mon, 07 Jul 2014 06:52:49 GMT
Hi,

the builds using maven are controlled by maven properties and build
profiles.
There are two profiles, one for hadoop1 and one for hadoop2. When the
hadoop2 profile is enabled, the system will also build the YARN client.
The hadoop1 profile is used by default. To enable the hadoop2 profile, set
-Dhadoop.profile=2 when building.
Depending on the profile, there are two Hadoop versions. For "hadoop1", we
use 1.2.1 by default, for "hadoop2" its 2.2.0.
You can change these versions with the "hadoop-two.version" (or
hadoop-one.version) property. For example -Dhadoop-two.version=2.4.0.

The build profile for cloudera just includes the cloudera maven repository.
If its enabled, you can build against a cloudera specific version, for
example "2.3.0-cdh5.0.0".
I will add similar profiles for Hortonworks and MapR as well:
https://issues.apache.org/jira/browse/FLINK-1004

In addition to that, I think we should provide pre-packaged binaries for
the different hadoop distributions so that users don't have to recompile.

Best,
Robert

On Sun, Jul 6, 2014 at 2:30 PM, Daniel Warneke <warneke@apache.org> wrote:

> Hi,
>
> I would like to compile Flink against a specific Hadoop distribution.
> What is the best way to do this?
>
> I saw there is already a special build profile defined for Cloudera.
> Would it make sense to introduce such a build profile for Hortonworks
> and MapR as well?
>
> Bests,
>
>     Daniel
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message