flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rmetz...@apache.org
Subject git commit: [FLINK-1068] Remove cdh4 specific build profile.
Date Wed, 27 Aug 2014 14:12:17 GMT
Repository: incubator-flink
Updated Branches:
  refs/heads/master ce3ed7e0b -> f96a7e0e5


[FLINK-1068] Remove cdh4 specific build profile.

Users using CDH4 can now build Flink against Hadoop 2.0.0-alpha.


Project: http://git-wip-us.apache.org/repos/asf/incubator-flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-flink/commit/f96a7e0e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-flink/tree/f96a7e0e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-flink/diff/f96a7e0e

Branch: refs/heads/master
Commit: f96a7e0e57c382d2154fa7b004d18e6a6e2eaa89
Parents: ce3ed7e
Author: Robert Metzger <rmetzger@apache.org>
Authored: Wed Aug 27 16:07:52 2014 +0200
Committer: Robert Metzger <rmetzger@apache.org>
Committed: Wed Aug 27 16:11:32 2014 +0200

----------------------------------------------------------------------
 docs/building.md                                | 39 +++++++-------------
 flink-addons/flink-avro/pom.xml                 | 17 ---------
 flink-addons/flink-hadoop-compatibility/pom.xml | 23 ------------
 flink-addons/flink-spargel/pom.xml              | 22 -----------
 flink-dist/pom.xml                              | 15 --------
 flink-runtime/pom.xml                           | 25 -------------
 flink-test-utils/pom.xml                        | 17 ---------
 flink-tests/pom.xml                             | 12 ------
 pom.xml                                         |  5 +++
 tools/create_release_files.sh                   |  4 +-
 10 files changed, 19 insertions(+), 160 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/f96a7e0e/docs/building.md
----------------------------------------------------------------------
diff --git a/docs/building.md b/docs/building.md
index a07b324..842dbc1 100644
--- a/docs/building.md
+++ b/docs/building.md
@@ -3,7 +3,7 @@ title:  "Build Flink"
 ---
 
 
-In order to build Flink, you need the source code. Either download the source of a release
or clone the git repository. In addition to that, you need Maven 3 and a JDK (Java Development
Kit). Note that you can not build Flink with Oracle JDK 6 due to a unresolved bug in the Java
compiler. It works well with OpenJDK 6 and all Java 7 and 8 compilers.
+In order to build Flink, you need the source code. Either download the source of a release
or clone the git repository. In addition to that, you need Maven 3 and a JDK (Java Development
Kit). Note that you can not build Flink with Oracle JDK 6 due to a unresolved bug in the Oracle
Java compiler. It works well with OpenJDK 6 and all Java 7 and 8 compilers.
 
 To clone from git, enter:
 ```
@@ -36,7 +36,7 @@ ERROR: The job was not successfully submitted to the nephele job manager:
 
 There are two main versions of Hadoop that we need to differentiate:
 - Hadoop 1, with all versions starting with zero or one, like 0.20, 0.23 or 1.2.1.
-- Hadoop 2 with all versions including or higher than 2.2.0.
+- Hadoop 2, with all versions starting with 2, like 2.2.0.
 The main differentiation between Hadoop 1 and Hadoop 2 is the availability of Hadoop YARN
(Hadoops cluster resource manager).
 
 **To build Flink for Hadoop 2**, issue the following command:
@@ -55,11 +55,21 @@ mvn clean package -DskipTests -Dhadoop.profile=2 -Dhadoop.version=2.4.1
 
 **To build Flink against a vendor specific Hadoop version**, issue the following command:
 ```
-mvn clean package -DskipTests -Pvendor-repos -Dhadoop.version=2.2.0-cdh5.0.0-beta-2
+mvn clean package -DskipTests -Pvendor-repos -Dhadoop.profile=2 -Dhadoop.version=2.2.0-cdh5.0.0-beta-2
 ```
 
 The `-Pvendor-repos` activates a Maven [build profile](http://maven.apache.org/guides/introduction/introduction-to-profiles.html)
that includes the repositories of popular Hadoop vendors such as Cloudera, Hortonworks, or
MapR.
 
+**Build Flink for `hadoop2` versions before 2.2.0**
+
+Maven will automatically build Flink with its YARN client if the `-Dhadoop.profile=2` is
set. But there were some changes in Hadoop versions before the 2.2.0 Hadoop release that are
not supported by Flink's YARN client. Therefore, you can disable building the YARN client
with the following string: `-P\!include-yarn`. 
+
+So if you are building Flink for Hadoop `2.0.0-alpha`, use the following command:
+
+```bash
+-P\!include-yarn -Dhadoop.profile=2 -Dhadoop.version=2.0.0-alpha
+```
+
 ### Background
 
 The builds with Maven are controlled by [properties](http://maven.apache.org/pom.html#Properties)
and <a href="http://maven.apache.org/guides/introduction/introduction-to-profiles.html">build
profiles</a>.
@@ -69,26 +79,3 @@ Depending on the profile, there are two Hadoop versions, set via properties.
For
 
 You can change these versions with the `hadoop-two.version` (or `hadoop-one.version`) property.
For example `-Dhadoop-two.version=2.4.0`.
 
-
-### Example for Cloudera Hadoop 5 Beta 2
-
-
-```
-mvn -Dhadoop.profile=2 -Pvendor-repos -Dhadoop.version=2.2.0-cdh5.0.0-beta-2 -DskipTests
package
-```
-
-The commands in detail:
-
-*  `-Dhadoop.profile=2` activates the Hadoop YARN profile of Flink. This will enable all
components of Flink that are compatible with Hadoop 2.2
-*  `-Pvendor-repos` is adding the Maven repositories of MapR, Cloudera and Hortonworks into
your Maven build.
-* `-Dhadoop.version=2.2.0-cdh5.0.0-beta-2` sets a special version of the Hadoop dependencies.
Make sure that the specified Hadoop version is compatible with the profile you activated.
-
-If you want to build HDFS for Hadoop 2 without YARN, use the following parameter:
-
-```
--P!include-yarn
-```
-
-Some Cloudera versions (such as `2.0.0-cdh4.2.0`) require this, since they have a new HDFS
version with the old YARN API.
-
-Please post to the _Flink mailinglist_(dev@flink.incubator.apache.org) or create an issue
on [Jira]({{site.FLINK_ISSUES_URL}}), if you have issues with your YARN setup and Flink.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/f96a7e0e/flink-addons/flink-avro/pom.xml
----------------------------------------------------------------------
diff --git a/flink-addons/flink-avro/pom.xml b/flink-addons/flink-avro/pom.xml
index 9045ee6..68f722c 100644
--- a/flink-addons/flink-avro/pom.xml
+++ b/flink-addons/flink-avro/pom.xml
@@ -183,23 +183,6 @@ under the License.
 				<maven.javadoc.skip>true</maven.javadoc.skip>
 			</properties>
 		</profile>
-		<profile>
-			<id>cdh4</id>
-			<dependencies>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-hdfs</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-					<scope>test</scope>
-				</dependency>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-common</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-					<scope>test</scope>
-				</dependency>
-			</dependencies>
-		</profile>
 	</profiles>
 
 </project>

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/f96a7e0e/flink-addons/flink-hadoop-compatibility/pom.xml
----------------------------------------------------------------------
diff --git a/flink-addons/flink-hadoop-compatibility/pom.xml b/flink-addons/flink-hadoop-compatibility/pom.xml
index dfae773..a8ebe34 100644
--- a/flink-addons/flink-hadoop-compatibility/pom.xml
+++ b/flink-addons/flink-hadoop-compatibility/pom.xml
@@ -78,28 +78,5 @@ under the License.
 				</dependency>
 			</dependencies>
 		</profile>
-			<profile>
-			<id>cdh4</id>
-			<!-- 
-				See flink-runtime/pom.xml for details
-			-->
-			<dependencies>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-mapreduce-client-core</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-				</dependency>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-hdfs</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-				</dependency>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-common</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-				</dependency>
-			</dependencies>
-		</profile>
 	</profiles>
 </project>

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/f96a7e0e/flink-addons/flink-spargel/pom.xml
----------------------------------------------------------------------
diff --git a/flink-addons/flink-spargel/pom.xml b/flink-addons/flink-spargel/pom.xml
index 17e0ff9..d638409 100644
--- a/flink-addons/flink-spargel/pom.xml
+++ b/flink-addons/flink-spargel/pom.xml
@@ -57,26 +57,4 @@ under the License.
 			<scope>test</scope>
 		</dependency>
 	</dependencies>
-
-	<profiles>
-
-		<profile>
-			<!-- Needed for the tests -->
-			<id>cdh4</id>
-			<dependencies>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-hdfs</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-					<scope>test</scope>
-				</dependency>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-common</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-					<scope>test</scope>
-				</dependency>
-			</dependencies>
-		</profile>
-	</profiles>
 </project>

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/f96a7e0e/flink-dist/pom.xml
----------------------------------------------------------------------
diff --git a/flink-dist/pom.xml b/flink-dist/pom.xml
index 327a8b2..0514b66 100644
--- a/flink-dist/pom.xml
+++ b/flink-dist/pom.xml
@@ -335,21 +335,6 @@ under the License.
 				</plugins>
 			</build>
 		</profile>
-		<profile>
-			<id>cdh4</id>
-			<dependencies>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-common</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-				</dependency>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-hdfs</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-				</dependency>
-			</dependencies>
-		</profile>
 	</profiles>
 	
 	

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/f96a7e0e/flink-runtime/pom.xml
----------------------------------------------------------------------
diff --git a/flink-runtime/pom.xml b/flink-runtime/pom.xml
index 71f251d..2a36dc8 100644
--- a/flink-runtime/pom.xml
+++ b/flink-runtime/pom.xml
@@ -220,30 +220,5 @@ under the License.
 				</dependency>
 			</dependencies>
 		</profile>
-
-		<profile>
-			<id>cdh4</id>
-			<!-- 
-
-			It seems http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Installation-Guide/cdh4ig_topic_31.html
-			that this particular CDH4 version has already separate packages for hdfs and common.
-
-			To activate the cdh4 profile, use the -Pcdh4 flag. (You probably want -Pcdh4,vendor-repos)
-			You also HAVE TO set the -Dhadoop.cdh4.hadoop.version property (for example to: 2.0.0-cdh4.2.0)
-
-			-->
-			<dependencies>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-hdfs</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-				</dependency>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-common</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-				</dependency>
-			</dependencies>
-		</profile>
 	</profiles>
 </project>

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/f96a7e0e/flink-test-utils/pom.xml
----------------------------------------------------------------------
diff --git a/flink-test-utils/pom.xml b/flink-test-utils/pom.xml
index c6f07dd..76a5b7d 100644
--- a/flink-test-utils/pom.xml
+++ b/flink-test-utils/pom.xml
@@ -66,21 +66,4 @@ under the License.
 			<version>4.11</version>
 		</dependency>
 	</dependencies>
-	<profiles>
-		<profile>
-			<id>cdh4</id>
-			<dependencies>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-common</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-				</dependency>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-hdfs</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-				</dependency>
-			</dependencies>
-		</profile>
-	</profiles>
 </project>

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/f96a7e0e/flink-tests/pom.xml
----------------------------------------------------------------------
diff --git a/flink-tests/pom.xml b/flink-tests/pom.xml
index 7074312..fb36226 100644
--- a/flink-tests/pom.xml
+++ b/flink-tests/pom.xml
@@ -231,16 +231,4 @@ under the License.
 			</plugins>
 		</pluginManagement>
 	</build>
-	<profiles>
-		<profile>
-			<id>cdh4</id>
-			<dependencies>
-				<dependency>
-					<groupId>org.apache.hadoop</groupId>
-					<artifactId>hadoop-common</artifactId>
-					<version>${hadoop.cdh4.hadoop.version}</version>
-				</dependency>
-			</dependencies>
-		</profile>
-	</profiles>
 </project>

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/f96a7e0e/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 009d9b8..229a8aa 100644
--- a/pom.xml
+++ b/pom.xml
@@ -198,6 +198,11 @@ under the License.
 						<artifactId>hadoop-hdfs</artifactId>
 						<version>${hadoop.version}</version>
 					</dependency>
+					<dependency>
+						<groupId>org.apache.hadoop</groupId>
+						<artifactId>hadoop-client</artifactId>
+						<version>${hadoop.version}</version>
+					</dependency>
 				</dependencies>
 			</dependencyManagement>
 		</profile>

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/f96a7e0e/tools/create_release_files.sh
----------------------------------------------------------------------
diff --git a/tools/create_release_files.sh b/tools/create_release_files.sh
index d580dc6..3693cfa 100755
--- a/tools/create_release_files.sh
+++ b/tools/create_release_files.sh
@@ -121,11 +121,9 @@ make_binary_release() {
 }
 
 make_binary_release "hadoop1" ""
-make_binary_release "cdh4" "-Pvendor-repos,cdh4 -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -Dhadoop.cdh4.hadoop.version=2.0.0-cdh4.2.0"
-# make_binary_release "cdh5" "-Dhadoop.profile=2 -Pvendor-repos -Dhadoop.version=2.3.0+cdh5.1.0+795"
+make_binary_release "hadoop200alpha" "-P\!include-yarn -Dhadoop.profile=2 -Dhadoop.version=2.0.0-alpha"
 make_binary_release "hadoop2" "-Dhadoop.profile=2"
 # make_binary_release "mapr4" "-Dhadoop.profile=2 -Pvendor-repos -Dhadoop.version=2.3.0-mapr-4.0.0-FCS"
-# make_binary_release "hdp21" "-Dhadoop.profile=2 -Pvendor-repos -Dhadoop.version=2.4.0.2.1.3.0-563"
 
 
 # Copy data


Mime
View raw message