flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rmetz...@apache.org
Subject [75/92] [abbrv] git commit: [FLINK-1004] Add the maven repositories for all vendors + documentation
Date Tue, 22 Jul 2014 10:41:15 GMT
[FLINK-1004] Add the maven repositories for all vendors + documentation

This closes #62


Project: http://git-wip-us.apache.org/repos/asf/incubator-flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-flink/commit/ca4e7b4b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-flink/tree/ca4e7b4b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-flink/diff/ca4e7b4b

Branch: refs/heads/travis_test
Commit: ca4e7b4b3b680955cae944d32a462bd18c8ba0e2
Parents: fa9daad
Author: Robert Metzger <rmetzger@apache.org>
Authored: Mon Jul 7 14:02:57 2014 +0200
Committer: Robert Metzger <rmetzger@apache.org>
Committed: Sat Jul 12 17:36:35 2014 +0200

----------------------------------------------------------------------
 docs/_layouts/docs.html |  1 +
 docs/building.md        | 66 ++++++++++++++++++++++++++++++++++++++++++++
 docs/yarn_setup.md      | 32 ++-------------------
 pom.xml                 | 32 +++++++++++++++++++--
 4 files changed, 99 insertions(+), 32 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/ca4e7b4b/docs/_layouts/docs.html
----------------------------------------------------------------------
diff --git a/docs/_layouts/docs.html b/docs/_layouts/docs.html
index f4d99e0..f69b26a 100644
--- a/docs/_layouts/docs.html
+++ b/docs/_layouts/docs.html
@@ -36,6 +36,7 @@
 
                     <li>Setup &amp; Configuration
                         <ul>
+                            <li><a href="building.html">Build Instructions</a></li>
                             <li><a href="local_setup.html">Local Setup</a></li>
                             <li><a href="cluster_setup.html">Cluster Setup</a></li>
                             <li><a href="yarn_setup.html">YARN Setup</a></li>

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/ca4e7b4b/docs/building.md
----------------------------------------------------------------------
diff --git a/docs/building.md b/docs/building.md
new file mode 100644
index 0000000..86810a4
--- /dev/null
+++ b/docs/building.md
@@ -0,0 +1,66 @@
+---
+title:  "Build Flink"
+---
+
+
+In order to build Flink, you need the source code. Either download the source of a release
or clone the git repository. In addition to that, you need Maven 3 and a JDK. Note that you
can not build Flink with Oracle JDK 6 due to a unresolved bug in the Java compiler. It works
well with OpenJDK 6 and all Java 7 and 8 compilers.
+
+To clone from git, enter:
+```
+git clone {{ site.FLINK_GITHUB_URL }}
+```
+
+The simplest way of building Flink is by running:
+
+```
+mvn clean package -DskipTests
+```
+
+This instructs Maven (`mvn`) to first remove all existing builds (`clean`) and then create
a new Flink binary (`package`). The `-DskipTests` command prevents Maven from executing the
unit tests.
+
+
+
+## Build Flink for a specific Hadoop Version
+
+This section covers building Flink for a specific Hadoop version. Most users do not need
to do this manually.
+The problem is that Flink uses HDFS and YARN which are both from Apache Hadoop. There exist
many different builds of Hadoop (from both the upstream project and the different Hadoop distributions).
Typically errors arise with the RPC services. An error could look like this:
+
+```
+ERROR: The job was not successfully submitted to the nephele job manager:
+    org.apache.flink.nephele.executiongraph.GraphConversionException: Cannot compute input
splits for TSV:
+    java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException:
+    Protocol message contained an invalid tag (zero).; Host Details :
+```
+
+### Background
+
+The builds with Maven are controlled by [properties](http://maven.apache.org/pom.html#Properties)
and <a href="http://maven.apache.org/guides/introduction/introduction-to-profiles.html">build
profiles</a>.
+There are two profiles, one for hadoop1 and one for hadoop2. When the hadoop2 profile is
enabled, the system will also build the YARN client.
+The hadoop1 profile is used by default. To enable the hadoop2 profile, set `-Dhadoop.profile=2`
when building.
+Depending on the profile, there are two Hadoop versions, set via properties. For "hadoop1",
we use 1.2.1 by default, for "hadoop2" it is 2.2.0.
+
+You can change these versions with the `hadoop-two.version` (or `hadoop-one.version`) property.
For example `-Dhadoop-two.version=2.4.0`.
+
+
+### Example for Cloudera Hadoop 5 Beta 2
+
+
+```
+mvn -Dhadoop.profile=2 -Pvendor-repos -Dhadoop.version=2.2.0-cdh5.0.0-beta-2 -DskipTests
package
+```
+
+The commands in detail:
+
+*  `-Dhadoop.profile=2` activates the Hadoop YARN profile of Flink. This will enable all
components of Flink that are compatible with Hadoop 2.2
+*  `-Pvendor-repos` is adding the Maven repositories of MapR, Cloudera and Hortonworks into
your Maven build.
+* `-Dhadoop.version=2.2.0-cdh5.0.0-beta-2` sets a special version of the Hadoop dependencies.
Make sure that the specified Hadoop version is compatible with the profile you activated.
+
+If you want to build HDFS for Hadoop 2 without YARN, use the following parameter:
+
+```
+-P!include-yarn
+```
+
+Some Cloudera versions (such as `2.0.0-cdh4.2.0`) require this, since they have a new HDFS
version with the old YARN API.
+
+Please post to the _Flink mailinglist_(dev@flink.incubator.apache.org) or create an issue
on [Jira]({{site.FLINK_ISSUES_URL}}), if you have issues with your YARN setup and Flink.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/ca4e7b4b/docs/yarn_setup.md
----------------------------------------------------------------------
diff --git a/docs/yarn_setup.md b/docs/yarn_setup.md
index 3335f8f..dc76826 100644
--- a/docs/yarn_setup.md
+++ b/docs/yarn_setup.md
@@ -134,39 +134,11 @@ You can check the number of TaskManagers in the JobManager web interface.
The ad
 
 If the TaskManagers do not show up after a minute, you should investigate the issue using
the log files.
 
-# Build Flink for a specific Hadoop Version
 
-This section covers building Flink for a specific Hadoop version. Most users do not need
to do this manually.
-The problem is that Flink uses HDFS and YARN which are both from Apache Hadoop. There exist
many different builds of Hadoop (from both the upstream project and the different Hadoop distributions).
Typically errors arise with the RPC services. An error could look like this:
+# Build YARN client for a specific Hadoop version
 
-```
-ERROR: The job was not successfully submitted to the nephele job manager:
-    org.apache.flinknephele.executiongraph.GraphConversionException: Cannot compute input
splits for TSV:
-    java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException:
-    Protocol message contained an invalid tag (zero).; Host Details :
-```
-
-**Example**
-
-```
-mvn -Dhadoop.profile=2 -Pcdh-repo -Dhadoop.version=2.2.0-cdh5.0.0-beta-2 -DskipTests package
-```
-
-The commands in detail:
-
-*  `-Dhadoop.profile=2` activates the Hadoop YARN profile of Flink. This will enable all
components of Flink that are compatible with Hadoop 2.2
-*  `-Pcdh-repo` activates the Cloudera Hadoop dependencies. If you want other vendor's Hadoop
dependencies (not in maven central) add the repository to your local maven configuration in
`~/.m2/`.
-* `-Dhadoop.version=2.2.0-cdh5.0.0-beta-2` sets a special version of the Hadoop dependencies.
Make sure that the specified Hadoop version is compatible with the profile you activated.
-
-If you want to build HDFS for Hadoop 2 without YARN, use the following parameter:
-
-```
--P!include-yarn
-```
-
-Some Cloudera versions (such as `2.0.0-cdh4.2.0`) require this, since they have a new HDFS
version with the old YARN API.
+Users using Hadoop distributions from companies like Hortonworks, Cloudera or MapR might
have to build Flink against their specific versions of Hadoop (HDFS) and YARN. Please read
the [build instructions](building.html) for more details.
 
-Please post to the _Flink mailinglist_(dev@flink.incubator.apache.org) or create an issue
on [Jira]({{site.FLINK_ISSUES_URL}}), if you have issues with your YARN setup and Flink.
 
 # Background
 

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/ca4e7b4b/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index aadcdf8..88bae88 100644
--- a/pom.xml
+++ b/pom.xml
@@ -193,9 +193,12 @@
 				</dependencies>
 			</dependencyManagement>
 		</profile>
+
 		<profile>
-			<id>cdh-repo</id>
-				<repositories>
+			<id>vendor-repos</id>
+			<!-- Add vendor maven repositories -->
+			<repositories>
+				<!-- Cloudera -->
 				<repository>
 					<id>cloudera-releases</id>
 					<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
@@ -206,8 +209,33 @@
 						<enabled>false</enabled>
 					</snapshots>
 				</repository>
+				<!-- Hortonworks -->
+				<repository>
+					<releases>
+						<enabled>true</enabled>
+						<updatePolicy>always</updatePolicy>
+						<checksumPolicy>warn</checksumPolicy>
+					</releases>
+					<snapshots>
+						<enabled>false</enabled>
+						<updatePolicy>never</updatePolicy>
+						<checksumPolicy>fail</checksumPolicy>
+					</snapshots>
+					<id>HDPReleases</id>
+					<name>HDP Releases</name>
+					<url>http://repo.hortonworks.com/content/repositories/releases/</url>
+					<layout>default</layout>
+				</repository>
+				<!-- MapR -->
+				<repository>
+					<id>mapr-releases</id>
+					<url>http://repository.mapr.com/maven/</url>
+					<snapshots><enabled>false</enabled></snapshots>
+					<releases><enabled>true</enabled></releases>
+				</repository>
 			</repositories>
 		</profile>
+
 		<profile>
 			<id>release</id>
 				<build>


Mime
View raw message