flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From se...@apache.org
Subject [4/4] incubator-flink git commit: [FLINK-1335] [docs] Add guide how to set up Eclipse, and bring README.md up to speed.
Date Tue, 06 Jan 2015 10:45:31 GMT
[FLINK-1335] [docs] Add guide how to set up Eclipse, and bring README.md up to speed.


Project: http://git-wip-us.apache.org/repos/asf/incubator-flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-flink/commit/2b86e953
Tree: http://git-wip-us.apache.org/repos/asf/incubator-flink/tree/2b86e953
Diff: http://git-wip-us.apache.org/repos/asf/incubator-flink/diff/2b86e953

Branch: refs/heads/master
Commit: 2b86e95368644794d3f7e319e53421f5e08e3a7f
Parents: cec30ff
Author: Stephan Ewen <sewen@apache.org>
Authored: Tue Jan 6 10:57:26 2015 +0100
Committer: Stephan Ewen <sewen@apache.org>
Committed: Tue Jan 6 11:28:55 2015 +0100

----------------------------------------------------------------------
 README.md                      | 71 ++++++++++++++++++++++++---
 docs/internal_howto.md         |  2 +
 docs/internal_setup_eclipse.md | 98 +++++++++++++++++++++++++++++++++++++
 3 files changed, 163 insertions(+), 8 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/2b86e953/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 248d8d4..c0fae3d 100644
--- a/README.md
+++ b/README.md
@@ -1,18 +1,35 @@
 # Apache Flink
 
+Apache Flink is an open source system for fast and versatile data analytics in clusters.
Flink supports batch and streaming analytics,
+in one system. Analytical programs can be written in concise and elegant APIs in Java and
Scala.
 
-Apache Flink is an open source system for expressive, declarative, fast, and efficient data
analysis. Flink combines the scalability and programming flexibility of distributed MapReduce-like
platforms with the efficiency, out-of-core execution, and query optimization capabilities
found in parallel databases.
+```scala
+case class WordWithCount(word: String, count: Int)
 
+val text = env.readTextFile(path)
 
-Learn more about Flink at http://flink.incubator.apache.org/
+val counts = text.flatMap { _.split("\\W+") }
+  .map { WordWithCount(_, 1) }
+  .groupBy("word")
+  .sum("count")
+
+counts.writeAsCsv(outputPath)
+```
+
+Flink is highlighted by some unique features:
+
+* Hybrid batch/streaming runtime that supports batch processing and data streaming programs.
+* Custom memory management to guarantee efficient, adaptive, and highly robust switching
between in-memory and data processing out-of-core algorithms.
+* Flexible and expressive windowing semantics for data stream programs
+* Built-in program optimizer that chooses the proper runtime operations for each program
+* Custom type analysis and serialization stack for high performance
 
 
-## Build Apache Flink
+Learn more about Flink at http://flink.incubator.apache.org/
 
-###  Build From Source
 
+## Building Apache Flink from Source
 
-#### Requirements
 * Unix-like environment (We use Linux, Mac OS X, Cygwin)
 * git
 * Maven (at least version 3.0.4)
@@ -27,8 +44,46 @@ mvn clean package -DskipTests # this will take up to 5 minutes
 Flink is now installed in `flink-dist/target`
 
 
+## Developing Flink
+
+The Flink committers use the IntelliJ IDE and Eclipse to develop the Flink codebase.
+
+Minimal requirements for an IDE are:
+* Support for Java and Scala (also mixed projects)
+* Support for Maven with Java and Scala
+
+
+### IntelliJ
+
+The IntelliJ IDE supports Maven out of the box and offers a plugin for Scala development.
+
+* IntelliJ download: [https://www.jetbrains.com/idea/](https://www.jetbrains.com/idea/)
+* IntelliJ Scala Plugin: [http://plugins.jetbrains.com/plugin/?id=1347](http://plugins.jetbrains.com/plugin/?id=1347)
+
+### Eclipse Scala IDE
+
+We recommend using Scala IDE 3.0.3, based on Eclipse Kepler. While this is a slightly older
version,
+we found it to be the verstion that works most robustly for a complex project like Flink.
+
+Further details, and a guide to newer Scala IDE versions can be found in the
+[How to setup Eclipse](https://github.com/StephanEwen/incubator-flink/blob/master/docs/internal_setup_eclipse.md)
docs.
+
+**Note:** Before following this setup, make sure to run the build from the command line once
+(`mvn clean package -DskipTests`, see above)
+
+1. Download the Scala IDE (preferred) or install the plugin to Eclipse Kepler. See [here]([How
to setup Eclipse](apache/incubator-flink/blob/master/docs/internal_setup_eclipse.md)
+   for download links and instructions.
+2. Add the "macroparadise" compiler plugin to the Scala compiler.
+   Open "Window" -> "Preferences" -> "Scala" -> "Compiler" -> "Advanced" and
put into the "Xplugin" field the path to
+   the *macroparadise* jar file (typically "/home/*-your-user-*/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar").
+   Note: If you do not have the jar file, you probably did not ran the command line build.
+3. Import the Flink Maven projects ("File" -> "Import" -> "Maven" -> "Existing Maven
Projects") 
+4. During the import, Eclipse will ask to automatically install additional Maven build helper
plugins.
+5. Close the "flink-java8" project. Since Eclipse Kepler does not support Java 8, you cannot
develop this project.
+
 
 ## Support
+
 Don’t hesitate to ask!
 
 Please contact the developers on our [mailing lists](http://flink.incubator.apache.org/community.html#mailing-lists)
if you need help.
@@ -38,7 +93,8 @@ Please contact the developers on our [mailing lists](http://flink.incubator.apac
 
 ## Documentation
 
-The documentation of Apache Flink is located on the website: http://flink.incubator.apache.org
or in the `docs/` directory of the source code.
+The documentation of Apache Flink is located on the website: [http://flink.incubator.apache.org](http://flink.incubator.apache.org)
+or in the `docs/` directory of the source code.
 
 
 ## Fork and Contribute
@@ -50,6 +106,5 @@ This article describes [how to contribute to Apache Flink](http://flink.incubato
 
 ## About
 
-Apache Flink is an effort undergoing incubation at The Apache Software Foundation (ASF),
sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects
until a further review indicates that the infrastructure, communications, and decision making
process have stabilized in a manner consistent with other successful ASF projects. While incubation
status is not necessarily a reflection of the completeness or stability of the code, it does
indicate that the project has yet to be fully endorsed by the ASF.
-
+Apache Flink is an open source project of The Apache Software Foundation (ASF).
 The Apache Flink project originated from the [Stratosphere](http://stratosphere.eu) research
project.

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/2b86e953/docs/internal_howto.md
----------------------------------------------------------------------
diff --git a/docs/internal_howto.md b/docs/internal_howto.md
index 415bda7..1aa1c9b 100644
--- a/docs/internal_howto.md
+++ b/docs/internal_howto.md
@@ -31,6 +31,8 @@ that are interested in the technology behind Flink.
 
 - [How-to: Using logging in Flink](internal_logging.html)
 
+- [How-to: Setting up Eclipse to develop Flink](internal_setup_eclipse.html)
+
 ---
 
 *This documentation is maintained by the contributors of the individual components.

http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/2b86e953/docs/internal_setup_eclipse.md
----------------------------------------------------------------------
diff --git a/docs/internal_setup_eclipse.md b/docs/internal_setup_eclipse.md
new file mode 100644
index 0000000..3b6e6a5
--- /dev/null
+++ b/docs/internal_setup_eclipse.md
@@ -0,0 +1,98 @@
+---
+title:  "How to set up Eclipse"
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+A brief guide how to set up Eclipse for development of the Flink core.
+Flink uses mixed Scala/Java projects, which pose a challenge to some IDEs.
+Below is the setup guide that works best from our personal experience.
+
+For Eclipse users, we currently recomment the Scala IDE 3.0.3, as the most robust solution.
+
+
+## Eclipse Scala IDE 3.0.3
+
+**NOTE:** While this version of the Scala IDE is not the newest, we have found it to be the
most reliably working
+version for complex projects like Flink. One restriction is, though, that it works only with
Java 7, not with Java 8.
+
+**Note:** Before following this setup, make sure to run the build from the command line once
+(`mvn clean package -DskipTests`)
+
+1. Download the Scala IDE (preferred) or install the plugin to Eclipse Kepler. See section
below for download links
+   and instructions.
+2. Add the "macroparadise" compiler plugin to the Scala compiler.
+   Open "Window" -> "Preferences" -> "Scala" -> "Compiler" -> "Advanced" and
put into the "Xplugin" field the path to
+   the *macroparadise* jar file (typically "/home/*-your-user-*/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar").
+   Note: If you do not have the jar file, you probably did not ran the command line build.
+3. Import the Flink Maven projects ("File" -> "Import" -> "Maven" -> "Existing Maven
Projects") 
+4. During the import, Eclipse will ask to automatically install additional Maven build helper
plugins.
+5. Close the "flink-java8" project. Since Eclipse Kepler does not support Java 8, you cannot
develop this project.
+
+
+### Download links for Scala IDE 3.0.3
+
+The Scala IDE 3.0.3 is a previous stable release, and download links are a bit hidden.
+
+The pre-packaged Scala IDE can be downloaded from the following links:
+
+* [Linux (64 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-linux.gtk.x86_64.tar.gz)
+* [Linux (32 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-linux.gtk.x86.tar.gz)
+* [MaxOS X Cocoa (64 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-macosx.cocoa.x86_64.zip)
+* [MaxOS X Cocoa (32 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-macosx.cocoa.x86.zip)
+* [Windows (64 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-win32.win32.x86_64.zip)
+* [Windows (32 bit)](http://downloads.typesafe.com/scalaide-pack/3.0.3.vfinal-210-20140327/scala-SDK-3.0.3-2.10-win32.win32.x86.zip)
+
+Alternatively, you can download Eclipse Kepler from [https://eclipse.org/downloads/packages/release/Kepler/SR2](https://eclipse.org/downloads/packages/release/Kepler/SR2)
+and manually add the Scala and Maven plugins by plugin site at [http://scala-ide.org/download/prev-stable.html](http://scala-ide.org/download/prev-stable.html).
+
+* Either use the update site to install the plugin ("Help" -> "Install new Software")
+* Or download the [zip file](http://download.scala-ide.org/sdk/helium/e38/scala211/stable/update-site.zip),
unpack it, and move the contents of the
+  "plugins" and "features" folders into the equally named folders of the Eclipse root directory
+
+
+## Eclipse Scala IDE 4.0.0
+
+**NOTE: From personal experience, the use of the Scala IDE 4.0.0 performs worse than previous
versions for complex projects like Flink.**
+**Version 4.0.0 does not handle mixed Java/Scala projects as robustly and it frequently raises
incorrect import and type errors.**
+
+*Note:* Before following this setup, make sure to run the build from the command line once
+(`mvn clean package -DskipTests`)
+
+1. Download the Scala IDE: [http://scala-ide.org/download/sdk.html](http://scala-ide.org/download/sdk.html)
+2. Import the Flink Maven projects (File -> Import -> Maven -> Existing Maven Projects)

+3. While importing the Flink project, the IDE may ask you to install an additional maven
build helper plugin. 
+4. After the import, you need to set the Scala version of your projects to Scala 2.10 (from
the default 2.11). 
+   To do that, select all projects that contain Scala code (marked by the small *S* on the
project icon),
+   right click and select "Scala -> Set the Scala Installation" and pick "2.10.4".
+   Currently, the project to which that is relevant are "flink-runtime", "flink-scala", "flink-scala-examples",
+   "flink-streaming-example", "flink-streaming-scala", "flink-tests", "flink-test-utils",
and "flink-yarn".
+5. Depending on your version of the Scala IDE, you may need to add the "macroparadise" compiler
plugin to the
+   Scala compiler. Open "Window" -> "Preferences" -> "Scala" -> "Compiler" ->
"Advanced" and put into the "Xplugin" field
+   the path to the *macroparadise* jar file (typically "/home/*-your-user-*/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar")
+6. In order to compile the "flink-java-8" project, you may need to add a Java 8 execution
environment.
+   See [this post](http://stackoverflow.com/questions/25391207/how-do-i-add-execution-environment-1-8-to-eclipse-luna)
+   for details.
+
+---
+
+*This documentation is maintained by the contributors of the individual components.
+We kindly ask anyone that adds and changes components to eventually provide a patch
+or pull request that updates these documents as well.*
+


Mime
View raw message