Return-Path: X-Original-To: apmail-flink-commits-archive@minotaur.apache.org Delivered-To: apmail-flink-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ADD9D11A23 for ; Fri, 4 Jul 2014 11:02:06 +0000 (UTC) Received: (qmail 78315 invoked by uid 500); 4 Jul 2014 11:02:04 -0000 Delivered-To: apmail-flink-commits-archive@flink.apache.org Received: (qmail 78252 invoked by uid 500); 4 Jul 2014 11:02:04 -0000 Mailing-List: contact commits-help@flink.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.incubator.apache.org Delivered-To: mailing list commits@flink.incubator.apache.org Received: (qmail 78204 invoked by uid 99); 4 Jul 2014 11:02:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jul 2014 11:02:04 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jul 2014 11:01:53 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id B19762388CB6; Fri, 4 Jul 2014 11:00:29 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1607832 [30/33] - in /incubator/flink: ./ _includes/ _layouts/ _plugins/ _posts/ blog/ css/ fonts/ img/ img/blog/ js/ site/ site/blog/ site/blog/page2/ site/css/ site/docs/ site/docs/0.6-SNAPSHOT/ site/docs/0.6-SNAPSHOT/css/ site/docs/0.6-... Date: Fri, 04 Jul 2014 11:00:20 -0000 To: commits@flink.incubator.apache.org From: rmetzger@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20140704110029.B19762388CB6@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Added: incubator/flink/site/downloads.html URL: http://svn.apache.org/viewvc/incubator/flink/site/downloads.html?rev=1607832&view=auto ============================================================================== --- incubator/flink/site/downloads.html (added) +++ incubator/flink/site/downloads.html Fri Jul 4 11:00:15 2014 @@ -0,0 +1,231 @@ + + + + + + + Apache Flink (incubating): Downloads + + + + + + + + +
+ +

Downloads

+ +
+
+ Note +
+
+ The source code and parts of the documentation have not been renamed yet to Flink. +
+
+ +

Pick the Apache Flink package matching your Hadoop version.

+ +

Stable

+ +

Apache Flink 0.5.1 is our latest stable release.

+ + + +

Maven Dependencies

+ +

You can add the following dependencies to your pom.xml to include Apache Flink in your project.

+
<dependency>
+  <groupId>eu.stratosphere</groupId>
+  <artifactId>stratosphere-java</artifactId>
+  <version>0.5.1</version>
+</dependency>
+<dependency>
+  <groupId>eu.stratosphere</groupId>
+  <artifactId>stratosphere-clients</artifactId>
+  <version>0.5.1</version>
+</dependency>
+
+

These dependencies include a local execution environment and thus support local testing.

+ +
    +
  • Hadoop 2: If you want to interact with Hadoop 2, use 0.5.1-hadoop2 as the version.
  • +
  • Scala API: To use the Scala API, replace the stratosphere-java artifact id with stratosphere-scala.
  • +
+ +

Latest

+ +

Apache Flink 0.6-SNAPSHOT is our latest development version.

+ +

You can download a packaged version of our nightly builds, which include +the most recent development code. You can use them if you need a feature +before its release. Only builds that pass all tests are published here.

+ + + +

Add the Sonatype Snapshot repository to your Maven pom.xml:

+
<repositories>
+  <repository>
+    <id>snapshots-repo</id>
+    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
+    <releases><enabled>false</enabled></releases>
+    <snapshots><enabled>true</enabled></snapshots>
+  </repository>
+</repositories>
+
+

You can now include Apache Flink as a Maven dependency (see above) with version 0.6-SNAPSHOT.

+ +

Checkout from Source

+ +

You can checkout Apache Flink 0.6-SNAPSHOT and build it on your own machine.

+
git clone https://github.com/apache/incubator-flink
+cd incubator-flink
+mvn clean package -DskipTests
+
+

Note: Flink does not build with Oracle JDK 6. It runs with Oracle JDK 6.

+ +

If you want to build for Hadoop 2, activate the build profile via mvn clean package -DskipTests -Dhadoop.profile=2.

+ + + +
+ + + + + + + + Added: incubator/flink/site/fonts/glyphicons-halflings-regular.eot URL: http://svn.apache.org/viewvc/incubator/flink/site/fonts/glyphicons-halflings-regular.eot?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/fonts/glyphicons-halflings-regular.eot ------------------------------------------------------------------------------ svn:mime-type = application/vnd.ms-fontobject Added: incubator/flink/site/fonts/glyphicons-halflings-regular.svg URL: http://svn.apache.org/viewvc/incubator/flink/site/fonts/glyphicons-halflings-regular.svg?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/fonts/glyphicons-halflings-regular.svg ------------------------------------------------------------------------------ svn:mime-type = image/svg+xml Added: incubator/flink/site/fonts/glyphicons-halflings-regular.ttf URL: http://svn.apache.org/viewvc/incubator/flink/site/fonts/glyphicons-halflings-regular.ttf?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/fonts/glyphicons-halflings-regular.ttf ------------------------------------------------------------------------------ svn:mime-type = application/x-font-ttf Added: incubator/flink/site/fonts/glyphicons-halflings-regular.woff URL: http://svn.apache.org/viewvc/incubator/flink/site/fonts/glyphicons-halflings-regular.woff?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/fonts/glyphicons-halflings-regular.woff ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: incubator/flink/site/how-to-contribute.html URL: http://svn.apache.org/viewvc/incubator/flink/site/how-to-contribute.html?rev=1607832&view=auto ============================================================================== --- incubator/flink/site/how-to-contribute.html (added) +++ incubator/flink/site/how-to-contribute.html Fri Jul 4 11:00:15 2014 @@ -0,0 +1,243 @@ + + + + + + + Apache Flink (incubating): How To Contribute + + + + + + + + +
+ +
+
+

How To Contribute

+ + + + +

The Flink project welcomes all sorts contributions in the form of code (improvements, features, bugfixes), documentation, tests, and community participation (discussions & questions).

+ +

Easy Issues for Starters

+ +

We maintain all known issues and feature drafts in the Flink project JIRA.

+ +

We also try to maintain a list of simple "starter issues" that we believe are good tasks for new contributors. Those tasks are meant to allow people to get into the project and become familiar with the process of contributing. Feel free to ask questions about issues that you would be interested in working on.

+ +

In addition, you can find a list of ideas for projects and improvements in the projects wiki page.

+ +

Contributing Code & Documentation

+ +

This section gives you a brief introduction in how to contribute code and documentation to Flink. We maintain both the code and the documentation in the same repository, so the process is essentially the same for both. We use git for the code and documentation version control.

+ +

The Flink project accepts code contributions though the GitHub Mirror, in the form of Pull Requests. Pull requests are basically a simpler way of offering a patch, by providing a pointer to a code branch that contains the change. It is also possible to attach a patch to a JIRA issue.

+ +

Setting up the Infrastructure and Creating a Pull Request

+ +
    +
  1. The first step is to create yourself a copy of the Flink code base. We suggest to fork the Flink GitHub Mirror Repository into your own GitHub account. You need to register on GitHub for that, if you have no account so far.

  2. +
  3. Next, clone your repository fork to your local machine.

    +
    git clone https://github.com/<your-user-name>/incubator-flink.git
    +
  4. +
  5. It is typically helpful to switch to a topic branch for the changes. To create a dedicated branch based on the current master, use the following command:

    +
    git checkout -b myBranch master
    +
  6. +
  7. Now you can create your changes, compile the code, and validate the changes. Here are some pointers on how to set up the Eclipse IDE for development, and how to build the code.

  8. +
  9. After you have finalized your contribution, verify the compliance with the contribution guidelines (see below), and commit them. To make the changes easily mergeable, please rebase them to the latest version of the main repositories master branch. Assuming you created a topic branch (step 3), you can follow this sequence of commands to do that: +Switch to the master branch, update it to the latest revision, switch back to your topic branch, and rebase it on top of the master branch.

    +
    git checkout master
    +git pull https://github.com/apache/incubator-flink.git master
    +git checkout myBranch
    +git rebase master
    +
    +

    Have a look here for more information about rebasing commits.

  10. +
  11. Push the contribution it back into your fork of the Flink repository.

    +
    git push origin myBranch
    +
    +

    Go the website of your repository fork (https://github.com/<your-user-name>/incubator-flink) and use the "Create Pull Request" button to start creating a pull request. Make sure that the base fork is apache/incubator-flink master and the head fork selects the branch with your changes. Give the pull request a meaningful description and send it.

  12. +
+ +

Verifying the Compliance of your Code

+ +

Before sending a patch or pull request, please verify that it complies with the guidelines of the project. While we try to keep the set of guidelines small and easy, it is important to follow those rules in order to guarantee good code quality, to allow efficient reviews, and to allow committers to easily merge your changes.

+ +

Please have a look at the coding guidelines for a guide to the format of code and pull requests.

+ +

Most important of all, verify that your changes are correct and do not break existing functionality. Run the existing tests by calling mvn verify in the root directory of the repository, and make sure that the tests succeed. We encourage every contributor to use a continuous integration service that will automatically test the code in your repository whenever you push a change. Flink is pre-configured for Travis CI, which can be easily enabled for your private repository fork (it uses GitHub for authentication, so you so not need an additional account). Simply add the Travis CI hook to your repository (settings --> webhooks & services --> add service) and enable tests for the "incubator-flink" repository on Travis.

+ +

When contributing documentation, please review the rendered HTML versions of the documents you changed. You can look at the HTML pages by using the rendering script in preview mode.

+
cd docs
+./build_docs.sh -p
+
+

Now, open your browser at http://localhost:4000 and check out the pages you changed.

+ +

How to become a committer

+ +

There is no strict protocol for becoming a committer. Candidates for new committers are typically people that are active contributors and community members.

+ +

Being an active community member means participating on mailing list discussions, helping to answer questions, being respectful towards others, and following the meritocratic principles of community management. Since the "Apache Way" has a strong focus on the project community, this part is very important.

+ +

Of course, contributing code to the project is important as well. A good way to start is contributing improvements, new features, or bugfixes. You need to show that you take responsibility for the code that you contribute, add tests/documentation, and help maintaining it.

+ +

Finally, candidates for new committers are suggested by current committers, mentors, or PMC members, and voted upon by the PMC.

+ +

How to use git as a committer

+ +

Only the infrastructure team of the ASF has administrative access to the GitHub mirror. Therefore, comitters have to push changes to the git repository at the ASF.

+ +

ASF writable git: https://git-wip-us.apache.org/repos/asf/incubator-flink.git

+ +

ASF read-only git: http://git-wip-us.apache.org/repos/asf/incubator-flink.git

+ +

ASF git web interface: https://git-wip-us.apache.org/repos/asf?p=incubator-flink.git;a=summary

+ +

Details on how to set the credentials for the ASF git repostiory are linked here. +To merge pull requests from our GitHub mirror, there is a script in the source ./tools/merge_pull_request.sh.template. Rename it to merge_pull_request.sh with the appropriate settings and use it for merging.

+ +
+
+ + + +
+ + + + + + + + Added: incubator/flink/site/img/ClientJmTm.svg URL: http://svn.apache.org/viewvc/incubator/flink/site/img/ClientJmTm.svg?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/ClientJmTm.svg ------------------------------------------------------------------------------ svn:mime-type = image/svg+xml Added: incubator/flink/site/img/apache-incubator-logo.png URL: http://svn.apache.org/viewvc/incubator/flink/site/img/apache-incubator-logo.png?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/apache-incubator-logo.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: incubator/flink/site/img/blog/emr-firefoxsettings.png URL: http://svn.apache.org/viewvc/incubator/flink/site/img/blog/emr-firefoxsettings.png?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/blog/emr-firefoxsettings.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: incubator/flink/site/img/blog/emr-hadoopversion.png URL: http://svn.apache.org/viewvc/incubator/flink/site/img/blog/emr-hadoopversion.png?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/blog/emr-hadoopversion.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: incubator/flink/site/img/blog/emr-jobmanager.png URL: http://svn.apache.org/viewvc/incubator/flink/site/img/blog/emr-jobmanager.png?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/blog/emr-jobmanager.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: incubator/flink/site/img/blog/emr-running.png URL: http://svn.apache.org/viewvc/incubator/flink/site/img/blog/emr-running.png?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/blog/emr-running.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: incubator/flink/site/img/blog/emr-security.png URL: http://svn.apache.org/viewvc/incubator/flink/site/img/blog/emr-security.png?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/blog/emr-security.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: incubator/flink/site/img/blog/emr-yarnappmaster.png URL: http://svn.apache.org/viewvc/incubator/flink/site/img/blog/emr-yarnappmaster.png?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/blog/emr-yarnappmaster.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: incubator/flink/site/img/blog/plan_visualizer1.png URL: http://svn.apache.org/viewvc/incubator/flink/site/img/blog/plan_visualizer1.png?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/blog/plan_visualizer1.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: incubator/flink/site/img/blog/plan_visualizer2.png URL: http://svn.apache.org/viewvc/incubator/flink/site/img/blog/plan_visualizer2.png?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/blog/plan_visualizer2.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: incubator/flink/site/img/blog/robomongo.png URL: http://svn.apache.org/viewvc/incubator/flink/site/img/blog/robomongo.png?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/blog/robomongo.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: incubator/flink/site/img/flink_stack.svg URL: http://svn.apache.org/viewvc/incubator/flink/site/img/flink_stack.svg?rev=1607832&view=auto ============================================================================== Binary file - no diff available. Propchange: incubator/flink/site/img/flink_stack.svg ------------------------------------------------------------------------------ svn:mime-type = image/svg+xml Modified: incubator/flink/site/index.html URL: http://svn.apache.org/viewvc/incubator/flink/site/index.html?rev=1607832&r1=1607831&r2=1607832&view=diff ============================================================================== --- incubator/flink/site/index.html (original) +++ incubator/flink/site/index.html Fri Jul 4 11:00:15 2014 @@ -1,34 +1,261 @@ - + + - - - - - Apache Flink - an open source system for expressive, declarative, fast, and efficient data analysis. - - - Apache Incubator -

Apache Flink

-

Flink is an open source system for expressive, declarative, fast, and efficient data analysis. Flink combines the - scalability and programming flexibility of distributed MapReduce-like platforms with the efficiency, out-of-core execution, - and query optimization capabilities found in parallel databases.

- -

Under construction

- - This website is currently under construction.
- You can find the documentation for the current release on our old website: stratosphere.eu
- The source code is located here: GitHub: apache/incubator-flink
- New mailing list: dev@flink.incubator.apache.org (subscribe | Archive) - -

Disclaimer

-

Apache Flink is an effort currently undergoing incubation at The Apache Software Foundation (ASF) sponsored by the - Apache Incubator PMC.

- - -
-

Copyright © 2014 Apache Software Foundation. All Rights Reserved.

-

Apache Flink, Apache, the Apache feather logo, and the Apache Flink project logos are trademarks of The Apache Software - Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.

-
+ + + + Apache Flink (incubating): Home + + + + + + + + +
+ +
+
+

Apache Flink is a platform for efficient, distributed, general-purpose data processing.

+ + + + +
+
+ Note +
+
+ Apache Flink originated from the Stratosphere project and is currently moving to the Apache Incubator. The first Apache release is under preparation - the latest stable release of the system is from the Stratosphere project. +
+
+ +
+ +

Project Overview

+ +

Flink features powerful programming abstractions in Java and Scala, a high-performance runtime, and automatic program optimization. It has native support for iterations, incremental iterations, and programs consisting of large DAGs of operations.

+ +
+

Concise and Expressive APIs

+
+
+
+

Flink allows you to express algorithms in a concise fashion in the programming languages Java and Scala. +Programs may freely compose many operations to long pipelines and mix and match built-in operations and +user-defined functions.

+ +

Flink programs have support for highly efficient iterative algorithms, allowing the system to model +complex tasks efficiently.

+ +
+
+
DataSet<String> input = env.readTextFile(inputPath);
+
+input.flatMap(new FlatMapFunction() {
+   public void flatMap(String value, Collector out) {
+       for (String s : value.split(" ")) {
+           out.collect(new Tuple2<String, Long>(s, 1L);
+       }
+   }
+})
+.groupBy(0)
+.sum(1)
+.writeAsText(outputPath);
+
+
+ +
+

System Stack

+
+
+
+

The Apache Flink stack consists of

+ +
    +
  • Programming APIs for different languages (Java, Scala) and +paradigms (record-oriented, graph-oriented).
  • +
  • A program optimizer that decides how to execute the program +for good performance. It decides among other things about data +movement and caching strategies.
  • +
  • A distributed runtime that executes programs in parallel +distributed over many machines.
  • +
+ +

Flink runs independently from Hadoop, but integrates +seamlessly with YARN (Hadoop's next-generation scheduler). +Various file systems (including the Hadoop Distributed +File System) can act as data sources.

+ +
+
+
+ Flink Stack +
+
+
+
+
+ + +
+ + + + + +