Return-Path: X-Original-To: apmail-apex-commits-archive@minotaur.apache.org Delivered-To: apmail-apex-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5E07B19FB4 for ; Fri, 8 Apr 2016 01:10:28 +0000 (UTC) Received: (qmail 10245 invoked by uid 500); 8 Apr 2016 01:10:28 -0000 Delivered-To: apmail-apex-commits-archive@apex.apache.org Received: (qmail 10213 invoked by uid 500); 8 Apr 2016 01:10:28 -0000 Mailing-List: contact commits-help@apex.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@apex.incubator.apache.org Delivered-To: mailing list commits@apex.incubator.apache.org Received: (qmail 10204 invoked by uid 99); 8 Apr 2016 01:10:28 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2016 01:10:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id D5E42C0D35 for ; Fri, 8 Apr 2016 01:10:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.221 X-Spam-Level: X-Spam-Status: No, score=-3.221 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001] autolearn=disabled Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id TzWlJeIyDwxv for ; Fri, 8 Apr 2016 01:10:24 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with SMTP id 4BC785F1BE for ; Fri, 8 Apr 2016 01:10:24 +0000 (UTC) Received: (qmail 9112 invoked by uid 99); 8 Apr 2016 01:10:23 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2016 01:10:23 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 687E0E0415; Fri, 8 Apr 2016 01:10:23 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: sashap@apache.org To: commits@apex.incubator.apache.org Date: Fri, 08 Apr 2016 01:10:25 -0000 Message-Id: <118c5bf2b6d24955b7ec6677864e8d64@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [3/3] incubator-apex-site git commit: Adding apex-3.3 documentation Adding apex-3.3 documentation Project: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/commit/8c7220af Tree: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/tree/8c7220af Diff: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/diff/8c7220af Branch: refs/heads/asf-site Commit: 8c7220af70a844c02a4d05f951e5d9d536176b86 Parents: defd433 Author: sashadt Authored: Thu Apr 7 18:10:17 2016 -0700 Committer: sashadt Committed: Thu Apr 7 18:10:17 2016 -0700 ---------------------------------------------------------------------- docs/apex-3.3/apex_development_setup/index.html | 30 ++++++++- .../apex-3.3/application_development/index.html | 65 ++++++++++++++++---- docs/apex-3.3/application_packages/index.html | 4 +- docs/apex-3.3/index.html | 2 +- docs/apex-3.3/mkdocs/search_index.json | 44 ++++++++----- docs/apex-3.3/security/index.html | 10 +-- docs/apex-3.3/sitemap.xml | 18 +++--- 7 files changed, 126 insertions(+), 47 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/8c7220af/docs/apex-3.3/apex_development_setup/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/apex_development_setup/index.html b/docs/apex-3.3/apex_development_setup/index.html index 5033893..294f865 100644 --- a/docs/apex-3.3/apex_development_setup/index.html +++ b/docs/apex-3.3/apex_development_setup/index.html @@ -79,6 +79,8 @@
  • Creating New Apex Project
    • +
  • Running Unit Tests
    • +
  • Building Apex Demos
  • Sandbox
    • @@ -205,7 +207,7 @@

    After installing these tools, make sure that the directories containing the executable files are in your PATH environment variable.

      -
    • Windows - Open a console window and enter the command echo %PATH% to see the value of the PATH variable and verify that the above directories for Java, git, and maven executables are present. JDK executables like java and javac, the directory might be something like C:\\Program Files\\Java\\jdk1.7.0\_80\\bin; for git it might be C:\\Program Files\\Git\\bin; and for maven it might be C:\\Users\\user\\Software\\apache-maven-3.3.3\\bin. If not, you can change its value clicking on the button at Control Panel ⇨ Advanced System Settings ⇨ Advanced tab ⇨ Environment Variables.
      • +
    • Windows - Open a console window and enter the command echo %PATH% to see the value of the PATH variable and verify that the above directories for Java, git, and maven executables are present. JDK executables like java and javac, the directory might be something like C:\Program Files\Java\jdk1.7.0\_80\bin; for git it might be C:\Program Files\Git\bin; and for maven it might be C:\Users\user\Software\apache-maven-3.3.3\bin. If not, you can change its value clicking on the button at Control Panel ⇨ Advanced System Settings ⇨ Advanced tab ⇨ Environment Variables.
    • Linux and Mac - Open a console/terminal window and enter the command echo $PATH to see the value of the PATH variable and verify that the above directories for Java, git, and maven executables are present. If not, make sure software is downloaded and installed, and optionally PATH reference is added and exported in a ~/.profile or ~/.bash_profile. For example to add maven located in /sfw/maven/apache-maven-3.3.3 to PATH add the line: export PATH=$PATH:/sfw/maven/apache-maven-3.3.3/bin

    Confirm by running the following commands and comparing with output that show in the table below:

    @@ -273,6 +275,32 @@ endlocal mvn clean package -DskipTests

    The build should create the application package file myapexapp/target/myapexapp-1.0-SNAPSHOT.apa. This application package can then be used to launch example application via dtCli, or other visual management tools. When running, this application will generate a stream of random numbers and print them out, each prefixed by the string hello world:.

    +

    Running Unit Tests

    +

    To run unit tests on Linux or OSX, simply run the usual maven command, for example: mvn test.

    +

    On Windows, an additional file, winutils.exe, is required; download it from +https://github.com/srccodes/hadoop-common-2.2.0-bin/archive/master.zip +and unpack the archive to, say, C:\hadoop; this file should be present under +hadoop-common-2.2.0-bin-master\bin within it.

    +

    Set the HADOOP_HOME environment variable system-wide to +c:\hadoop\hadoop-common-2.2.0-bin-master as described at: +https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/sysdm_advancd_environmnt_addchange_variable.mspx?mfr=true. You should now be able to run unit tests normally.

    +

    If you prefer not to set the variable globally, you can set it on the command line or within +your IDE. For example, on the command line, specify the maven +property hadoop.home.dir:

    +
    mvn -Dhadoop.home.dir=c:\hadoop\hadoop-common-2.2.0-bin-master test
+
    +

    or set the environment variable separately:

    +
    set HADOOP_HOME=c:\hadoop\hadoop-common-2.2.0-bin-master
+mvn test
+
    +

    Within your IDE, set the environment variable and then run the desired +unit test in the usual way. For example, with NetBeans you can add:

    +
    Env.HADOOP_HOME=c:/hadoop/hadoop-common-2.2.0-bin-master
+
    +

    at Properties ⇒ Actions ⇒ Run project ⇒ Set Properties.

    +

    Similarly, in Eclipse (Mars) add it to the +project properties at Properties ⇒ Run/Debug Settings ⇒ ApplicationTest +⇒ Environment tab.

    Building Apex Demos

    If you want to see more substantial Apex demo applications and the associated source code, you can follow these simple steps to check out and build them.

      http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/8c7220af/docs/apex-3.3/application_development/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/application_development/index.html b/docs/apex-3.3/application_development/index.html index b7b3faa..802a2c7 100644 --- a/docs/apex-3.3/application_development/index.html +++ b/docs/apex-3.3/application_development/index.html @@ -109,6 +109,8 @@
    1. Application API
      2. +
    2. JSON File DAG Specification
      3. +
    3. Operators
    4. Streams
      5. @@ -461,8 +463,7 @@ public class StockTickInput implements InputOperator int statusCode = client.executeMethod(method); if (statusCode != HttpStatus.SC_OK) { System.err.println("Method failed: " + method.getStatusLine()); - } - else { + } else { InputStream istream = method.getResponseBodyAsStream(); // Process response InputStreamReader isr = new InputStreamReader(istream); @@ -493,11 +494,9 @@ public class StockTickInput implements InputOperator } } Thread.sleep(readIntervalMillis); - } - catch (InterruptedException ex) { + } catch (InterruptedException ex) { logger.debug(ex.toString()); - } - catch (IOException ex) { + } catch (IOException ex) { logger.debug(ex.toString()); } } @@ -514,7 +513,7 @@ public class StockTickInput implements InputOperator public void setOutputEvenIfZeroVolume(boolean outputEvenIfZeroVolume) { - this.outputEvenIfZeroVolume = outputEvenIfZeroVolume; + this.outputEvenIfZeroVolume = outputEvenIfZeroVolume; } } @@ -1139,7 +1138,7 @@ libraries, version of the platform etc.)

      killing, viewing, etc.). This tool was already discussed above briefly in the section entitled Running the Test Application. It will introspect the jar file specified with the launch command for applications (classes -that implement ApplicationFactory) or property files that define +that implement ApplicationFactory) or properties files that define applications. It will also deploy the dependency jar files from the application package to the cluster.

      Dtcli can run the application in local mode (i.e. outside a @@ -1240,8 +1239,49 @@ DAG in local mode within the IDE.

      } -

      Property File API

      -

      The platform also supports specification of a DAG via a property +

      JSON File DAG Specification

      +

      In addition to Java, you can also specify the DAG using JSON, provided the operators in the DAG are present in the dependency jars. Create src/main/resources/app directory under your app package project, and put your JSON files there. This is the specification of a JSON file that specifies an application.

      +

      Create a json file under src/main/resources/app, For example myApplication.json

      +
      {
+  "description": "{application description}",
+  "operators": [
+    {
+      "name": "{operator name}",
+      "class": "{fully qualified class name of the operator}",
+      "properties": {
+        "{property key}": "{property value}",
+        ...
+      }
+    }, ...
+  ],
+  "streams": [
+    {
+      "name": "{stream name}",
+      "source": {
+        "operatorName": "{source operator name}",
+        "portName": "{source operator output port name}"
+      }
+      "sinks": [
+        {
+          "operatorName": "{sink operator name}",
+          "portName": "{sink operator input port name}"
+        }, ...
+      ]
+    }, ...
+  ]
+}
+
+
      + +
        +
      • The name of the JSON file is taken as the name of the application.
        • +
      • The description field is the description of the application and is optional.
        • +
      • The operators field is the list of operators the application has. You can specifiy the name, the Java class, and the properties of each operator here.
        • +
      • The streams field is the list of streams that connects the operators together to form the DAG. Each stream consists of the stream name, the operator and port that it connects from, and the list of operators and ports that it connects to. Note that you can connect from one output port of an operator to multiple different input ports of different operators.
        • +
      +

      In Apex Malhar, there is an example in the Pi Demo doing just that.

      +

      Properties File DAG Specification

      +

      The platform also supports specification of a DAG via a properties file. The aim here to make it easy for tools to create and run an application. This method of specification does not have the Java compiler support of compile time check, but since these applications @@ -1249,7 +1289,8 @@ would be created by software, they should be correct by construction. The syntax is derived from Hadoop properties and should be easy for folks who are used to creating software that integrated with Hadoop.

      -

      Create an application (DAG): myApplication.properties

      +

      Under the src/main/resources/app directory (create if it doesn't exist), create a properties file. +For example myApplication.properties

      # input operator that reads from a file
 dt.operator.inputOp.classname=com.acme.SampleInputOperator
 dt.operator.inputOp.fileName=somefile.txt
@@ -1263,7 +1304,7 @@ dt.stream.inputStream.sinks=outputOp.inputPort

      Above snippet is intended to convey the basic idea of specifying -the DAG without using Java. Operators would come from a predefined +the DAG using properties file. Operators would come from a predefined library and referenced in the specification by class name and port names (obtained from the library providers documentation or runtime introspection by tools). For those interested in details, see later http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/8c7220af/docs/apex-3.3/application_packages/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/application_packages/index.html b/docs/apex-3.3/application_packages/index.html index cd0eea7..faae829 100644 --- a/docs/apex-3.3/application_packages/index.html +++ b/docs/apex-3.3/application_packages/index.html @@ -271,7 +271,7 @@ Apex installation.

      Artifact ID: apex-app-archetype Version: 3.2.0-incubating (or any later version)

      Writing Your Own App Package

      -

      Please refer to the Creating Apps on the basics on how to write an Apache Apex application. In your AppPackage project, you can add custom operators (refer to Operator Development Guide, project dependencies, default and required configuration properties, pre-set configurations and other metadata.

      +

      Please refer to the [Application Developer Guide][application_development.md] on the basics on how to write an Apache Apex application. In your AppPackage project, you can add custom operators (refer to Operator Development Guide, project dependencies, default and required configuration properties, pre-set configurations and other metadata. Note that you can also specify the DAG using Java, JSON or properties files.

      Adding (and removing) project dependencies

      Under the project, you can add project dependencies in pom.xml, or do it through your IDE. Here’s the section that describes the dependencies in @@ -595,7 +595,7 @@ that file is only used for the unit test.

      Apache Apex Application Package files are zip files. You can examine the content of any Application Package by using unzip -t on your Linux command line.

      There are four top level directories in an Application Package:

        -
      1. "app" contains the jar files of the DAG code and any custom operators.
        2. +
      2. "app" contains the jar files of the DAG code and any custom operators, and any JSON or properties files that specify a DAG.
      3. "lib" contains all dependency jars
      4. "conf" contains all the pre-set configuration XML files.
      5. "META-INF" contains the MANIFEST.MF file and the properties.xml file.
        6. http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/8c7220af/docs/apex-3.3/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/index.html b/docs/apex-3.3/index.html index 9d9b251..795c645 100644 --- a/docs/apex-3.3/index.html +++ b/docs/apex-3.3/index.html @@ -232,5 +232,5 @@