apex-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sas...@apache.org
Subject [3/3] incubator-apex-site git commit: Adding apex-3.3 documentation
Date Fri, 08 Apr 2016 01:10:25 GMT
Adding apex-3.3 documentation


Project: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/commit/8c7220af
Tree: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/tree/8c7220af
Diff: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/diff/8c7220af

Branch: refs/heads/asf-site
Commit: 8c7220af70a844c02a4d05f951e5d9d536176b86
Parents: defd433
Author: sashadt <sasha@datatorrent.com>
Authored: Thu Apr 7 18:10:17 2016 -0700
Committer: sashadt <sasha@datatorrent.com>
Committed: Thu Apr 7 18:10:17 2016 -0700

----------------------------------------------------------------------
 docs/apex-3.3/apex_development_setup/index.html | 30 ++++++++-
 .../apex-3.3/application_development/index.html | 65 ++++++++++++++++----
 docs/apex-3.3/application_packages/index.html   |  4 +-
 docs/apex-3.3/index.html                        |  2 +-
 docs/apex-3.3/mkdocs/search_index.json          | 44 ++++++++-----
 docs/apex-3.3/security/index.html               | 10 +--
 docs/apex-3.3/sitemap.xml                       | 18 +++---
 7 files changed, 126 insertions(+), 47 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/8c7220af/docs/apex-3.3/apex_development_setup/index.html
----------------------------------------------------------------------
diff --git a/docs/apex-3.3/apex_development_setup/index.html b/docs/apex-3.3/apex_development_setup/index.html
index 5033893..294f865 100644
--- a/docs/apex-3.3/apex_development_setup/index.html
+++ b/docs/apex-3.3/apex_development_setup/index.html
@@ -79,6 +79,8 @@
                 
                     <li><a class="toctree-l4" href="#creating-new-apex-project">Creating
New Apex Project</a></li>
                 
+                    <li><a class="toctree-l4" href="#running-unit-tests">Running
Unit Tests</a></li>
+                
                     <li><a class="toctree-l4" href="#building-apex-demos">Building
Apex Demos</a></li>
                 
                     <li><a class="toctree-l4" href="#sandbox">Sandbox</a></li>
@@ -205,7 +207,7 @@
 </ol>
 <p>After installing these tools, make sure that the directories containing the executable
files are in your PATH environment variable.</p>
 <ul>
-<li><strong>Windows</strong> - Open a console window and enter the command
<code>echo %PATH%</code> to see the value of the <code>PATH</code>
variable and verify that the above directories for Java, git, and maven executables are present.
 JDK executables like <em>java</em> and <em>javac</em>, the directory
might be something like <code>C:\\Program Files\\Java\\jdk1.7.0\_80\\bin</code>;
for <em>git</em> it might be <code>C:\\Program Files\\Git\\bin</code>;
and for maven it might be <code>C:\\Users\\user\\Software\\apache-maven-3.3.3\\bin</code>.
 If not, you can change its value clicking on the button at <em>Control Panel</em>
&#x21e8; <em>Advanced System Settings</em> &#x21e8; <em>Advanced
tab</em> &#x21e8; <em>Environment Variables</em>.</li>
+<li><strong>Windows</strong> - Open a console window and enter the command
<code>echo %PATH%</code> to see the value of the <code>PATH</code>
variable and verify that the above directories for Java, git, and maven executables are present.
 JDK executables like <em>java</em> and <em>javac</em>, the directory
might be something like <code>C:\Program Files\Java\jdk1.7.0\_80\bin</code>; for
<em>git</em> it might be <code>C:\Program Files\Git\bin</code>; and
for maven it might be <code>C:\Users\user\Software\apache-maven-3.3.3\bin</code>.
 If not, you can change its value clicking on the button at <em>Control Panel</em>
&#x21e8; <em>Advanced System Settings</em> &#x21e8; <em>Advanced
tab</em> &#x21e8; <em>Environment Variables</em>.</li>
 <li><strong>Linux and Mac</strong> - Open a console/terminal window and
enter the command <code>echo $PATH</code> to see the value of the <code>PATH</code>
variable and verify that the above directories for Java, git, and maven executables are present.
 If not, make sure software is downloaded and installed, and optionally PATH reference is
added and exported  in a <code>~/.profile</code> or <code>~/.bash_profile</code>.
 For example to add maven located in <code>/sfw/maven/apache-maven-3.3.3</code>
to PATH add the line: <code>export PATH=$PATH:/sfw/maven/apache-maven-3.3.3/bin</code></li>
 </ul>
 <p>Confirm by running the following commands and comparing with output that show in
the table below:</p>
@@ -273,6 +275,32 @@ endlocal
 mvn clean package -DskipTests
 </code></pre>
 <p>The build should create the application package file <code>myapexapp/target/myapexapp-1.0-SNAPSHOT.apa</code>.
This application package can then be used to launch example application via <strong>dtCli</strong>,
or other visual management tools.  When running, this application will generate a stream of
random numbers and print them out, each prefixed by the string <code>hello world:</code>.</p>
+<h2 id="running-unit-tests">Running Unit Tests</h2>
+<p>To run unit tests on Linux or OSX, simply run the usual maven command, for example:
<code>mvn test</code>.</p>
+<p>On Windows, an additional file, <code>winutils.exe</code>, is required;
download it from
+<a href="https://github.com/srccodes/hadoop-common-2.2.0-bin/archive/master.zip">https://github.com/srccodes/hadoop-common-2.2.0-bin/archive/master.zip</a>
+and unpack the archive to, say, <code>C:\hadoop</code>; this file should be present
under
+<code>hadoop-common-2.2.0-bin-master\bin</code> within it.</p>
+<p>Set the <code>HADOOP_HOME</code> environment variable system-wide to
+<code>c:\hadoop\hadoop-common-2.2.0-bin-master</code> as described at:
+<a href="https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/sysdm_advancd_environmnt_addchange_variable.mspx?mfr=true">https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/sysdm_advancd_environmnt_addchange_variable.mspx?mfr=true</a>.
You should now be able to run unit tests normally.</p>
+<p>If you prefer not to set the variable globally, you can set it on the command line
or within
+your IDE. For example, on the command line, specify the maven
+property <code>hadoop.home.dir</code>:</p>
+<pre><code>mvn -Dhadoop.home.dir=c:\hadoop\hadoop-common-2.2.0-bin-master test
+</code></pre>
+<p>or set the environment variable separately:</p>
+<pre><code>set HADOOP_HOME=c:\hadoop\hadoop-common-2.2.0-bin-master
+mvn test
+</code></pre>
+<p>Within your IDE, set the environment variable and then run the desired
+unit test in the usual way. For example, with NetBeans you can add:</p>
+<pre><code>Env.HADOOP_HOME=c:/hadoop/hadoop-common-2.2.0-bin-master
+</code></pre>
+<p>at <em>Properties &#8658; Actions &#8658; Run project &#8658;
Set Properties</em>.</p>
+<p>Similarly, in Eclipse (Mars) add it to the
+project properties at <em>Properties &#8658; Run/Debug Settings &#8658; ApplicationTest
+&#8658; Environment</em> tab.</p>
 <h2 id="building-apex-demos">Building Apex Demos</h2>
 <p>If you want to see more substantial Apex demo applications and the associated source
code, you can follow these simple steps to check out and build them.</p>
 <ol>

http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/8c7220af/docs/apex-3.3/application_development/index.html
----------------------------------------------------------------------
diff --git a/docs/apex-3.3/application_development/index.html b/docs/apex-3.3/application_development/index.html
index b7b3faa..802a2c7 100644
--- a/docs/apex-3.3/application_development/index.html
+++ b/docs/apex-3.3/application_development/index.html
@@ -109,6 +109,8 @@
                 
                     <li><a class="toctree-l4" href="#application-api">Application
API</a></li>
                 
+                    <li><a class="toctree-l4" href="#json-file-dag-specification">JSON
File DAG Specification</a></li>
+                
                     <li><a class="toctree-l4" href="#operators">Operators</a></li>
                 
                     <li><a class="toctree-l4" href="#streams">Streams</a></li>
@@ -461,8 +463,7 @@ public class StockTickInput implements InputOperator
       int statusCode = client.executeMethod(method);
       if (statusCode != HttpStatus.SC_OK) {
         System.err.println(&quot;Method failed: &quot; + method.getStatusLine());
-      }
-      else {
+      } else {
         InputStream istream = method.getResponseBodyAsStream();
         // Process response
         InputStreamReader isr = new InputStreamReader(istream);
@@ -493,11 +494,9 @@ public class StockTickInput implements InputOperator
         }
       }
       Thread.sleep(readIntervalMillis);
-    }
-    catch (InterruptedException ex) {
+    } catch (InterruptedException ex) {
       logger.debug(ex.toString());
-    }
-    catch (IOException ex) {
+    } catch (IOException ex) {
       logger.debug(ex.toString());
     }
   }
@@ -514,7 +513,7 @@ public class StockTickInput implements InputOperator
 
   public void setOutputEvenIfZeroVolume(boolean outputEvenIfZeroVolume)
   {
-       this.outputEvenIfZeroVolume = outputEvenIfZeroVolume;
+    this.outputEvenIfZeroVolume = outputEvenIfZeroVolume;
   }
 
 }
@@ -1139,7 +1138,7 @@ libraries, version of the platform etc.)</p>
 killing, viewing, etc.). This tool was already discussed above briefly
 in the section entitled Running the Test Application. It will introspect
 the jar file specified with the launch command for applications (classes
-that implement ApplicationFactory) or property files that define
+that implement ApplicationFactory) or properties files that define
 applications. It will also deploy the dependency jar files from the
 application package to the cluster.</p>
 <p>Dtcli can run the application in local mode (i.e. outside a
@@ -1240,8 +1239,49 @@ DAG in local mode within the IDE.</p>
 }
 </code></pre>
 
-<h3 id="property-file-api">Property File API</h3>
-<p>The platform also supports specification of a DAG via a property
+<h2 id="json-file-dag-specification">JSON File DAG Specification</h2>
+<p>In addition to Java, you can also specify the DAG using JSON, provided the operators
in the DAG are present in the dependency jars. Create src/main/resources/app directory under
your app package project, and put your JSON files there. This is the specification of a JSON
file that specifies an application.</p>
+<p>Create a json file under src/main/resources/app, For example <code>myApplication.json</code></p>
+<pre><code>{
+  &quot;description&quot;: &quot;{application description}&quot;,
+  &quot;operators&quot;: [
+    {
+      &quot;name&quot;: &quot;{operator name}&quot;,
+      &quot;class&quot;: &quot;{fully qualified class name of the operator}&quot;,
+      &quot;properties&quot;: {
+        &quot;{property key}&quot;: &quot;{property value}&quot;,
+        ...
+      }
+    }, ...
+  ],
+  &quot;streams&quot;: [
+    {
+      &quot;name&quot;: &quot;{stream name}&quot;,
+      &quot;source&quot;: {
+        &quot;operatorName&quot;: &quot;{source operator name}&quot;,
+        &quot;portName&quot;: &quot;{source operator output port name}&quot;
+      }
+      &quot;sinks&quot;: [
+        {
+          &quot;operatorName&quot;: &quot;{sink operator name}&quot;,
+          &quot;portName&quot;: &quot;{sink operator input port name}&quot;
+        }, ...
+      ]
+    }, ...
+  ]
+}
+
+</code></pre>
+
+<ul>
+<li>The name of the JSON file is taken as the name of the application.</li>
+<li>The <code>description</code> field is the description of the application
and is optional.</li>
+<li>The <code>operators</code> field is the list of operators the application
has. You can specifiy the name, the Java class, and the properties of each operator here.</li>
+<li>The <code>streams</code> field is the list of streams that connects
the operators together to form the DAG. Each stream consists of the stream name, the operator
and port that it connects from, and the list of operators and ports that it connects to. Note
that you can connect from <em>one</em> output port of an operator to <em>multiple</em>
different input ports of different operators.</li>
+</ul>
+<p>In Apex Malhar, there is an <a href="https://github.com/apache/incubator-apex-malhar/blob/master/demos/pi/src/main/resources/app/PiJsonDemo.json">example</a>
in the Pi Demo doing just that.</p>
+<h3 id="properties-file-dag-specification">Properties File DAG Specification</h3>
+<p>The platform also supports specification of a DAG via a properties
 file. The aim here to make it easy for tools to create and run an
 application. This method of specification does not have the Java
 compiler support of compile time check, but since these applications
@@ -1249,7 +1289,8 @@ would be created by software, they should be correct by construction.
 The syntax is derived from Hadoop properties and should be easy for
 folks who are used to creating software that integrated with
 Hadoop.</p>
-<p>Create an application (DAG): myApplication.properties</p>
+<p>Under the src/main/resources/app directory (create if it doesn't exist), create
a properties file.
+For example <code>myApplication.properties</code></p>
 <pre><code># input operator that reads from a file
 dt.operator.inputOp.classname=com.acme.SampleInputOperator
 dt.operator.inputOp.fileName=somefile.txt
@@ -1263,7 +1304,7 @@ dt.stream.inputStream.sinks=outputOp.inputPort
 </code></pre>
 
 <p>Above snippet is intended to convey the basic idea of specifying
-the DAG without using Java. Operators would come from a predefined
+the DAG using properties file. Operators would come from a predefined
 library and referenced in the specification by class name and port names
 (obtained from the library providers documentation or runtime
 introspection by tools). For those interested in details, see later

http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/8c7220af/docs/apex-3.3/application_packages/index.html
----------------------------------------------------------------------
diff --git a/docs/apex-3.3/application_packages/index.html b/docs/apex-3.3/application_packages/index.html
index cd0eea7..faae829 100644
--- a/docs/apex-3.3/application_packages/index.html
+++ b/docs/apex-3.3/application_packages/index.html
@@ -271,7 +271,7 @@ Apex installation.</p>
 Artifact ID: apex-app-archetype
 Version: 3.2.0-incubating (or any later version)</p>
 <h2 id="writing-your-own-app-package">Writing Your Own App Package</h2>
-<p>Please refer to the <a href="http://docs.datatorrent.com/create/">Creating
Apps</a> on the basics on how to write an Apache Apex application.  In your AppPackage
project, you can add custom operators (refer to <a href="../operator_development/">Operator
Development Guide</a>, project dependencies, default and required configuration properties,
pre-set configurations and other metadata.</p>
+<p>Please refer to the [Application Developer Guide][application_development.md] on
the basics on how to write an Apache Apex application. In your AppPackage project, you can
add custom operators (refer to <a href="../operator_development/">Operator Development
Guide</a>, project dependencies, default and required configuration properties, pre-set
configurations and other metadata. Note that you can also specify the DAG using Java, JSON
or properties files. </p>
 <h3 id="adding-and-removing-project-dependencies">Adding (and removing) project dependencies</h3>
 <p>Under the project, you can add project dependencies in pom.xml, or do it
 through your IDE.  Here’s the section that describes the dependencies in
@@ -595,7 +595,7 @@ that file is only used for the unit test.</p>
 <p>Apache Apex Application Package files are zip files.  You can examine the content
of any Application Package by using unzip -t on your Linux command line.</p>
 <p>There are four top level directories in an Application Package:</p>
 <ol>
-<li>"app" contains the jar files of the DAG code and any custom operators.</li>
+<li>"app" contains the jar files of the DAG code and any custom operators, and any
JSON or properties files that specify a DAG.</li>
 <li>"lib" contains all dependency jars</li>
 <li>"conf" contains all the pre-set configuration XML files.</li>
 <li>"META-INF" contains the MANIFEST.MF file and the properties.xml file.</li>

http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/8c7220af/docs/apex-3.3/index.html
----------------------------------------------------------------------
diff --git a/docs/apex-3.3/index.html b/docs/apex-3.3/index.html
index 9d9b251..795c645 100644
--- a/docs/apex-3.3/index.html
+++ b/docs/apex-3.3/index.html
@@ -232,5 +232,5 @@
 
 <!--
 MkDocs version : 0.15.3
-Build Date UTC : 2016-03-22 01:44:26.357028
+Build Date UTC : 2016-04-08 01:01:16.151593
 -->


Mime
View raw message