incubator-oodt-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mattm...@apache.org
Subject svn commit: r1023641 - in /incubator/oodt/trunk: CHANGES.txt pushpull/src/site/site.xml pushpull/src/site/xdoc/development/developer.xml pushpull/src/site/xdoc/user/advanced.xml pushpull/src/site/xdoc/user/basic.xml pushpull/src/site/xdoc/user/index.xml
Date Mon, 18 Oct 2010 03:17:10 GMT
Author: mattmann
Date: Mon Oct 18 03:17:08 2010
New Revision: 1023641

URL: http://svn.apache.org/viewvc?rev=1023641&view=rev
Log:
- OODT-49: wrapup: mirror new guide structure, stub in user guide and advanced user guides.
add in basic developer guide.

Added:
    incubator/oodt/trunk/pushpull/src/site/site.xml
    incubator/oodt/trunk/pushpull/src/site/xdoc/development/developer.xml
    incubator/oodt/trunk/pushpull/src/site/xdoc/user/advanced.xml
    incubator/oodt/trunk/pushpull/src/site/xdoc/user/basic.xml
    incubator/oodt/trunk/pushpull/src/site/xdoc/user/index.xml
Modified:
    incubator/oodt/trunk/CHANGES.txt

Modified: incubator/oodt/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/incubator/oodt/trunk/CHANGES.txt?rev=1023641&r1=1023640&r2=1023641&view=diff
==============================================================================
--- incubator/oodt/trunk/CHANGES.txt (original)
+++ incubator/oodt/trunk/CHANGES.txt Mon Oct 18 03:17:08 2010
@@ -5,6 +5,8 @@ Apache OODT Change Log
 Release 0.1-incubating - Current Development
 --------------------------------------------
 
+* OODT-49 Basic user guide for PushPull (mattmann)
+
 * OODT-22 remove pushpull's jvftp dependency (mattmann)
 
 * OODT-47 Web-grid admin pages reference jpl.eda instead of org.apache.oodt (mattmann)

Added: incubator/oodt/trunk/pushpull/src/site/site.xml
URL: http://svn.apache.org/viewvc/incubator/oodt/trunk/pushpull/src/site/site.xml?rev=1023641&view=auto
==============================================================================
--- incubator/oodt/trunk/pushpull/src/site/site.xml (added)
+++ incubator/oodt/trunk/pushpull/src/site/site.xml Mon Oct 18 03:17:08 2010
@@ -0,0 +1,32 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more contributor
+license agreements.  See the NOTICE.txt file distributed with this work for
+additional information regarding copyright ownership.  The ASF licenses this
+file to you under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License.  You may obtain a copy of
+the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
+License for the specific language governing permissions and limitations under
+the License.
+-->
+<project name="cas-pushpull">
+
+  <body>
+    <links>
+      <item name="OODT" href="../oodt-site/"/>
+    </links>
+
+    <menu ref="reports" inherit="bottom"/>
+    <menu name="Software Documentation">
+      <item name="Developer Guide" href="development/developer.html"/>
+      <item name="Basic User Guide" href="user/basic.html"/>
+      <item name="Advanced User Guide" href="user/advanced.html"/>
+    </menu>
+  </body>
+</project>

Added: incubator/oodt/trunk/pushpull/src/site/xdoc/development/developer.xml
URL: http://svn.apache.org/viewvc/incubator/oodt/trunk/pushpull/src/site/xdoc/development/developer.xml?rev=1023641&view=auto
==============================================================================
--- incubator/oodt/trunk/pushpull/src/site/xdoc/development/developer.xml (added)
+++ incubator/oodt/trunk/pushpull/src/site/xdoc/development/developer.xml Mon Oct 18 03:17:08
2010
@@ -0,0 +1,286 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more contributor
+license agreements.  See the NOTICE.txt file distributed with this work for
+additional information regarding copyright ownership.  The ASF licenses this
+file to you under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License.  You may obtain a copy of
+the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
+License for the specific language governing permissions and limitations under
+the License.
+-->
+<document>
+   <properties>
+      <title>CAS Push Pull Framework Developer Guide</title>
+      <author email="Brian.Foster@jpl.nasa.gov">Brian Foster</author>
+      <author email="Chris.Mattmann@jpl.nasa.gov">Chris Mattmann</author>
+   </properties>
+
+   <body>
+   
+      <section name="Introduction">
+        <p>
+          This is the developer guide for the Apache OODT Catalog and Archive Service (CAS)

+          Push Pull framework, or Push Pull for short. Primarily, this guide 
+          will explain the Push Pull architecture and interfaces, including its 
+          tailorable extension points. For information on installation, configuration,
+          and examples, please see our <a href="../user/basic.html">User Guides.</a>
+          
+          <p>The remainder of this guide is separated into the following sections:</p>
+      <ul>
+        <li><a href="#section1">Project Description</a></li>
+        <li><a href="#section2">Architecture</a></li>
+        <li><a href="#section3">Extension Points</a></li>
+        <li><a href="#section4">Current Extension Point Implementations</a></li>
+      </ul>
+          
+        </p>     
+      </section>
+      
+      <a name="section1"/>
+      <section name="Project Description">
+        <p>The Push Pull framework is responsible for downloading remote content (pull),
+        or accepting the delivery of remote content (push) to a local system staging area

+        for use by the <a href="../../crawler">CAS Crawler Framework</a> to ingest
into 
+        the <a href="../../filemgr">CAS File Manager</a>. The Push Pull framework
is extensible 
+        and provides a fully tailorable Java-based API for the acquisition of remote content.</p>

+      </section>
+      
+      <a name="section2"/>
+      <section name="Architecture">
+      
+        <p>In this section, we will describe the architecture of the Push Pull framework,
+        including its constituent components, object model, and key capabilities.</p>
+      
+        <subsection name="Components">
+        
+        <p>The major components of the Push Pull Framework are the Daemon Launcher,
the Daemon, 
+        the Protocol Layer, and the File Retrieval System, to name a few.  The relationship
between 
+        all of these components are shown in the diagram below:
+        </p>
+        
+        <p><img src="../images/pp_extension_points.png" alt="Push Pull Framework
Architecture"/></p>
+        
+        <p>The Push Pull Framework provides a Daemon Launcher, responsible for creating
new 
+        Daemon instances. Each Daemon has an associated Daemon Configuration, and has the

+        ability to use a File Retrieval Setup extension point. This class is responsible
for 
+        leveraging both a Protocol and a File Retrieval System to obtain ProtocolFiles, based
on 
+        a File Restrictions Parser, that yields eventually a VirtualFileStructure (VFS) model.
The 
+        VFS defines what files to accept and pull down from a remote site.
+        </p> 
+        
+        </subsection>
+        <subsection name="Object Model">
+        <p>The critical objects managed by the Push Pull Framework include:</p>
+        
+        <ul>
+          <li><strong>Protocol</strong> - A pluggable means of obtaining
content over 
+          some file acquisition method, e.g., FTP, SCP, HTTP, etc.</li>
+          
+          <li><strong>Protocol File</strong> - Metadata information about
a remote file, 
+          including its ProtocolPath.</li>
+          
+          <li><strong>Protocol Path</strong> - A pointer to a remote Product
file's (or files') 
+          location, which can be used to derive metadata and determine where to place the
file
+          in the local staging area built by the Push Pull Framework.</li>
+          
+          <li><strong>Remote Site</strong> - Descriptive information about
a remote site, including 
+          the username/password combination, as well as a origin directory to start interrogating.</li>
+        </ul>
+
+        <p>Each Protocol delivers one or more Protocol Files. Each ProtocoFile is associated
with a 
+        single RemoteSite, and each ProtocolFile is associated with a single ProtocolPath.
These 
+        relationships are shown in the below figure.</p>
+        
+        <img src="../images/pp_object_model.png" alt="Push Pull Framework Object Model"/>

+        </subsection> 
+      
+        <subsection name="Key Capabilities">
+        <p>The Push Pull Framework has been designed with a new of key capabilities
in mind.
+        These capabilities include:</p>
+        
+        <p><strong>Flexibility</strong> - ability to plug in different
Metadata Extractors, 
+        Data Protocols, Content Types, etc.</p>
+        
+        <p><strong>Support Push/Pull</strong> - Support of both &quot;Push&quot;
and &quot;Pull&quot; 
+        style data transfers.</p>
+        
+        <p><strong>Extensibility</strong> - ability to add new, previously
undiscovered Data Protocols, 
+        and &quot;plug&quot; them into the framework.</p>
+        
+        <p><strong>Java-based</strong> - Use of Java programming language
and development kit for Multi-Platform 
+        deployment (using the Java Virtual Machine).</p>
+        
+        <p><strong>Fast Data-transfer</strong> - Support of Parallel File
Transfers and Data Downloads.</p>
+        
+        <p><strong>Email-based Push</strong> - Support for Email-based
Push Data Acceptance using IMAP, SMTP protocols.</p>
+        
+        <p><strong>Modeling of remote data sites</strong> - Ability to
configure “Virtual” remote directories (based on Metadata 
+        such as Date/Time) to download files from.</p>
+        
+        <p><strong>Integration with other CAS components</strong> - Ability
to &quot;plug-in&quot; to the CAS File Management 
+        and CAS Crawl Framework components for Data Ingestion.</p>         
+        </subsection>
+      </section>
+      
+      <a name="section3"/>
+      <section name="Extension Points">
+          <p>We have constructed the Push Pull Framework making use of the <i>factory

+          method pattern</i> to provide multiple extension points for the Push Pull
Framework. An 
+          extension point is an interface within the Push Pull Framework that can have many

+          implementations. This is particularly useful when it comes to software 
+          component configuration because it allows different implementations of 
+          an existing interface to be selected at deployment time.</p> 
+          
+          <div class="info">The factory method pattern is a creational pattern common
to
+          object oriented design. Each Push Pull Framework extension point involves the 
+          implementation of two interfaces: an <i>extension factory</i> and an

+          <i>extension</i> implementation. At run-time, the Push Pull Framework
loads a 
+          properties file specifies a factory class to use during extension point
+          instantiation. For example, the Push Pull Framework may communicate with a 
+          remote FTP site to obtain content, or it may use an IMAPS protocol plugin to 
+          accept email-push notifications of available files.
+          </div> 
+          
+          <p>Using extension points, it is fairly simple to support many different
types 
+          of what are typically referred to as &quot;plug-in architectures&quot;.
Each of the core 
+          extension points for the Push Pull Framework is described below:</p>
+          
+          <table>
+            <tr>
+              <td>Protocol</td>
+              <td>The Protocol extension point is the heart of the Push Pull framework,

+              responsible for modeling remote sites, and for obtaining their content via

+              different Retrieval Methods, using different File Restrictions Parsers.
+              </td>
+            </tr>
+            <tr>
+              <td>Retrieval Method</td>
+              <td>The Retrieval Method extension point is responsible for orchestrating

+              download (pull) and acceptance (push) of remote content.
+              </td>
+            </tr>
+            <tr>
+              <td>File Restrictions Parser</td>
+              <td>The File Restrictions Parser extension point is responsible for defining

+              how to accept or decline files encountered by a Retrieval Method, in essence

+              modeling remote file and directory structures.
+              </td>
+            </tr>
+             <tr>
+              <td>System</td>
+              <td>The extension point that provides the external interface to the 
+              Push Pull Framework services. This includes the Daemon Launcher interface,

+              as well as the associated Daemon interface, that 
+              is managed by with the Daemon Launcher.
+              </td>
+            </tr>          
+          
+          </table>
+       </section>
+       
+       <a name="section4"/>   
+       <section name="Current Extension Point Implementations">
+        
+         <p>There are at least two implementations of all of the aforementioned 
+         extension points for the Push Pull Framework. Each extension point implementation

+         is detailed in this section.</p>
+
+         <subsection name="Protocol">
+            <ul>
+             <li><strong>Cog JGlobus FTP.</strong> An implementation of

+             the Protocol extension point for FTP using 
+             <a href="http://dev.globus.org/wiki/CoG_jglobus">CoG jglobus</a>.
+             </li>
+             
+             <li><strong>Commons Net FTP.</strong> An implementation of
the 
+             of the Protocol extension point for FTP using 
+             <a href="http://commons.apache.org/net/">Commons Net</a> FTP client.
+             </li>
+             
+             <li><strong>HTTP.</strong> An implementation of 
+             the Protocol extension point using Java's URL class, as well as 
+             <a href="http://tika.apche.org/">Apache Tika</a>'s HTMLParser.</li>
+             
+             <li><strong>IMAPS.</strong> An implementation of the 
+             Protocol extension point using IMAPS javax.mail classes from 
+             <a href="http://geronimo.apache.org">Apache Geronimo</a> and 
+             HTML parsing from <a href="http://tika.apache.org/">Apache Tika</a>.</li>
+
+             <li><strong>Local.</strong> An implementation of 
+             the Protocol extension point using Java NIO for local 
+             data acquisition.</li>
+             
+             <li><strong>SFTP.</strong> An implementation of the 
+             Protocol extension point using <a href="http://www.jcraft.org">JCraft</a>'s

+             <a href="http://www.jcraft.org/jsch">JSch</a> library.</li>
+           </ul>
+         </subsection> 
+
+         <subsection name="Retrieval Method">
+            <ul>
+             <li><strong>Remote Crawler.</strong> An implementation of

+             the Retrieval Method interface that uses an XML based set of 
+             policy files to determine which remote directories and files 
+             to crawl and obtain.</li>
+             
+             <li><strong>List Retriever.</strong> An implementation of
the 
+             Retrieval Method interface that accepts a list of URLs that point 
+             to content to obtain.</li>
+           </ul>
+         </subsection> 
+
+         <subsection name="File Restrictions Parser">
+            <ul>
+             <li><strong>DirStructXml Parser.</strong> An implementation
of 
+             the File Restrictions Parser interface that interprets an XML file 
+             specifying the remote directories and files to obtain.</li>
+             
+             <li><strong>FileList Parser.</strong> An implementation of
the 
+             File Restrictions Parser interface that specifies an ASCII newline separated

+             list of URLs pointing to remote directories and files to obtain.</li>
+
+             <li><strong>Class NOAA Email Parser.</strong> An implementation
of the 
+             File Restrictions Parser interface that reads email files from NOAA's CLASS

+             archive which specify lists of directory and file URLs to obtain.</li>
+           </ul>
+         </subsection>          
+         
+         
+         <subsection name="Daemon Launcher (Daemon client and Daemon server)">    
   
+             <ul>
+               <li><strong>Java RMI based server.</strong> An 
+               implementation of the external server interface for the Push Pull Framework
that 
+               uses RMI as the transportation medium to launch Push Pull Daemons.</li>
+               
+               <li><strong>Push Pull Daemon.</strong> An 
+               implementation of the client interface for the Java RMI-based  
+               server that uses RMI as the transportation medium to manage and control the

+               Push Pull services.</li>
+             </ul>
+         </subsection> 
+         
+      </section>
+      
+      <section name="Conclusion">
+      <p>The aim of this document is to provide information relevant to developers
+      about the CAS Push Pull Framework. Specifically, this document has described the 
+      Push Pull Framework's architecture, including its constituent components, object model
and
+      key capabilities. Additionally, the this document provides an overview of the 
+      current implementations of the Push Pull Framework's extension points.</p> 
+      
+      <p>In the <a href="../user/basic.html">Basic User Guide</a> and 
+      <a href="../user/advanced.html">Advanced User Guide</a>, we will cover
topics
+      like installation, configuration, and example uses as well as advanced topics
+      like scaling and other tips and tricks.</p> 
+      
+      </section>
+
+   </body>
+</document>

Added: incubator/oodt/trunk/pushpull/src/site/xdoc/user/advanced.xml
URL: http://svn.apache.org/viewvc/incubator/oodt/trunk/pushpull/src/site/xdoc/user/advanced.xml?rev=1023641&view=auto
==============================================================================
--- incubator/oodt/trunk/pushpull/src/site/xdoc/user/advanced.xml (added)
+++ incubator/oodt/trunk/pushpull/src/site/xdoc/user/advanced.xml Mon Oct 18 03:17:08 2010
@@ -0,0 +1,50 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more contributor
+license agreements.  See the NOTICE.txt file distributed with this work for
+additional information regarding copyright ownership.  The ASF licenses this
+file to you under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License.  You may obtain a copy of
+the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
+License for the specific language governing permissions and limitations under
+the License.
+-->
+<document>
+   <properties>
+      <title>CAS Push Pull User Guide</title>
+      <author email="Chris.Mattmann@jpl.nasa.gov">Chris Mattmann</author>
+      <author email="woollard@jpl.nasa.gov">Dave Woollard</author>
+   </properties>
+
+   <body>
+      <div class="warning">This document is in the process of being reorganized as
+      part of our ongoing documentation effort. Please check back for updates.</div>
 
+   
+      <section name="Introduction">
+      
+      <p>In the CAS Push Pull Framework Advanced User Guide, we will cover a number
of 
+      advanced topics related to using the system, including situations
+      that have arisen in the authors' experiences. Basic
+      concepts, such as installation and configuration, can be found in the 
+      <a href="../user/basic.html">Basic User Guide</a>.</p>
+      
+      <p>The remainder of this guide is separated into the following sections:</p>
+      <ul>
+        <li><a href="#section1">TBD</a></li>
+      </ul>
+      
+      </section>
+      
+      <a name="section1"/>
+      <section name="TBD">
+      <p>Coming Soon...</p>
+      </section>
+      
+   </body>
+</document> 

Added: incubator/oodt/trunk/pushpull/src/site/xdoc/user/basic.xml
URL: http://svn.apache.org/viewvc/incubator/oodt/trunk/pushpull/src/site/xdoc/user/basic.xml?rev=1023641&view=auto
==============================================================================
--- incubator/oodt/trunk/pushpull/src/site/xdoc/user/basic.xml (added)
+++ incubator/oodt/trunk/pushpull/src/site/xdoc/user/basic.xml Mon Oct 18 03:17:08 2010
@@ -0,0 +1,50 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more contributor
+license agreements.  See the NOTICE.txt file distributed with this work for
+additional information regarding copyright ownership.  The ASF licenses this
+file to you under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License.  You may obtain a copy of
+the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
+License for the specific language governing permissions and limitations under
+the License.
+-->
+<document>
+   <properties>
+      <title>CAS Push Pull User Guide</title>
+      <author email="Chris.Mattmann@jpl.nasa.gov">Chris Mattmann</author>
+      <author email="woollard@jpl.nasa.gov">Dave Woollard</author>
+   </properties>
+
+   <body>
+      <div class="warning">This document is in the process of being reorganized as
+      part of our ongoing documentation effort. Please check back for updates.</div>
+      
+      <section name="Introduction">
+      
+      <p>This document serves as a basic user's guide for the CAS Push Pull Framework
+      project. The goal of the document is to allow users to check out, build, and 
+      install a base version of the CAS Push Pull Framework as well as perform basic 
+      configuration tasks. For advanced topics, please see our 
+      <a href="../user/advanced.html">Advanced Guide.</a></p>
+      
+      <p>The remainder of this guide is separated into the following sections:</p>
+      <ul>
+        <li><a href="#section1">TBD</a></li>
+      </ul>
+      
+      </section>
+      
+      <a name="section1"/>
+      <section name="TBD">
+      <p>Coming Soon...</p>
+      </section>
+      
+   </body>
+</document> 

Added: incubator/oodt/trunk/pushpull/src/site/xdoc/user/index.xml
URL: http://svn.apache.org/viewvc/incubator/oodt/trunk/pushpull/src/site/xdoc/user/index.xml?rev=1023641&view=auto
==============================================================================
--- incubator/oodt/trunk/pushpull/src/site/xdoc/user/index.xml (added)
+++ incubator/oodt/trunk/pushpull/src/site/xdoc/user/index.xml Mon Oct 18 03:17:08 2010
@@ -0,0 +1,49 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more contributor
+license agreements.  See the NOTICE.txt file distributed with this work for
+additional information regarding copyright ownership.  The ASF licenses this
+file to you under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License.  You may obtain a copy of
+the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
+License for the specific language governing permissions and limitations under
+the License.
+-->
+<document>
+   <properties>
+      <title>CAS Push Pull Framework User Guide</title>
+      <author email="Chris.Mattmann@jpl.nasa.gov">Chris Mattmann</author>
+      <author email="woollard@jpl.nasa.gov">Dave Woollard</author>
+   </properties>
+
+   <body>
+  
+      <div class="warning">This document is in the process of being reorganized as
+      part of our ongoing documentation effort. Please follow the links below to find
+      information that has previously been hosted on this page.</div>
+ 
+
+      <p>We are in the process of upgrading OODT documentation. In the near
+      future, we hope to provide information for developers regarding each 
+      component, including its purpose, structure, interfaces, and 
+      configuration points in a <a href="../development/developer.html">Developer
+      Guide.</a></p>
+      
+      <p>Information for users, including installation and basic configuration,
+      as well as example usage are provided in a 
+      <a href="../user/basic.html">Basic User Guide.</a> Advanced topics, like
+      look and feel, lessons-learned, and best practices are provided in an
+      <a href="../user/advanced.html">Advanced User Guide.</a></p>
+      
+      <p>Finally, we are planning to provide a number of case studies in the 
+      future, in which multi-component deployments are configured to handle
+      common data and computation management scenarios.</p> 
+
+   </body>
+</document> 



Mime
View raw message