oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Starch" <starc...@umich.edu>
Subject Review Request 22791: Streaming OODT Changes
Date Thu, 19 Jun 2014 21:37:28 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/
-----------------------------------------------------------

Review request for oodt.


Repository: oodt


Description
-------

This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository.

There are four main portions:
   -Mesos Framework for Resource Manager (Prototype working)
   -Spark Runner for Workflow Manager (Prototype working)
   -Filemanager "streaming" type (In development)
   -Deployment and cluster management scripts (In development)

Where can this stuff be put so that it is available to use, even while it is in development?


Diffs
-----

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION

  http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/MaybeunusedMesosScheduler.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java
1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION

  http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/java/org/apache/oods/cas/streamer/system/MultiSourceStreamer.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/java/org/apache/oodt/cas/streamer/publisher/Publisher.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/java/org/apache/oodt/cas/streamer/reader/Reader.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/workflow/pom.xml 1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/engine/runner/SparkRunner.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/engine/runner/SparkRunnerFactory.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/examples/SparkHelloWorld.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/structs/WorkflowTaskSparkInstance.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/workflow/src/main/scala/org/apache/oodt/cas/workflow/examples/ScalaHelloWorld.scala
PRE-CREATION 

Diff: https://reviews.apache.org/r/22791/diff/


Testing
-------

Basic functionality tests done for both the resource-manger and workflow manager pieces. 
Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog
only.


Thanks,

Michael Starch


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message