Return-Path: X-Original-To: apmail-oodt-dev-archive@www.apache.org Delivered-To: apmail-oodt-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4B9E111D0D for ; Thu, 4 Sep 2014 17:39:07 +0000 (UTC) Received: (qmail 58395 invoked by uid 500); 4 Sep 2014 17:39:07 -0000 Delivered-To: apmail-oodt-dev-archive@oodt.apache.org Received: (qmail 58357 invoked by uid 500); 4 Sep 2014 17:39:07 -0000 Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@oodt.apache.org Delivered-To: mailing list dev@oodt.apache.org Delivered-To: moderator for dev@oodt.apache.org Received: (qmail 57155 invoked by uid 99); 4 Sep 2014 17:37:53 -0000 Content-Type: multipart/alternative; boundary="===============1609206431078821596==" MIME-Version: 1.0 Subject: Re: Review Request 22791: Streaming OODT Changes From: "Chris Mattmann" To: "Lewis McGibbney" , "Chris Mattmann" Cc: "Michael Starch" , "oodt" Date: Thu, 04 Sep 2014 17:37:52 -0000 Message-ID: <20140904173752.16961.46697@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Chris Mattmann" X-ReviewGroup: oodt X-ReviewRequest-URL: https://reviews.apache.org/r/22791/ X-Sender: "Chris Mattmann" References: <20140904172312.16960.6571@reviews.apache.org> In-Reply-To: <20140904172312.16960.6571@reviews.apache.org> Reply-To: "Chris Mattmann" X-ReviewRequest-Repository: oodt --===============1609206431078821596== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22791/#review52324 ----------------------------------------------------------- http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh Mike, instead of directly including cluster-tools in oodt, can you simply make: # A Maven AntRun script (build.xml) or something that downloads the cluster-tools as part of the Resource Manager build? We shouldn't have to maintain these scripts in OODT. http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml Custom Maven repos are difficult since the Central repository is phasing them out: http://blog.sonatype.com/2010/03/why-external-repos-are-being-phased-out-of-central/ If we absolutely need to ref this repo, can you make it a Maven profile, not enabled by default, so that users won't have the issue for this when downloading and building OODT? http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java how about instead of NA, for the filename, we call it a "stream-UUID" where we generate a unique stream UUID as the FILENAME field. http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java how about a similarly derived STREAM loc here, perhaps the same as the FILENAME suggestion using UUID? http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java rather than silently do something, maybe take this out until there is something to do? http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java not sure what this change is? http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java All of these files need ALv2 license headers. http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java We shoudln't be checking in generated files - can this simply be generated during the Resmgr build? http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml if we don't need this, remove it. http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak ALv2 headers here and everywhere http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties remove JPL specific URLs. - Chris Mattmann On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22791/ > ----------------------------------------------------------- > > (Updated Sept. 4, 2014, 5:23 p.m.) > > > Review request for oodt, Lewis McGibbney and Chris Mattmann. > > > Repository: oodt > > > Description > ------- > > This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository. > > There are four main portions: > -Mesos Framework for Resource Manager (Prototype working) > -Spark Runner for Workflow Manager (Prototype working) > -Filemanager "streaming" type (In development) > -Deployment and cluster management scripts (In development) > > Where can this stuff be put so that it is available to use, even while it is in development? > > > Diffs > ----- > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION > > Diff: https://reviews.apache.org/r/22791/diff/ > > > Testing > ------- > > Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only. > > > Thanks, > > Michael Starch > > --===============1609206431078821596==--