Return-Path: X-Original-To: apmail-oodt-dev-archive@www.apache.org Delivered-To: apmail-oodt-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E129DCDFA for ; Sat, 9 Aug 2014 20:57:23 +0000 (UTC) Received: (qmail 72701 invoked by uid 500); 9 Aug 2014 20:57:23 -0000 Delivered-To: apmail-oodt-dev-archive@oodt.apache.org Received: (qmail 72662 invoked by uid 500); 9 Aug 2014 20:57:23 -0000 Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@oodt.apache.org Delivered-To: mailing list dev@oodt.apache.org Delivered-To: moderator for dev@oodt.apache.org Received: (qmail 69119 invoked by uid 99); 9 Aug 2014 20:53:25 -0000 Content-Type: multipart/alternative; boundary="===============5258580233895648610==" MIME-Version: 1.0 Subject: Review Request 24529: CAS-PGE no longer respects writers and file tags from earlier pgeConfig.xml files From: "Chris Mattmann" To: "oodt" , "Chris Mattmann" Date: Sat, 09 Aug 2014 20:53:11 -0000 Message-ID: <20140809205311.1587.68205@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Chris Mattmann" X-ReviewGroup: oodt X-ReviewRequest-URL: https://reviews.apache.org/r/24529/ X-Sender: "Chris Mattmann" Reply-To: "Chris Mattmann" X-ReviewRequest-Repository: oodt --===============5258580233895648610== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24529/ ----------------------------------------------------------- Review request for oodt. Bugs: OODT-667 https://issues.apache.org/jira/browse/OODT-667 Repository: oodt Description ------- This patch restores functionality and fixes CAS-PGE in trunk for users expecting 0.3 CAS-PGE and before style configuration in which you didn't need a MIME extractor repo and could configure CAS-PGE entirely from the pge-config.xml files. This patch effectively makes CAS-PGE usable again in trunk for 0.7 and going forward and is fully forward compatible with Brian's changes. If you specify a MIME extractor repo, you get an AutoDetectCrawler - otherwise you get the familiar StdProductCrawler. Diffs ----- ./trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/filenaming/PathUtilsNamingConvention.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeMetadata.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskStatus.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/GenericPgeObjectFactory.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/XmlHelper.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/CsvConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/DynamicConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/MetadataKeyReplacerTemplateWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/TextConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/VelocityConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/XslTransformWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java PRE-CREATION ./trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1616402 ./trunk/pge/src/test/java/org/apache/oodt/cas/pge/staging/TestFileStager.java 1616402 ./trunk/pge/src/test/java/org/apache/oodt/cas/pge/writers/MockDynamicConfigFileWriter.java 1616402 Diff: https://reviews.apache.org/r/24529/diff/ Testing ------- I've tested this on my DARPA XDATA translation ETL pipeline. Full tests are ongoing, but this works up the point of ingestion. There is something weird going on here with InPlaceIngestion, which I'm going to take a look at, and fix, but it's pretty much done. Enjoy! Thanks, Chris Mattmann --===============5258580233895648610==--