oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Mattmann" <mattm...@apache.org>
Subject Review Request 24529: CAS-PGE no longer respects writers and file tags from earlier pgeConfig.xml files
Date Sat, 09 Aug 2014 20:53:11 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24529/
-----------------------------------------------------------

Review request for oodt.


Bugs: OODT-667
    https://issues.apache.org/jira/browse/OODT-667


Repository: oodt


Description
-------

This patch restores functionality and fixes CAS-PGE in trunk for users expecting 0.3 CAS-PGE
and before style configuration in which you didn't need a MIME extractor repo and could configure
CAS-PGE entirely from the pge-config.xml files. This patch effectively makes CAS-PGE usable
again in trunk for 0.7 and going forward and is fully forward compatible with Brian's changes.
If you specify a MIME extractor repo, you get an AutoDetectCrawler - otherwise you get the
familiar StdProductCrawler.


Diffs
-----

  ./trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/filenaming/PathUtilsNamingConvention.java
1616402 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1616402 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1616402 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1616402 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java PRE-CREATION

  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeMetadata.java 1616402 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1616402 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskStatus.java 1616402 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/GenericPgeObjectFactory.java 1616402

  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/XmlHelper.java 1616402 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/CsvConfigFileWriter.java 1616402

  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/DynamicConfigFileWriter.java 1616402

  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java
PRE-CREATION 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java PRE-CREATION

  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/MetadataKeyReplacerTemplateWriter.java
1616402 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java PRE-CREATION

  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java PRE-CREATION

  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/TextConfigFileWriter.java 1616402

  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/VelocityConfigFileWriter.java
1616402 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/XslTransformWriter.java 1616402

  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java
PRE-CREATION 
  ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java PRE-CREATION

  ./trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1616402 
  ./trunk/pge/src/test/java/org/apache/oodt/cas/pge/staging/TestFileStager.java 1616402 
  ./trunk/pge/src/test/java/org/apache/oodt/cas/pge/writers/MockDynamicConfigFileWriter.java
1616402 

Diff: https://reviews.apache.org/r/24529/diff/


Testing
-------

I've tested this on my DARPA XDATA translation ETL pipeline. Full tests are ongoing, but this
works up the point of ingestion. There is something weird going on here with InPlaceIngestion,
which I'm going to take a look at, and fix, but it's pretty much done. Enjoy!


Thanks,

Chris Mattmann


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message