oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (3980)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Failed to build PgeConfig, exception in PathUtils.doDynamicReplacement
Date Fri, 10 Oct 2014 23:58:32 GMT
Thanks Val you are close!

Looking at what you show below, I think the issue is one of the following:

1. You are using the new style CAS-PGE Keys. Try using the old ones (e.g.,
the ones present here:
http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/examples/
WorkflowTask/tasks.xml

(note the ³_² and not the ³/³ used in keys).
These keys are used by CAS-PGE if you see something in your wmgr bin
script 
(or your resource manager batch_stub script) stating ³legacyMode=true²).
If you
are using RADIX, I believe that¹s the case:

http://s.apache.org/hO

2. Regarding better building environments. The m2e plugin for Eclipse is
fantastic
nowadays and can literally checkout a multi-module Maven project from SVN
(when paired
with Subversive as a plugin or Subclipse). Another thing to check out is
this page on
the wiki:

https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Developer+Hel
p


Let me know if that fixes it. Sorry for all the trouble :) Trust me
it will be worth it. CAS-PGE is awesome once working.

Cheers,
Chris


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: <Mallder>, Valerie <Valerie.Mallder@jhuapl.edu>
Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Date: Friday, October 10, 2014 at 3:40 PM
To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Subject: Failed to build PgeConfig, exception in
PathUtils.doDynamicReplacement

>Hi All,
>
>I'm still working on this! Still trying to get a CA-PGE Task to run.  I'm
>almost there. Now it at least trys to build the PGE config file. But the
>XmlFilePgeConfigBuilder is failing.   I looks like it is picking up a
>null string somewhere, but it also looks like it is crashing in a strange
>place.  It is unable to read the .xml file that contains my
><pgeConfig>...</pgeConfig> but, the correct path and file are shown in
>the output log messages I've include below which also shows the stack
>trace.  I looks like it is failing during a recursive call to
>PathUtils.doDynamicReplacement.  I saw some chatter about some prior
>errors in this code in the mailing list archives from 2010 and 2012. So I
>am hoping someone might remember and be able to tell me if I am doing
>something that is causing this error. I included my tasks.xml and
>fei-crawler-pge-config.xml file after the log messages.
>
>At this point, I think need to start running these processes in an
>environment where I can debug this better.  The runtime output and stack
>traces simply aren't enough for me to track this down.  I am newbie to
>using eclipse and mvn, so, does anyone have some notes on the best way to
>import oodt-0.7 into an exclipe workspace and tell it to build it using
>mvn??  I am hoping I can leverage other peoples knowledge of how to do
>this so that I can do it quickly and not waste another week. I imported
>the oodt-0.7 directory into eclipse so I could view the files easier, but
>I just made eclipse link to the where the files are located rather that
>copy them into a workspace folder.  If there's a better way, please tell
>me. And now, how do I tell exclipse to build stuff?
>
>Thanks!!
>Valerie
>
>
>
>Using CATALINA_BASE:
>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
>Using CATALINA_HOME:
>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
>Using CATALINA_TMPDIR:
>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat/temp
>Using JRE_HOME:       /project/jedi/users/jedi-pipeline/jdk1.7.0_55
>Workflow Manager started PID file
>(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow/run
>/cas.workflow.pid).
>Oct 10, 2014 5:53:52 PM
>org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager loadProperties
>INFO: Loading Workflow Manager Configuration Properties from:
>[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow/etc
>/workflow.properties]
>Oct 10, 2014 5:53:52 PM
>org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactory
>getResmgrUrl
>INFO: No Resource Manager URL provided or malformed URL: executing jobs
>locally. URL: [null]
>Oct 10, 2014 5:53:52 PM
>org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init>
>INFO: Workflow Manager started by malldva1
>Oct 10, 2014 5:54:12 PM
>org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent
>INFO: WorkflowManager: Received event: startJediPipeline
>Oct 10, 2014 5:54:12 PM
>org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent
>INFO: WorkflowManager: Workflow jediWorkflowName retrieved for event
>startJediPipeline
>Oct 10, 2014 5:54:13 PM
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>checkTaskRequiredMetadata
>INFO: Task: [feiCrawlerTaskName] has no required metadata fields
>Oct 10, 2014 5:54:13 PM
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>executeTaskLocally
>INFO: Executing task: [feiCrawlerTaskName] locally
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>INFO: Converting workflow configuration to static metadata...
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key = [PGETask/WorkflowManagerUrl] value
>= [http://localhost:9001]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key = [PGETask/Ingest/CrawlerConfigFile]
>value = 
>[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler
>/policy/crawler-config.xml]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key = [PGETask/Name] value =
>[feiCrawlerTaskName]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key = [PGETask/Ingest/ActionIds] value =
>[MoveFileToLevel0Dir]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key = [PGETask/DumpMetadata] value =
>[true]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key =
>[PGETask/Query/ClientTransferServiceFactory] value =
>[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key = [PGE_HOME] value =
>[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key = [PGETask/Query/FileManagerUrl]
>value = [http://localhost:9000]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key = [PGETask/Ingest/MimeExtractorRepo]
>value = 
>[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensi
>ons/policy/mime-extractor-map.xml]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key = [PGETask/ConfigFilePath] value =
>[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensi
>ons/config/fei-crawler-pge-config.xml]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding static metadata: key = [PGETask/Ingest/FileManagerUrl]
>value = [http://localhost:9000]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>INFO: Loading workflow context metadata...
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding dynamic metadata: key = [WorkflowInstId] value =
>[f8730997-50c7-11e4-b9aa-57625eee7ebd]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding dynamic metadata: key = [JobId] value =
>[f8730997-50c7-11e4-b9aa-57625eee7ebd]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value =
>[http://slothrop.jhuapl.edu:9001]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding dynamic metadata: key = [TaskId] value =
>[urn:oodt:feiCrawlerTaskId]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeMetadata
>FINEST: Adding dynamic metadata: key = [ProcessingNode] value =
>[slothrop.jhuapl.edu]
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeConfig
>INFO: Create PgeConfig...
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>createPgeConfig
>INFO: Using default PgeConfigBuilder:
>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
>Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance run
>SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse
>value: null
>java.io.IOException: Failed to build PgeConfig : Failed to parse value:
>null
>     at 
>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCon
>figBuilder.java:87)
>     at 
>org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.ja
>va:230)
>     at 
>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
>     at 
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.execu
>teTaskLocally(IterativeWorkflowProcessorThread.java:574)
>     at 
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(I
>terativeWorkflowProcessorThread.java:321)
>     at 
>EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
>     at java.lang.Thread.run(Thread.java:745)
>Caused by: java.lang.Exception: Failed to parse value: null
>     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
>     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
>     at 
>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCon
>figBuilder.java:77)
>     ... 6 more
>Caused by: java.lang.NullPointerException
>     at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
>     at java.util.regex.Matcher.reset(Matcher.java:308)
>     at java.util.regex.Matcher.<init>(Matcher.java:228)
>     at java.util.regex.Pattern.matcher(Pattern.java:1088)
>     at 
>org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplaceme
>nt(PathUtils.java:321)
>     at 
>org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUtils
>.java:96)
>     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
>     ... 8 more
>
>org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExcept
>ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse
>value: null
>     at 
>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
>     at 
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.execu
>teTaskLocally(IterativeWorkflowProcessorThread.java:574)
>     at 
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(I
>terativeWorkflowProcessorThread.java:321)
>     at 
>EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
>     at java.lang.Thread.run(Thread.java:745)
>Caused by: java.io.IOException: Failed to build PgeConfig : Failed to
>parse value: null
>     at 
>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCon
>figBuilder.java:87)
>     at 
>org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.ja
>va:230)
>     at 
>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
>     ... 4 more
>Caused by: java.lang.Exception: Failed to parse value: null
>     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
>     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
>     at 
>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCon
>figBuilder.java:77)
>     ... 6 more
>Caused by: java.lang.NullPointerException
>     at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
>     at java.util.regex.Matcher.reset(Matcher.java:308)
>     at java.util.regex.Matcher.<init>(Matcher.java:228)
>     at java.util.regex.Pattern.matcher(Pattern.java:1088)
>     at 
>org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplaceme
>nt(PathUtils.java:321)
>     at 
>org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUtils
>.java:96)
>     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
>     ... 8 more
>Oct 10, 2014 5:54:13 PM
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>executeTaskLocally
>WARNING: Exception executing task: [feiCrawlerTaskName] locally: Message:
>PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
>null
>
>
>
>
>Content of tasks.xml
>
>
><cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas">
>   <task id="urn:oodt:feiCrawlerTaskId" name="feiCrawlerTaskName"
>class="org.apache.oodt.cas.pge.StdPGETaskInstance">
>      <conditions/>
>
>      <configuration>
>         <property name="PGETask/Name" value="feiCrawlerTaskName"/>
>         <property name="PGETask/ConfigFilePath"
>value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/e
>xtensions/config/fei-crawler-pge-config.xml"/>
>         <property name="PGETask/DumpMetadata" value="true"/>
>         <property name="PGETask/WorkflowManagerUrl"
>value="http://localhost:9001" />
>         <property name="PGETask/Query/FileManagerUrl"
>value="http://localhost:9000" />
>         <property name="PGETask/Ingest/FileManagerUrl"
>value="http://localhost:9000"/>
>
>         <property name="PGETask/Query/ClientTransferServiceFactory"
>value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory"/
>>
>         <property name="PGETask/Ingest/CrawlerConfigFile"
>value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/c
>rawler/policy/crawler-config.xml"/>
>         <property name="PGETask/Ingest/MimeExtractorRepo"
>value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/e
>xtensions/policy/mime-extractor-map.xml"/>
>         <property name="PGETask/Ingest/ActionIds"
>value="MoveFileToLevel0Dir"/>
>         <property name="PGE_HOME"
>value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"/>
>      </configuration>
>
>      <requiredMetFields/>
>   </task>
></cas:tasks>
>
>
>Contents of fei-crawler-pge-config.xml
>
>
><pgeConfig>
>
>   <!-- How to run the PGE -->
>   <!-just echoing the current directory to a file so I can see if this
>thing ever starts -->
>   <exe 
>dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge
>/jobs" shell="/bin/sh">
>      <cmd>echo "Current Working Directory is `pwd`" >
>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/pge.log<
>/cmd>
>   </exe>
>
>   <!-- Files to ingest -->
>   <output>
>      <!-- trying this approach. Telling PGE there is output files should
>invoke the crawler. -->
>      
><dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/st
>aging"/>
>   </output>
>
>   <!-- Custom metadata to add to output files -->
>   <customMetadata>
>
>      <!-- each of these directories exist -->
>      <metadata key="JobDir" val="[OODT_HOME]/data/pge/jobs"
>envReplace="true"/>
>      <metadata key="JobInputDir" val="[JobDir]/input"/>
>      <metadata key="JobOutputDir" val="[JobDir]/output"/>
>      <metadata key="JobLogDir" val="[JobDir]/logs"/>
>
>  </customMetadata>
></pgeConfig>
>
>Valerie A. Mallder
>
>New Horizons Deputy Mission System Engineer
>The Johns Hopkins University/Applied Physics Laboratory
>11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
>240-228-7846 (Office) 410-504-2233 (Blackberry)
>


Mime
View raw message