oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mallder, Valerie" <Valerie.Mall...@jhuapl.edu>
Subject RE: Failed to build PgeConfig, exception in PathUtils.doDynamicReplacement
Date Tue, 14 Oct 2014 19:58:55 GMT
Hi Chris,

Thanks for your emails from Friday and Saturday. They were extremely helpful and I have now
worked my way through all of my setup issues so far.  I was using the new style CASE-PGE keys
because Rishi used those in his working example and at one point I had been trying to mimic
that example. I didn't know anything about the legacy mode flag. But, after changing the keys
to the old style, everything started to come together a little faster.

To start my crawler at the beginning of my pipeline, I chose the approach that Rishi had taken
where the pgeConfig starts the crawler_launcher script directly rather than rely on the definition
of the output files to trigger the crawler to run.  This was because the met extractor I am
using is a perl script, and your DRAT system only shows how to specify a class for the metFileWriterClass
attribute. I could not find anything to show me how to specify a perl script as the met file
writer instead.  Also, having to specify the file types to crawl for by using a regular expression
in the config file seemed like an unnecessary duplication of the definition of the files types
to crawl for (since the regular expression just specifies a mime-type that is already defined
in mime-types.xml). I didn't want to specify the mime-type in two different places.

So, now I've been through the process of setting up my CAS-PGE to ingest the raw science and
spacecraft housekeeping telemetry files, and run a postIngestSuccess action to copy some of
those files to a new directory for further processing.

The next step in my pipeline may be a little tricky. I need to query the filemgr and pass
the results of the query (it could be a long list) as an input parameter to the first algorithm
in the pipeline which, happens to be another perl script.  Does anyone know how I might do
this in a PGE config file?

Thanks!!
Val





Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory


> -----Original Message-----
> From: Mattmann, Chris A (3980) [mailto:chris.a.mattmann@jpl.nasa.gov]
> Sent: Friday, October 10, 2014 7:59 PM
> To: dev@oodt.apache.org
> Subject: Re: Failed to build PgeConfig, exception in
> PathUtils.doDynamicReplacement
>
> Thanks Val you are close!
>
> Looking at what you show below, I think the issue is one of the following:
>
> 1. You are using the new style CAS-PGE Keys. Try using the old ones (e.g., the
> ones present here:
> http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/examples/
> WorkflowTask/tasks.xml
>
> (note the ?_? and not the ?/? used in keys).
> These keys are used by CAS-PGE if you see something in your wmgr bin script (or
> your resource manager batch_stub script) stating ?legacyMode=true?).
> If you
> are using RADIX, I believe that?s the case:
>
> http://s.apache.org/hO
>
> 2. Regarding better building environments. The m2e plugin for Eclipse is fantastic
> nowadays and can literally checkout a multi-module Maven project from SVN
> (when paired with Subversive as a plugin or Subclipse). Another thing to check out
> is this page on the wiki:
>
> https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Developer+Hel
> p
>
>
> Let me know if that fixes it. Sorry for all the trouble :) Trust me it will be worth
it.
> CAS-PGE is awesome once working.
>
> Cheers,
> Chris
>
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> +++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398) NASA Jet
> Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> +++++
> Adjunct Associate Professor, Computer Science Department University of
> Southern California, Los Angeles, CA 90089 USA
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> +++++
>
>
>
>
>
>
> -----Original Message-----
> From: <Mallder>, Valerie <Valerie.Mallder@jhuapl.edu>
> Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Date: Friday, October 10, 2014 at 3:40 PM
> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Subject: Failed to build PgeConfig, exception in PathUtils.doDynamicReplacement
>
> >Hi All,
> >
> >I'm still working on this! Still trying to get a CA-PGE Task to run.
> >I'm almost there. Now it at least trys to build the PGE config file. But the
> >XmlFilePgeConfigBuilder is failing.   I looks like it is picking up a
> >null string somewhere, but it also looks like it is crashing in a
> >strange place.  It is unable to read the .xml file that contains my
> ><pgeConfig>...</pgeConfig> but, the correct path and file are shown in
> >the output log messages I've include below which also shows the stack
> >trace.  I looks like it is failing during a recursive call to
> >PathUtils.doDynamicReplacement.  I saw some chatter about some prior
> >errors in this code in the mailing list archives from 2010 and 2012. So
> >I am hoping someone might remember and be able to tell me if I am doing
> >something that is causing this error. I included my tasks.xml and
> >fei-crawler-pge-config.xml file after the log messages.
> >
> >At this point, I think need to start running these processes in an
> >environment where I can debug this better.  The runtime output and
> >stack traces simply aren't enough for me to track this down.  I am
> >newbie to using eclipse and mvn, so, does anyone have some notes on the
> >best way to import oodt-0.7 into an exclipe workspace and tell it to
> >build it using mvn??  I am hoping I can leverage other peoples
> >knowledge of how to do this so that I can do it quickly and not waste
> >another week. I imported the oodt-0.7 directory into eclipse so I could
> >view the files easier, but I just made eclipse link to the where the
> >files are located rather that copy them into a workspace folder.  If
> >there's a better way, please tell me. And now, how do I tell exclipse to build
> stuff?
> >
> >Thanks!!
> >Valerie
> >
> >
> >
> >Using CATALINA_BASE:
> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
> >Using CATALINA_HOME:
> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
> >Using CATALINA_TMPDIR:
> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat/temp
> >Using JRE_HOME:       /project/jedi/users/jedi-pipeline/jdk1.7.0_55
> >Workflow Manager started PID file
> >(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow/
> >run
> >/cas.workflow.pid).
> >Oct 10, 2014 5:53:52 PM
> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> >loadProperties
> >INFO: Loading Workflow Manager Configuration Properties from:
> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow/
> >etc
> >/workflow.properties]
> >Oct 10, 2014 5:53:52 PM
> >org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactory
> >getResmgrUrl
> >INFO: No Resource Manager URL provided or malformed URL: executing jobs
> >locally. URL: [null] Oct 10, 2014 5:53:52 PM
> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init>
> >INFO: Workflow Manager started by malldva1 Oct 10, 2014 5:54:12 PM
> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent
> >INFO: WorkflowManager: Received event: startJediPipeline Oct 10, 2014
> >5:54:12 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> >handleEvent
> >INFO: WorkflowManager: Workflow jediWorkflowName retrieved for event
> >startJediPipeline Oct 10, 2014 5:54:13 PM
> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >checkTaskRequiredMetadata
> >INFO: Task: [feiCrawlerTaskName] has no required metadata fields Oct
> >10, 2014 5:54:13 PM
> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >executeTaskLocally
> >INFO: Executing task: [feiCrawlerTaskName] locally Oct 10, 2014 5:54:13
> >PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >INFO: Converting workflow configuration to static metadata...
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key = [PGETask/WorkflowManagerUrl]
> >value = [http://localhost:9001] Oct 10, 2014 5:54:13 PM
> >org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key =
> >[PGETask/Ingest/CrawlerConfigFile]
> >value =
> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/craw
> >ler
> >/policy/crawler-config.xml]
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key = [PGETask/Name] value =
> >[feiCrawlerTaskName] Oct 10, 2014 5:54:13 PM
> >org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key = [PGETask/Ingest/ActionIds] value
> >= [MoveFileToLevel0Dir] Oct 10, 2014 5:54:13 PM
> >org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key = [PGETask/DumpMetadata] value =
> >[true] Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key =
> >[PGETask/Query/ClientTransferServiceFactory] value =
> >[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key = [PGE_HOME] value =
> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge]
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key = [PGETask/Query/FileManagerUrl]
> >value = [http://localhost:9000] Oct 10, 2014 5:54:13 PM
> >org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key =
> >[PGETask/Ingest/MimeExtractorRepo]
> >value =
> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/exte
> >nsi
> >ons/policy/mime-extractor-map.xml]
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key = [PGETask/ConfigFilePath] value =
> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/exte
> >nsi
> >ons/config/fei-crawler-pge-config.xml]
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding static metadata: key = [PGETask/Ingest/FileManagerUrl]
> >value = [http://localhost:9000] Oct 10, 2014 5:54:13 PM
> >org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >INFO: Loading workflow context metadata...
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding dynamic metadata: key = [WorkflowInstId] value =
> >[f8730997-50c7-11e4-b9aa-57625eee7ebd]
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding dynamic metadata: key = [JobId] value =
> >[f8730997-50c7-11e4-b9aa-57625eee7ebd]
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value =
> >[http://slothrop.jhuapl.edu:9001]
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding dynamic metadata: key = [TaskId] value =
> >[urn:oodt:feiCrawlerTaskId] Oct 10, 2014 5:54:13 PM
> >org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeMetadata
> >FINEST: Adding dynamic metadata: key = [ProcessingNode] value =
> >[slothrop.jhuapl.edu] Oct 10, 2014 5:54:13 PM
> >org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeConfig
> >INFO: Create PgeConfig...
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >createPgeConfig
> >INFO: Using default PgeConfigBuilder:
> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance run
> >SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse
> >value: null
> >java.io.IOException: Failed to build PgeConfig : Failed to parse value:
> >null
> >     at
> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePge
> >Con
> >figBuilder.java:87)
> >     at
> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance
> >.ja
> >va:230)
> >     at
> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
> >     at
> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ex
> >ecu
> >teTaskLocally(IterativeWorkflowProcessorThread.java:574)
> >     at
> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ru
> >n(I
> >terativeWorkflowProcessorThread.java:321)
> >     at
> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
> >     at java.lang.Thread.run(Thread.java:745)
> >Caused by: java.lang.Exception: Failed to parse value: null
> >     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
> >     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
> >     at
> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePge
> >Con
> >figBuilder.java:77)
> >     ... 6 more
> >Caused by: java.lang.NullPointerException
> >     at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
> >     at java.util.regex.Matcher.reset(Matcher.java:308)
> >     at java.util.regex.Matcher.<init>(Matcher.java:228)
> >     at java.util.regex.Pattern.matcher(Pattern.java:1088)
> >     at
> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplac
> >eme
> >nt(PathUtils.java:321)
> >     at
> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUt
> >ils
> >.java:96)
> >     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
> >     ... 8 more
> >
> >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExc
> >ept
> >ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse
> >value: null
> >     at
> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
> >     at
> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ex
> >ecu
> >teTaskLocally(IterativeWorkflowProcessorThread.java:574)
> >     at
> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ru
> >n(I
> >terativeWorkflowProcessorThread.java:321)
> >     at
> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
> >     at java.lang.Thread.run(Thread.java:745)
> >Caused by: java.io.IOException: Failed to build PgeConfig : Failed to
> >parse value: null
> >     at
> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePge
> >Con
> >figBuilder.java:87)
> >     at
> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance
> >.ja
> >va:230)
> >     at
> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
> >     ... 4 more
> >Caused by: java.lang.Exception: Failed to parse value: null
> >     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
> >     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
> >     at
> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePge
> >Con
> >figBuilder.java:77)
> >     ... 6 more
> >Caused by: java.lang.NullPointerException
> >     at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
> >     at java.util.regex.Matcher.reset(Matcher.java:308)
> >     at java.util.regex.Matcher.<init>(Matcher.java:228)
> >     at java.util.regex.Pattern.matcher(Pattern.java:1088)
> >     at
> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplac
> >eme
> >nt(PathUtils.java:321)
> >     at
> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUt
> >ils
> >.java:96)
> >     at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
> >     ... 8 more
> >Oct 10, 2014 5:54:13 PM
> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >executeTaskLocally
> >WARNING: Exception executing task: [feiCrawlerTaskName] locally: Message:
> >PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
> >null
> >
> >
> >
> >
> >Content of tasks.xml
> >
> >
> ><cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas">
> >   <task id="urn:oodt:feiCrawlerTaskId" name="feiCrawlerTaskName"
> >class="org.apache.oodt.cas.pge.StdPGETaskInstance">
> >      <conditions/>
> >
> >      <configuration>
> >         <property name="PGETask/Name" value="feiCrawlerTaskName"/>
> >         <property name="PGETask/ConfigFilePath"
> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deplo
> >y/e xtensions/config/fei-crawler-pge-config.xml"/>
> >         <property name="PGETask/DumpMetadata" value="true"/>
> >         <property name="PGETask/WorkflowManagerUrl"
> >value="http://localhost:9001" />
> >         <property name="PGETask/Query/FileManagerUrl"
> >value="http://localhost:9000" />
> >         <property name="PGETask/Ingest/FileManagerUrl"
> >value="http://localhost:9000"/>
> >
> >         <property name="PGETask/Query/ClientTransferServiceFactory"
> >value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactor
> >y"/
> >>
> >         <property name="PGETask/Ingest/CrawlerConfigFile"
> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deplo
> >y/c
> >rawler/policy/crawler-config.xml"/>
> >         <property name="PGETask/Ingest/MimeExtractorRepo"
> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deplo
> >y/e xtensions/policy/mime-extractor-map.xml"/>
> >         <property name="PGETask/Ingest/ActionIds"
> >value="MoveFileToLevel0Dir"/>
> >         <property name="PGE_HOME"
> >value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"/>
> >      </configuration>
> >
> >      <requiredMetFields/>
> >   </task>
> ></cas:tasks>
> >
> >
> >Contents of fei-crawler-pge-config.xml
> >
> >
> ><pgeConfig>
> >
> >   <!-- How to run the PGE -->
> >   <!-just echoing the current directory to a file so I can see if this
> >thing ever starts -->
> >   <exe
> >dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/
> >pge
> >/jobs" shell="/bin/sh">
> >      <cmd>echo "Current Working Directory is `pwd`" >
> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/pge.l
> >og<
> >/cmd>
> >   </exe>
> >
> >   <!-- Files to ingest -->
> >   <output>
> >      <!-- trying this approach. Telling PGE there is output files
> >should invoke the crawler. -->
> >
> ><dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data
> >/st
> >aging"/>
> >   </output>
> >
> >   <!-- Custom metadata to add to output files -->
> >   <customMetadata>
> >
> >      <!-- each of these directories exist -->
> >      <metadata key="JobDir" val="[OODT_HOME]/data/pge/jobs"
> >envReplace="true"/>
> >      <metadata key="JobInputDir" val="[JobDir]/input"/>
> >      <metadata key="JobOutputDir" val="[JobDir]/output"/>
> >      <metadata key="JobLogDir" val="[JobDir]/logs"/>
> >
> >  </customMetadata>
> ></pgeConfig>
> >
> >Valerie A. Mallder
> >
> >New Horizons Deputy Mission System Engineer The Johns Hopkins
> >University/Applied Physics Laboratory
> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
> >240-228-7846 (Office) 410-504-2233 (Blackberry)
> >


Mime
View raw message