oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: CAS PGE Crawler Question
Date Thu, 09 Feb 2012 03:44:59 GMT
Great job, Mike! Would you be willing to write up a quick wiki FAQ on this,
here?

https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Documentation

Cheers,
Chris

On Feb 8, 2012, at 8:34 AM, Cayanan, Michael D (388J) wrote:

> Hey Brian and Chris,
> 
> So, I first re-tried it using Chris' suggestion of using spaces and I still got an error.
Here is what I'm seeing in the log:
> 
> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
> INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
> WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output'
(all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir'
is defined
> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow
MoveMetadataToBackupDir' is defined
> 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
> 	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
> 	at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
> 	at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
> 	at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
> 	at org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
> 	at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
> 	at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
> 	at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
> 	at java.lang.Thread.run(Thread.java:680)
> 
> 
> I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with
the action IDs:
> 
> <property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir"
/>
> 
> and added a line in the PgeConfig.xml as Brian suggested:
> 
> <customMetadata>
> 	  <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
> 
> and still got an error in the log as shown above. 
> 
> So, I ended up re-naming the property name in the tasks.xml to some other name (so as
to not reference itself essentially) and that did the trick!
> 
> tasks.xml:
> 
> <property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir"
/>
> 
> PgeConfig.xml:
> 
> <customMetadata>
> 	  <metadata key="PCS_ActionsIds" val="[Actions]"/>
> 
> 
> Also, I found out that you can't have spaces in between the commas like so,
> 
> <property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir"
/>
> 
> 
> Thanks for all your help guys! 
> 
> 
> -Mike
> 
> 
> From: holenoter <holenoter@mac.com>
> Reply-To: "user@oodt.apache.org" <user@oodt.apache.org>
> Date: Tue, 7 Feb 2012 22:06:27 -0800
> To: "user@oodt.apache.org" <user@oodt.apache.org>
> Subject: Re: CAS PGE Crawler Question
> 
> workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration
which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because
you have to allow for workflow metadata to contain values with commas without always being
split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it
becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma...
context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values)
but chris later changed this to the way it is now because this broke OCO since OCO had static
workflow metadata which contained metadata with commas that was not supposed to be parsed
into multi-values.
> 
> -brian
> 
> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
wrote:
> 
>> Hi Mike,
>> 
>> Space should work in the property value below. Can you show me
>> the log output from CAS-PGE in the workflow log file (or the resource manager
>> batch stubs?) It should show you what Actions are being executed
>> and that can help to debug.
>> 
>> Thanks!
>> 
>> Cheers,
>> Chris
>> 
>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>> 
>> > If I wanted to specify more than one PCS_ActionsIds values, how would I do
>> > this?
>> > 
>> > I've tried the following with no success:
>> > 
>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>> > 
>> > 
>> > and
>> > 
>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>> > 
>> > 
>> > and 
>> > 
>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>> > 
>> > 
>> > 
>> > -Mike
>> > 
>> > 
>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>> > <chris.a.mattmann@jpl.nasa.gov> wrote:
>> > 
>> >> Hi Mike,
>> >> 
>> >> Glad to hear!
>> >> 
>> >> Cheers,
>> >> Chris
>> >> 
>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>> >> 
>> >>> Hi all,
>> >>> 
>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>> >>> 
>> >>> -Mike
>> >>> 
>> >>> From: "Cayanan, Michael D (388J)" <michael.d.cayanan@jpl.nasa.gov>
>> >>> Reply-To: "user@oodt.apache.org" <user@oodt.apache.org>
>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>> >>> To: "user@oodt.apache.org" <user@oodt.apache.org>
>> >>> Cc: "Wong, Cynthia L (388J)" <cynthia.l.wong@jpl.nasa.gov>
>> >>> Subject: CAS PGE Crawler Question
>> >>> 
>> >>> Hi all,
>> >>> 
>> >>> I would like CAS PGE to run a post ingest success action that I created
>> >>> upon a successful ingestion of the output file(s). How do you configure
>> >>> CAS PGE for this type of scenario?
>> >>> 
>> >>> I have a tasks.xml file in the workflow, where I define my post ingest
>> >>> action under the 'PCS_ActionIds' property:
>> >>> 
>> >>> <task id="urn:my:Task" name="Task"
>> >>> ...
>> >>> <property name="PCS_ActionIds"
>> >>> value="MyPostIngestSuccessAction" />
>> >>> <property name="PCS_ActionRepoFile"
>> >>> value="crawler-config.xml" />
>> >>> </configuration>
>> >>> </task>
>> >>> 
>> >>> My crawler-config.xml has the following:
>> >>> 
>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> xmlns:p="http://www.springframework.org/schema/p"
>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>> >>> 
>> >>> <import 
>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>> >>> <import resource="./action-beans.xml"/>
>> >>> <import 
>> >>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>> >>> ans.xml"/>
>> >>> <import 
>> >>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>> >>> ns.xml"/>
>> >>> </beans>
>> >>> 
>> >>> Within my action-beans.xml, I added the post ingest success action:
>> >>> 
>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>> >>> 
>> >>> <bean 
>> >>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>> >>> r"/>
>> >>> 
>> >>> <!-- Crawler Actions -->
>> >>> 
>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>> >>> <property name="description" value="Kickoff Another Workflow
>> >>> Event"/>
>> >>> <property name="executeCommand" value="sh
>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>> >>> <property name="phases">
>> >>> <list>
>> >>> <value type="java.lang.String">postIngestSuccess</value>
>> >>> </list>
>> >>> </property>
>> >>> </bean>
>> >>> 
>> >>> Not sure if I'm missing anything else.
>> >>> 
>> >>> Thanks in advance,
>> >>> Mike
>> >>> 
>> >> 
>> >> 
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Chris Mattmann, Ph.D.
>> >> Senior Computer Scientist
>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> Office: 171-266B, Mailstop: 171-246
>> >> Email: chris.a.mattmann@nasa.gov
>> >> WWW: http://sunset.usc.edu/~mattmann/
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Adjunct Assistant Professor, Computer Science Department
>> >> University of Southern California, Los Angeles, CA 90089 USA
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> 
>> > 
>> 
>> 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mime
View raw message