oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: CAS PGE Crawler Question
Date Fri, 10 Feb 2012 04:56:19 GMT
Thanks Mike, you rock!

Cheers,
Chris

On Feb 9, 2012, at 7:21 AM, Cayanan, Michael D (388J) wrote:

> No problemo Mr Mattmann. I've added a FAQ section to the wiki on how to
> specify multiple crawler actions.
> 
> Thanks to you and Brian for your help on this,
> Mike
> 
> On 2/8/12 7:44 PM, "Mattmann, Chris A (388J)"
> <chris.a.mattmann@jpl.nasa.gov> wrote:
> 
>> Great job, Mike! Would you be willing to write up a quick wiki FAQ on
>> this,
>> here?
>> 
>> https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Document
>> ation
>> 
>> Cheers,
>> Chris
>> 
>> On Feb 8, 2012, at 8:34 AM, Cayanan, Michael D (388J) wrote:
>> 
>>> Hey Brian and Chris,
>>> 
>>> So, I first re-tried it using Chris' suggestion of using spaces and I
>>> still got an error. Here is what I'm seeing in the log:
>>> 
>>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance
>>> runIngestCrawler
>>> INFO: Executing StdProductCrawler in productPath:
>>> [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance
>>> runIngestCrawler
>>> WARNING: Failed while attempting to ingest products while crawling
>>> directory
>>> '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all
>>> products may not have been ingested) : No bean named
>>> 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>>> org.springframework.beans.factory.NoSuchBeanDefinitionException: No
>>> bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>>>     at
>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getB
>>> eanDefinition(DefaultListableBeanFactory.java:360)
>>>     at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBe
>>> anDefinition(AbstractBeanFactory.java:914)
>>>     at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs
>>> tractBeanFactory.java:241)
>>>     at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs
>>> tractBeanFactory.java:164)
>>>     at
>>> org.springframework.context.support.AbstractApplicationContext.getBean(Ab
>>> stractApplicationContext.java:761)
>>>     at
>>> org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFac
>>> tory(CrawlerActionRepo.java:68)
>>>     at
>>> org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
>>>     at
>>> org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.
>>> java:405)
>>>     at
>>> org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.ja
>>> va:349)
>>>     at
>>> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>>>     at
>>> org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>>>     at
>>> org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.ru
>>> n(XmlRpcBatchStub.java:229)
>>>     at java.lang.Thread.run(Thread.java:680)
>>> 
>>> 
>>> I then modified my tasks.xml file where I specified the
>>> 'PCS_ActionsIds' property with the action IDs:
>>> 
>>> <property name="PCS_ActionsIds"
>>> value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>> 
>>> and added a line in the PgeConfig.xml as Brian suggested:
>>> 
>>> <customMetadata>
>>>       <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>>> 
>>> and still got an error in the log as shown above.
>>> 
>>> So, I ended up re-naming the property name in the tasks.xml to some
>>> other name (so as to not reference itself essentially) and that did the
>>> trick!
>>> 
>>> tasks.xml:
>>> 
>>> <property name="Actions"
>>> value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>> 
>>> PgeConfig.xml:
>>> 
>>> <customMetadata>
>>>       <metadata key="PCS_ActionsIds" val="[Actions]"/>
>>> 
>>> 
>>> Also, I found out that you can't have spaces in between the commas like
>>> so,
>>> 
>>> <property name="Actions" value="KickoffL1BS0LoResWorkflow,
>>> MoveMetadataToBackupDir" />
>>> 
>>> 
>>> Thanks for all your help guys!
>>> 
>>> 
>>> -Mike
>>> 
>>> 
>>> From: holenoter <holenoter@mac.com>
>>> Reply-To: "user@oodt.apache.org" <user@oodt.apache.org>
>>> Date: Tue, 7 Feb 2012 22:06:27 -0800
>>> To: "user@oodt.apache.org" <user@oodt.apache.org>
>>> Subject: Re: CAS PGE Crawler Question
>>> 
>>> workflow manager properties (static metadata) are passed to CAS-PGE via
>>> WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE
>>> doesn't re-parse out these metadata value because you have to allow for
>>> workflow metadata to contain values with commas without always being
>>> split out by CAS-PGE.. so you have to explicitly repoint the metadata
>>> field at itself so it becomes custom (or local) metadata for CAS-PGE and
>>> then the value will be parsed on comma... context: CAS-PGE initially did
>>> the opposite (i.e. always re-parse workflow metadata into multi-values)
>>> but chris later changed this to the way it is now because this broke OCO
>>> since OCO had static workflow metadata which contained metadata with
>>> commas that was not supposed to be parsed into multi-values.
>>> 
>>> -brian
>>> 
>>> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)"
>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>> 
>>>> Hi Mike,
>>>> 
>>>> Space should work in the property value below. Can you show me
>>>> the log output from CAS-PGE in the workflow log file (or the resource
>>>> manager
>>>> batch stubs?) It should show you what Actions are being executed
>>>> and that can help to debug.
>>>> 
>>>> Thanks!
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>>> 
>>>>> If I wanted to specify more than one PCS_ActionsIds values, how
>>>> would I do
>>>>> this?
>>>>> 
>>>>> I've tried the following with no success:
>>>>> 
>>>>> <property name="PCS_ActionIds" value="Action1, Action2" />
>>>>> 
>>>>> 
>>>>> and
>>>>> 
>>>>> <property name="PCS_ActionIds" value="Action1,Action2" />
>>>>> 
>>>>> 
>>>>> and
>>>>> 
>>>>> <property name="PCS_ActionIds" value="Action1 Action2" />
>>>>> 
>>>>> 
>>>>> 
>>>>> -Mike
>>>>> 
>>>>> 
>>>>> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>>>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>> 
>>>>>> Hi Mike,
>>>>>> 
>>>>>> Glad to hear!
>>>>>> 
>>>>>> Cheers,
>>>>>> Chris
>>>>>> 
>>>>>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> Disregard my previous message. Turns out I had a typo. I had
to set
>>>>>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>>>>> 
>>>>>>> -Mike
>>>>>>> 
>>>>>>> From: "Cayanan, Michael D (388J)" <michael.d.cayanan@jpl.nasa.gov>
>>>>>>> Reply-To: "user@oodt.apache.org" <user@oodt.apache.org>
>>>>>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>>>>>> To: "user@oodt.apache.org" <user@oodt.apache.org>
>>>>>>> Cc: "Wong, Cynthia L (388J)" <cynthia.l.wong@jpl.nasa.gov>
>>>>>>> Subject: CAS PGE Crawler Question
>>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> I would like CAS PGE to run a post ingest success action that
I
>>>> created
>>>>>>> upon a successful ingestion of the output file(s). How do you
>>>> configure
>>>>>>> CAS PGE for this type of scenario?
>>>>>>> 
>>>>>>> I have a tasks.xml file in the workflow, where I define my post
>>>> ingest
>>>>>>> action under the 'PCS_ActionIds' property:
>>>>>>> 
>>>>>>> <task id="urn:my:Task" name="Task"
>>>>>>> ...
>>>>>>> <property name="PCS_ActionIds"
>>>>>>> value="MyPostIngestSuccessAction" />
>>>>>>> <property name="PCS_ActionRepoFile"
>>>>>>> value="crawler-config.xml" />
>>>>>>> </configuration>
>>>>>>> </task>
>>>>>>> 
>>>>>>> My crawler-config.xml has the following:
>>>>>>> 
>>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>>>> xmlns:p="http://www.springframework.org/schema/p"
>>>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>>>>> 
>>>>>>> <import
>>>>>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>>>>>> <import resource="./action-beans.xml"/>
>>>>>>> <import
>>>>>>> 
>>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-b
>>>> e
>>>>>>> ans.xml"/>
>>>>>>> <import
>>>>>>> 
>>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-be
>>>> a
>>>>>>> ns.xml"/>
>>>>>>> </beans>
>>>>>>> 
>>>>>>> Within my action-beans.xml, I added the post ingest success action:
>>>>>>> 
>>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>>>>> 
>>>>>>> <bean
>>>>>>> 
>>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcess
>>>> o
>>>>>>> r"/>
>>>>>>> 
>>>>>>> <!-- Crawler Actions -->
>>>>>>> 
>>>>>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>>>>>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>>>>>> <property name="description" value="Kickoff Another Workflow
>>>>>>> Event"/>
>>>>>>> <property name="executeCommand" value="sh
>>>>>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>>>>>> <property name="phases">
>>>>>>> <list>
>>>>>>> <value type="java.lang.String">postIngestSuccess</value>
>>>>>>> </list>
>>>>>>> </property>
>>>>>>> </bean>
>>>>>>> 
>>>>>>> Not sure if I'm missing anything else.
>>>>>>> 
>>>>>>> Thanks in advance,
>>>>>>> Mike
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Chris Mattmann, Ph.D.
>>>>>> Senior Computer Scientist
>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Senior Computer Scientist
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 171-266B, Mailstop: 171-246
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Assistant Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> 
>> 
>> 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 
> 


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mime
View raw message