oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MengYing Wang <mengyingwa...@gmail.com>
Subject Re: RADiX CAS-Crawler cannot trigger workflow event
Date Thu, 06 Nov 2014 06:18:18 GMT
Dear Prof. Mattmann,

Yes, you are right. To solve the problem, we have to change the
[WORKFLOWMGR_URL]
to [WORKFLOW_URL] in the action-beans.xml file.

The default CAS-crawler has been changed to [WORKFLOW_URL] in
https://issues.apache.org/jira/browse/OODT-724.

I have created a new JIRA ticket (
https://issues.apache.org/jira/browse/OODT-783) to change the RADiX crawler.

Also the path file has been uploaded, and a pull request has been committed
(https://github.com/apache/oodt/pull/9). Thanks.

Best,
Mengying (Angela) Wang



On Wed, Nov 5, 2014 at 12:37 PM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> That’s the problem - the policy is wrong in the RADIX crawler
> policy. Can you open up a JIRA ticket to fix this? It’s probably
> wrong too in the default CAS-crawler policy. The value inside of
> that file should be [WORKFLOW_URL].
>
> Please open up a JIRA and submit a patch. Thanks Angela!
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: MengYing Wang <mengyingwang1@gmail.com>
> Date: Tuesday, November 4, 2014 at 11:35 AM
> To: Chris Mattmann <Chris.A.Mattmann@jpl.nasa.gov>
> Cc: "dev@oodt.apache.org" <dev@oodt.apache.org>, Chris Mattmann
> <mattmann@apache.org>
> Subject: Re: RADiX CAS-Crawler cannot trigger workflow event
>
> >Dear Prof. Mattmann,
> >
> >
> >In the default property, there is a line "<prop
> >key="crawler.workflowmgr.url">[WORKFLOWMGR_URL]</prop>" while no
> >definition of the [WORKFLOW_URL].
> >
> >
> >        <!-- Default Properties -->
> >        <property name="properties">
> >            <props>
> >                <prop key="crawler.failure.dir">[FAILURE_DIR]</prop>
> >                <prop key="crawler.backup.dir">[BACKUP_DIR]</prop>
> >                <prop
> >key="crawler.workflowmgr.url">[WORKFLOWMGR_URL]</prop>
> >                <prop key="crawler.filemgr.url">[FILEMGR_URL]</prop>
> >                <prop
> >key="crawler.client.transferer">org.apache.oodt.cas.filemgr.datatransfer.L
> >ocalDataTransferFactory</prop>
> >                <prop key="crawler.met.file.ext">met</prop>
> >                <prop
> >key="crawler.pushpull.met.file.ext">[PUSHPULL_MET_FILE_EXT]</prop>
> >                <prop key="crawler.anc.file.ext">anc</prop>
> >                <prop key="crawler.anc.file.suffix"></prop>
> >                <prop key="notification.mail.host"></prop>
> >                <prop key="notification.sender"></prop>
> >                <prop key="notification.success.subject"></prop>
> >                <prop key="notification.success.message"></prop>
> >                <prop key="notification.success.recipients"></prop>
> >            </props>
> >        </property>
> >
> >
> >
> >Below is the definition of the TriggerPostIngestWorkflow event with the
> >workflowMgrUrl.
> >
> >
> >    <!-- Workflow Manager Actions -->
> >    <bean id="TriggerPostIngestWorkflow" lazy-init="true"
> >class="org.apache.oodt.cas.crawl.action.WorkflowMgrStatusUpdate">
> >        <property name="description" value="Triggers workflow event with
> >the name [ProductType]Ingest" />
> >        <property name="ingestSuffix" value="Ingest" />
> >        <property name="workflowMgrUrl"
> >value="${crawler.workflowmgr.url}" />
> >        <property name="phases">
> >            <list>
> >                <value type="java.lang.String">postIngestSuccess</value>
> >            </list>
> >        </property>
> >    </bean>
> >
> >
> >
> >Thanks!
> >
> >
> >Best,
> >Mengying (Angela) Wang
> >
> >
> >On Tue, Nov 4, 2014 at 10:21 AM, Mattmann, Chris A (3980)
> ><chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >Thanks Angela. What do you see in terms of your policy in
> >$OODT_HOME/crawler/policy in terms of action-beans.xml - do you
> >see a [WORKFLOW_URL] defined in there?
> >
> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >Chris Mattmann, Ph.D.
> >Chief Architect
> >Instrument Software and Science Data Systems Section (398)
> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >Office: 168-519, Mailstop: 168-527
> >Email: chris.a.mattmann@nasa.gov
> >WWW:  http://sunset.usc.edu/~mattmann/
> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >Adjunct Associate Professor, Computer Science Department
> >University of Southern California, Los Angeles, CA 90089 USA
> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> >-----Original Message-----
> >From: MengYing Wang <mengyingwang1@gmail.com>
> >Date: Tuesday, November 4, 2014 at 11:07 AM
> >To: Chris Mattmann <Chris.A.Mattmann@jpl.nasa.gov>
> >Cc: "dev@oodt.apache.org" <dev@oodt.apache.org>, Chris Mattmann
> ><mattmann@apache.org>
> >Subject: Re: RADiX CAS-Crawler cannot trigger workflow event
> >
> >>Dear Prof. Mattmann,
> >>
> >>
> >>I have set all the required environment variables.
> >>
> >>
> >>export OODT_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7"
> >>export
> >>CRAWLER_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/crawler"
> >>export
> >>FILEMGR_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr"
> >>export
> >>WORKFLOW_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/workflow"
> >>export PGE_ROOT="/Users/AngelaWang/Documents/programs/oodt-0.7/pge"
> >>export WORKFLOW_URL="http://localhost:9200"
> >>export FILEMGR_URL="http://localhost:9000"
> >>
> >>
> >>
> >>It seems the RADiX OODT may not as stable as the original OODT? Thanks.
> >>
> >>
> >>Mengying (Angela) Wang
> >>
> >>
> >>On Sat, Nov 1, 2014 at 10:43 AM, Mattmann, Chris A (3980)
> >><chris.a.mattmann@jpl.nasa.gov> wrote:
> >>
> >>Hi Angela,
> >>
> >>This is an issue with you not having your WORKFLOW_URL environment
> >>variable set. Can you make sure it¹s set before you start the workflow
> >>manager and RADIX? Please make sure it¹s getting passed down..
> >>
> >>Cheers,
> >>Chris
> >>
> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>Chris Mattmann, Ph.D.
> >>Chief Architect
> >>Instrument Software and Science Data Systems Section (398)
> >>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>Office: 168-519, Mailstop: 168-527
> >>Email: chris.a.mattmann@nasa.gov
> >>WWW:  http://sunset.usc.edu/~mattmann/
> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>Adjunct Associate Professor, Computer Science Department
> >>University of Southern California, Los Angeles, CA 90089 USA
> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >>
> >>
> >>-----Original Message-----
> >>From: MengYing Wang <mengyingwang1@gmail.com>
> >>Date: Thursday, October 30, 2014 at 9:30 PM
> >>To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> >>Cc: Chris Mattmann <mattmann@apache.org>
> >>Subject: RADiX CAS-Crawler cannot trigger workflow event
> >>
> >>>Hi Everyone,
> >>>
> >>>
> >>>I am trying to trigger some workflow evert using the crawler. However,
> >>>it
> >>>raises a "no protocol" error:
> >>>
> >>>
> >>>INFO: Performing action (id = TriggerPostIngestWorkflow : description =
> >>>Triggers workflow event with the name [ProductType]Ingest)
> >>>Oct 30, 2014 4:41:45 PM org.apache.oodt.cas.crawl.ProductCrawler
> >>>performProductCrawlerActions
> >>>WARNING: Failed to perform crawler action : Failed to update workflow
> >>>manager : no protocol: null
> >>>org.apache.oodt.cas.crawl.structs.exceptions.CrawlerActionException:
> >>>Failed to update workflow manager : no protocol: null
> >>>at
> >>>org.apache.oodt.cas.crawl.action.WorkflowMgrStatusUpdate.performAction(W
> >>>o
> >>>r
> >>>kflowMgrStatusUpdate.java:59)
> >>>at
> >>>org.apache.oodt.cas.crawl.ProductCrawler.performProductCrawlerActions(Pr
> >>>o
> >>>d
> >>>uctCrawler.java:357)
> >>>at
> >>>org.apache.oodt.cas.crawl.ProductCrawler.performPostIngestOnSuccessActio
> >>>n
> >>>s
> >>>(ProductCrawler.java:331)
> >>>at
> >>>org.apache.oodt.cas.crawl.ProductCrawler.handleFile(ProductCrawler.java:
> >>>1
> >>>9
> >>>4)
> >>>at
> >>>org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:108)
> >>>at
> >>>org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:75)
> >>>at
> >>>org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(Cr
> >>>a
> >>>w
> >>>lerLauncherCliAction.java:58)
> >>>at
> >>>org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
> >>>at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
> >>>at
> >>>org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)
> >>>
> >>>
> >>>It seems some thing wrong with my crawler. Should I do some
> >>>configuration
> >>>to my crawler? And how? Thank you for your time and help!
> >>>
> >>>
> >>>--
> >>>Best,
> >>>Mengying (Angela) Wang
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>--
> >>Best,
> >>Mengying (Angela) Wang
> >>
> >>
> >>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >--
> >Best,
> >Mengying (Angela) Wang
> >
> >
> >
>
>


-- 
Best,
Mengying (Angela) Wang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message