oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Foster <holeno...@me.com>
Subject Re: Registering a custom ProductCrawler with cas-crawler
Date Thu, 26 Apr 2012 09:04:21 GMT

Hey Rishi,

Do what Chris said AND add a CmdLineAction to cmd-line-action.xml to run you crawler (should
just be a copy/paste and change some names)

-Brian

On Apr 25, 2012, at 10:12 PM, "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
wrote:

> Hey Rishi,
> 
> I think you need to change the actionRepo, something akin to this:
> 
> 1. Edit your crawler_launcher script:
> - make sure that the -Dorg.apache.oodt.cas.crawl.bean.repo is set, e.g., to 
> something like file:/path/to/crawler/policy/crawler-config.xml
> 
> 2. Make sure that /path/to/crawler/policy/crawler-config.xml has the configuration
> that you are trying to override (e.g., your new bean definitions).
> 
> HTH!
> 
> Cheers,
> Chris
> 
> On Apr 25, 2012, at 4:36 PM, Verma, Rishi (388J) wrote:
> 
>> Hi all,
>> 
>> I wrote a custom cas-crawler ProductCrawler, but I'm having some difficulty registering
my custom product crawler with cas-crawler.
>> 
>> I created a product crawler by extending StdProductCrawler, and I've added this product-crawler
name to crawler config files (following the example of StdProductCrawler):
>> * crawler/policy/crawler-beans.xml
>> * crawler/policy/cmd-line-option-beans.xml
>> 
>> However, after running the below command, I can clearly see my custom product crawler
(called LabCASProductCrawler) is not available. A crawler ingest try also tells me that there
is no "bean" by the name of my "LabCASProductCrawler" available:
>>> bash-3.2$ ./crawler_launcher —printSupportedCrawlers
>> ProductCrawlers:
>> Id: StdProductCrawler
>> Id: MetExtractorProductCrawler
>> Id: AutoDetectProductCrawler
>> 
>>> ./crawler_launcher --crawlerId LabCASProductCrawler --filemgrUrl http://localhost:9000
--productPath /data/staging/HGHAGA9 --failureDir /tmp/failed_ingest --metFileExtension met
—clientTransferer org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
>> Failed to parse options : No bean named 'LabCASProductCrawler' is defined
>> 
>> I noticed in files like crawler-config.xml and cmd-line-option-beans.xml, there were
references made to crawler config files stored in the cas-crawler JAR. Looking more into this,
it seems to me that crawler is pre-loading config files directly from that JAR and overshadowing
any of my config changes:
>> * crawler/lib/cas-crawler-0.3.jar:org/apache/oodt/cas/crawl/crawler-beans.xml
>> * crawler/lib/cas-crawler-0.3.jar:org/apache/oodt/cas/crawl/crawler-config.xml
>> 
>> So two questions:
>> 1. Am I editing the correct policy files, in order to register my custom product
crawler with cas-crawler?
>> 2. It seems the cas-crawler JAR contains crawler config files that take greater precedence
than the ones available for editing under crawler/policy. Is there a way around this?
>> 
>> Thanks!
>> rishi
> 
> 
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 

Mime
View raw message