oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Problem when running pushpull
Date Fri, 20 Jul 2012 17:29:43 GMT
Hi YunHee,

On Jul 19, 2012, at 11:19 AM, YunHee Kang wrote:

> I just found a problem that is in a file
> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml" in  oodt 0.4.
> 
> The problem is caused by missing classes related with an element
> "protocolFactory" in the file described above.
> To fix this problem,  It needs to replace a wrong class name with  a
> correct class name.
> For example  there is no class named
> org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory in any jar
> file in the directory,  $CAS_PP_HOME/lib.  To run the script
> $CAS_PP_HOME/bin/pushpull,   CogJGlobusFtpClientFactory  needs to be
> replaced with CogJGlobusFtpProtocolFactory.

Ahh gotcha. You've found a config file bug in oodt0.4. Can you please:

1. Create yourself an account at our JIRA issue tracker:
https://issues.apache.org/jira/browse/OODT

2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects
version to 0.4)

If you are willing to, we'd love to have a patch file from you that fixes
this config file. The patch process would be something like:

1. svn co http://svn.apache.org/repos/asf/oodt/trunk oodt0.5
2. edit odot0.5/pushpull/src/main/resources/policy/ProtocolFactoryInfo.xml and
fix XML policy file.
3. svn diff > OODT-xxx.ykang.yyMMdd.patch.txt

where xxx is the JIRA issue ID from #2 above.
4. attach your patch to the JIRA issue from #2.

> 
> But I don't know how to replace
> org.apache.oodt.cas.protocol.local.LocalClientFactory.

I'm not sure you have to -- this is probably OK.

> To be honest, I am not familiar with OODT.
> 
> 
> <protocol type="ftp">
>        <!--protocolFactory
> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory"/-->
>        <!-- cas-protocol-ftp-0.4.jar -->
>        <protocolFactory
> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpProtocolFactory"/>
>        <!--protocolFactory
> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/-->
>        <protocolFactory
> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/>
>    </protocol>
> 
> <protocol type="http">
>        <!--protocolFactory
> class="org.apache.oodt.cas.protocol.http.HttpClientFactory"/-->
>        <!-- cas-protocol-http-0.4.jar-->
>        <protocolFactory
> class="org.apache.oodt.cas.protocol.http.HttpProtocolFactory"/>
>    </protocol>
> 
>    <protocol type="sftp">
>        <!--protocolFactory
> class="org.apache.oodt.cas.protocol.sftp.JschSftpClientFactory"/-->
>        <!-- cas-protocol-sftp-0.4.jar -->
>        <protocolFactory
> class="org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory"/>
>    </protocol>
> 
>    <protocol type="imaps">
>        <!--protocolFactory
> class="org.apache.oodt.cas.protocol.imaps.ImapsClientFactory"/ -->
>        <!-- cas-protocol-sftp-0.4.jar -->
>        <protocolFactory
> class="org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory"/>
>    </protocol>
> 
>    <!--protocol type="file">
>        <protocolFactory
> class="org.apache.oodt.cas.protocol.local.LocalClientFactory"/>
>    </protocol>
>    -->
> 

Brian Foster -- any idea here?

> But  up to now I am having a trouble to run the script
> $CAS_PP_HOME/bin/pushpull  after I changed the file
> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml".
> 
> The following is the part of the error message when running the script.
> SEVERE: Failed to finish downloading per property files
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
> : null

Ahh, what FTP site are you trying to access? That way I can try and 
test this myself in my own local checkout.

> 
> I think that the error may occur the file RetrievalSetup.java in the
> following directory:
> ./apache-oodt-0.4-src/pushpull/src/main/java/org/apache/oodt/cas/pushpull/retrievalsystem/RetrievalSetup.java

Yep might be -- the problem is that many remote FTP sites have their
own intricacies and some protocol libraries handle it better than others.
That's why we integrate many of these libraries and try to allow others
to plug in their own -- we may need a specific FTP library for this site
or hopefully we can find a protocol factory implementation that works.

> 
> Please let me know how to fix the problem.
> 
> 
> yhkang@yunhee-kisti:~/oodt/cas-pushpull/bin$ ./pushpull
> Jul 20, 2012 3:06:44 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure
> INFO: Configuring DaemonLauncher. . .
> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
> loadProtocolTypes
> INFO: Associating protocol types with ProtocolFactories . . .
> Jul 20, 2012 3:06:44 AM
> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
> INFO: Assiging protocol 'ftp' with class 'class
> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory'
> Jul 20, 2012 3:06:44 AM
> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
> INFO: Assiging protocol 'http' with class 'class
> org.apache.oodt.cas.protocol.http.HttpProtocolFactory'
> Jul 20, 2012 3:06:44 AM
> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
> INFO: Assiging protocol 'sftp' with class 'class
> org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory'
> Jul 20, 2012 3:06:44 AM
> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
> INFO: Assiging protocol 'imaps' with class 'class
> org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory'
> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
> loadParserInfo
> INFO: Associating parsers with RetrievalMethods . . .
> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
> loadParserInfo
> INFO: Assiging parser
> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser'
> with retrievalmethod
> 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler'
> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
> loadParserInfo
> INFO: Assiging parser
> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.FileListParser'
> with retrievalmethod
> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever'
> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
> loadParserInfo
> INFO: Assiging parser
> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.ClassNoaaEmailParser'
> with retrievalmethod
> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever'
> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config loadIngester
> INFO: Configuring and building ingester:
> [org.apache.oodt.cas.filemgr.ingest.StdIngester]: data transfer:
> [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]:
> to ingest to file manager: [http://localhost:9000]
> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config loadIngester
> INFO: Configuring Ingester cache: []
> TRANSFER: org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
> loadProductTypeDetection
> INFO: Loading product type detection file:
> /home/yhkang/oodt/cas-pushpull/policy/mimetypes.xml
> Jul 20, 2012 3:06:44 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure
> INFO: Loading SiteInfo file
> '/home/yhkang/oodt/cas-pushpull/etc/examples/RemoteSpecsFiles/RemoteSpecs.xml'
> Jul 20, 2012 3:06:44 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons
> INFO: Launching Daemons . . .
> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> registerRMIServer
> INFO: Created Daemon ID = 91131 on RMI registry port 9113
> Jul 20, 2012 3:06:44 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons
> INFO: Creating Daemon with ID = 1
> Jul 20, 2012 3:06:44 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning
> INFO: Daemon with ID = 91131 was given permission to run
> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
> Jul 20, 2012 3:06:44 AM
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
> retrieveFiles
> INFO: Creating 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler'
> to download data files
> Jul 20, 2012 3:06:49 AM
> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
> INFO: Testing protocol
> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this may
> take a few minutes . . .
> Jul 20, 2012 3:06:59 AM
> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
> username 'anonymous'
> Jul 20, 2012 3:06:59 AM
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
> retrieveFiles
> SEVERE: Failed to finish downloading per property files
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
> : null
> Jul 20, 2012 3:06:59 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to
> sleep until Fri Jul 20 03:09:00 KST 2012
> 
Thanks let me know what you think above!

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mime
View raw message