oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From YunHee Kang <yunh.k...@gmail.com>
Subject Re: Problem when running pushpull
Date Sun, 22 Jul 2012 05:15:47 GMT
HI chris,

> Ahh gotcha. You've found a config file bug in oodt0.4. Can you please:
>
> 1. Create yourself an account at our JIRA issue tracker:
> https://issues.apache.org/jira/browse/OODT
>
> 2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects
> version to 0.4)
>
I just created  the isse named "OODT-474"  related with config file
bug in oodt0.4.
It is the first time to write an issue report.  Can you check the
issue that I wrote ?


> Ahh, what FTP site are you trying to access? That way I can try and
> test this myself in my own local checkout.

I tried to test downloading a file  based on WIKI site
"https://cwiki.apache.org/OODT/oodt-push-pull-user-guide.html". There
is no problem to download the file  as follows:
 $ ftp l4ftl01.larc.nasa.gov
Connected to l4ftl01.larc.nasa.gov.
220-
220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
220-
220- This US Government computer is for authorized users only.
220- By accessing this system you are consenting to complete
220- monitoring with no expectation of privacy. Unauthorized
220- access or use may subject you to disciplinary action and
220- criminal prosecution.
220-
220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
220-
220-
220-
220 FTP server ready.
Name (l4ftl01.larc.nasa.gov:yhkang): anonymous
331 Guest login ok, send your complete e-mail address as password.
Password:
230 Guest login ok, access restrictions apply.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd /TES/TL2CO2N.005
250 CWD command successful.
ftp> cd 2004.09.20
250 CWD command successful.
ftp> get TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5
local: TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 remote:
TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5
200 PORT command successful.
150 Opening BINARY mode data connection for
TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 (92150280 bytes).
226 Transfer complete. Checksum matched.
92150280 bytes received in 171.84 secs (523.7 kB/s)


Thanks,

Yunhee

2012/7/21 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
> Hi YunHee,
>
> On Jul 19, 2012, at 11:19 AM, YunHee Kang wrote:
>
>> I just found a problem that is in a file
>> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml" in  oodt 0.4.
>>
>> The problem is caused by missing classes related with an element
>> "protocolFactory" in the file described above.
>> To fix this problem,  It needs to replace a wrong class name with  a
>> correct class name.
>> For example  there is no class named
>> org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory in any jar
>> file in the directory,  $CAS_PP_HOME/lib.  To run the script
>> $CAS_PP_HOME/bin/pushpull,   CogJGlobusFtpClientFactory  needs to be
>> replaced with CogJGlobusFtpProtocolFactory.
>
> Ahh gotcha. You've found a config file bug in oodt0.4. Can you please:
>
> 1. Create yourself an account at our JIRA issue tracker:
> https://issues.apache.org/jira/browse/OODT
>
> 2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects
> version to 0.4)
>
> If you are willing to, we'd love to have a patch file from you that fixes
> this config file. The patch process would be something like:
>
> 1. svn co http://svn.apache.org/repos/asf/oodt/trunk oodt0.5
> 2. edit odot0.5/pushpull/src/main/resources/policy/ProtocolFactoryInfo.xml and
> fix XML policy file.
> 3. svn diff > OODT-xxx.ykang.yyMMdd.patch.txt
>
> where xxx is the JIRA issue ID from #2 above.
> 4. attach your patch to the JIRA issue from #2.
>
>>
>> But I don't know how to replace
>> org.apache.oodt.cas.protocol.local.LocalClientFactory.
>
> I'm not sure you have to -- this is probably OK.
>
>> To be honest, I am not familiar with OODT.
>>
>>
>> <protocol type="ftp">
>>        <!--protocolFactory
>> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory"/-->
>>        <!-- cas-protocol-ftp-0.4.jar -->
>>        <protocolFactory
>> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpProtocolFactory"/>
>>        <!--protocolFactory
>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/-->
>>        <protocolFactory
>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/>
>>    </protocol>
>>
>> <protocol type="http">
>>        <!--protocolFactory
>> class="org.apache.oodt.cas.protocol.http.HttpClientFactory"/-->
>>        <!-- cas-protocol-http-0.4.jar-->
>>        <protocolFactory
>> class="org.apache.oodt.cas.protocol.http.HttpProtocolFactory"/>
>>    </protocol>
>>
>>    <protocol type="sftp">
>>        <!--protocolFactory
>> class="org.apache.oodt.cas.protocol.sftp.JschSftpClientFactory"/-->
>>        <!-- cas-protocol-sftp-0.4.jar -->
>>        <protocolFactory
>> class="org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory"/>
>>    </protocol>
>>
>>    <protocol type="imaps">
>>        <!--protocolFactory
>> class="org.apache.oodt.cas.protocol.imaps.ImapsClientFactory"/ -->
>>        <!-- cas-protocol-sftp-0.4.jar -->
>>        <protocolFactory
>> class="org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory"/>
>>    </protocol>
>>
>>    <!--protocol type="file">
>>        <protocolFactory
>> class="org.apache.oodt.cas.protocol.local.LocalClientFactory"/>
>>    </protocol>
>>    -->
>>
>
> Brian Foster -- any idea here?
>
>> But  up to now I am having a trouble to run the script
>> $CAS_PP_HOME/bin/pushpull  after I changed the file
>> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml".
>>
>> The following is the part of the error message when running the script.
>> SEVERE: Failed to finish downloading per property files
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>> : null
>

>
>>
>> I think that the error may occur the file RetrievalSetup.java in the
>> following directory:
>> ./apache-oodt-0.4-src/pushpull/src/main/java/org/apache/oodt/cas/pushpull/retrievalsystem/RetrievalSetup.java
>
> Yep might be -- the problem is that many remote FTP sites have their
> own intricacies and some protocol libraries handle it better than others.
> That's why we integrate many of these libraries and try to allow others
> to plug in their own -- we may need a specific FTP library for this site
> or hopefully we can find a protocol factory implementation that works.
>
>>
>> Please let me know how to fix the problem.
>>
>>
>> yhkang@yunhee-kisti:~/oodt/cas-pushpull/bin$ ./pushpull
>> Jul 20, 2012 3:06:44 AM
>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure
>> INFO: Configuring DaemonLauncher. . .
>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
>> loadProtocolTypes
>> INFO: Associating protocol types with ProtocolFactories . . .
>> Jul 20, 2012 3:06:44 AM
>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
>> INFO: Assiging protocol 'ftp' with class 'class
>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory'
>> Jul 20, 2012 3:06:44 AM
>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
>> INFO: Assiging protocol 'http' with class 'class
>> org.apache.oodt.cas.protocol.http.HttpProtocolFactory'
>> Jul 20, 2012 3:06:44 AM
>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
>> INFO: Assiging protocol 'sftp' with class 'class
>> org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory'
>> Jul 20, 2012 3:06:44 AM
>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
>> INFO: Assiging protocol 'imaps' with class 'class
>> org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory'
>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
>> loadParserInfo
>> INFO: Associating parsers with RetrievalMethods . . .
>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
>> loadParserInfo
>> INFO: Assiging parser
>> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser'
>> with retrievalmethod
>> 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler'
>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
>> loadParserInfo
>> INFO: Assiging parser
>> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.FileListParser'
>> with retrievalmethod
>> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever'
>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
>> loadParserInfo
>> INFO: Assiging parser
>> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.ClassNoaaEmailParser'
>> with retrievalmethod
>> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever'
>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config loadIngester
>> INFO: Configuring and building ingester:
>> [org.apache.oodt.cas.filemgr.ingest.StdIngester]: data transfer:
>> [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]:
>> to ingest to file manager: [http://localhost:9000]
>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config loadIngester
>> INFO: Configuring Ingester cache: []
>> TRANSFER: org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
>> loadProductTypeDetection
>> INFO: Loading product type detection file:
>> /home/yhkang/oodt/cas-pushpull/policy/mimetypes.xml
>> Jul 20, 2012 3:06:44 AM
>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure
>> INFO: Loading SiteInfo file
>> '/home/yhkang/oodt/cas-pushpull/etc/examples/RemoteSpecsFiles/RemoteSpecs.xml'
>> Jul 20, 2012 3:06:44 AM
>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons
>> INFO: Launching Daemons . . .
>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>> registerRMIServer
>> INFO: Created Daemon ID = 91131 on RMI registry port 9113
>> Jul 20, 2012 3:06:44 AM
>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons
>> INFO: Creating Daemon with ID = 1
>> Jul 20, 2012 3:06:44 AM
>> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning
>> INFO: Daemon with ID = 91131 was given permission to run
>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>> Jul 20, 2012 3:06:44 AM
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>> retrieveFiles
>> INFO: Creating 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler'
>> to download data files
>> Jul 20, 2012 3:06:49 AM
>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
>> INFO: Testing protocol
>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this may
>> take a few minutes . . .
>> Jul 20, 2012 3:06:59 AM
>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>> username 'anonymous'
>> Jul 20, 2012 3:06:59 AM
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>> retrieveFiles
>> SEVERE: Failed to finish downloading per property files
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>> : null
>> Jul 20, 2012 3:06:59 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to
>> sleep until Fri Jul 20 03:09:00 KST 2012
>>
> Thanks let me know what you think above!
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>

Mime
View raw message