oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From YunHee Kang <yunh.k...@gmail.com>
Subject Re: Problem when running pushpull
Date Tue, 24 Jul 2012 07:30:30 GMT
Hi Chris,

I am stuck on the problem related with  "pushpull" script  occurred
the following error message:
....
INFO: Testing protocol
org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this may
take a few minutes . . .
Jul 24, 2012 4:05:02 PM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
username 'anonymous'
Jul 24, 2012 4:05:02 PM
org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
retrieveFiles
SEVERE: Failed to finish downloading per property files
/home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
: null
Jul 24, 2012 4:05:02 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to
sleep until Tue Jul 24 16:06:00 KST 2012
Jul 24, 2012 4:06:00 PM
org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning
INFO: Daemon with ID = 91131 was given permission to run
org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser

First of all, I would like test the pushpull framework  to move data
written in NetCDF from a remote site by a network protocols (like ftp,
http, etc).
I think  that there is a test suite which can be used to  check the
pushpull framework.
Can you give  configuration files(a set of XML files)  that I can test
 downloading  files from a  FTP site (public anonymous site) ?
or
Please let me know  a reference implementation of pushpull framework
that I can use to move NetCDF files.

Thanks,
Yunhee

2012/7/22 YunHee Kang <yunh.kang@gmail.com>:
> HI chris,
>
>> Ahh gotcha. You've found a config file bug in oodt0.4. Can you please:
>>
>> 1. Create yourself an account at our JIRA issue tracker:
>> https://issues.apache.org/jira/browse/OODT
>>
>> 2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects
>> version to 0.4)
>>
> I just created  the isse named "OODT-474"  related with config file
> bug in oodt0.4.
> It is the first time to write an issue report.  Can you check the
> issue that I wrote ?
>
>
>> Ahh, what FTP site are you trying to access? That way I can try and
>> test this myself in my own local checkout.
>
> I tried to test downloading a file  based on WIKI site
> "https://cwiki.apache.org/OODT/oodt-push-pull-user-guide.html". There
> is no problem to download the file  as follows:
>  $ ftp l4ftl01.larc.nasa.gov
> Connected to l4ftl01.larc.nasa.gov.
> 220-
> 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 220-
> 220- This US Government computer is for authorized users only.
> 220- By accessing this system you are consenting to complete
> 220- monitoring with no expectation of privacy. Unauthorized
> 220- access or use may subject you to disciplinary action and
> 220- criminal prosecution.
> 220-
> 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 220-
> 220-
> 220-
> 220 FTP server ready.
> Name (l4ftl01.larc.nasa.gov:yhkang): anonymous
> 331 Guest login ok, send your complete e-mail address as password.
> Password:
> 230 Guest login ok, access restrictions apply.
> Remote system type is UNIX.
> Using binary mode to transfer files.
> ftp> cd /TES/TL2CO2N.005
> 250 CWD command successful.
> ftp> cd 2004.09.20
> 250 CWD command successful.
> ftp> get TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5
> local: TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 remote:
> TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5
> 200 PORT command successful.
> 150 Opening BINARY mode data connection for
> TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 (92150280 bytes).
> 226 Transfer complete. Checksum matched.
> 92150280 bytes received in 171.84 secs (523.7 kB/s)
>
>
> Thanks,
>
> Yunhee
>
> 2012/7/21 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>> Hi YunHee,
>>
>> On Jul 19, 2012, at 11:19 AM, YunHee Kang wrote:
>>
>>> I just found a problem that is in a file
>>> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml" in  oodt 0.4.
>>>
>>> The problem is caused by missing classes related with an element
>>> "protocolFactory" in the file described above.
>>> To fix this problem,  It needs to replace a wrong class name with  a
>>> correct class name.
>>> For example  there is no class named
>>> org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory in any jar
>>> file in the directory,  $CAS_PP_HOME/lib.  To run the script
>>> $CAS_PP_HOME/bin/pushpull,   CogJGlobusFtpClientFactory  needs to be
>>> replaced with CogJGlobusFtpProtocolFactory.
>>
>> Ahh gotcha. You've found a config file bug in oodt0.4. Can you please:
>>
>> 1. Create yourself an account at our JIRA issue tracker:
>> https://issues.apache.org/jira/browse/OODT
>>
>> 2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects
>> version to 0.4)
>>
>> If you are willing to, we'd love to have a patch file from you that fixes
>> this config file. The patch process would be something like:
>>
>> 1. svn co http://svn.apache.org/repos/asf/oodt/trunk oodt0.5
>> 2. edit odot0.5/pushpull/src/main/resources/policy/ProtocolFactoryInfo.xml and
>> fix XML policy file.
>> 3. svn diff > OODT-xxx.ykang.yyMMdd.patch.txt
>>
>> where xxx is the JIRA issue ID from #2 above.
>> 4. attach your patch to the JIRA issue from #2.
>>
>>>
>>> But I don't know how to replace
>>> org.apache.oodt.cas.protocol.local.LocalClientFactory.
>>
>> I'm not sure you have to -- this is probably OK.
>>
>>> To be honest, I am not familiar with OODT.
>>>
>>>
>>> <protocol type="ftp">
>>>        <!--protocolFactory
>>> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory"/-->
>>>        <!-- cas-protocol-ftp-0.4.jar -->
>>>        <protocolFactory
>>> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpProtocolFactory"/>
>>>        <!--protocolFactory
>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/-->
>>>        <protocolFactory
>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/>
>>>    </protocol>
>>>
>>> <protocol type="http">
>>>        <!--protocolFactory
>>> class="org.apache.oodt.cas.protocol.http.HttpClientFactory"/-->
>>>        <!-- cas-protocol-http-0.4.jar-->
>>>        <protocolFactory
>>> class="org.apache.oodt.cas.protocol.http.HttpProtocolFactory"/>
>>>    </protocol>
>>>
>>>    <protocol type="sftp">
>>>        <!--protocolFactory
>>> class="org.apache.oodt.cas.protocol.sftp.JschSftpClientFactory"/-->
>>>        <!-- cas-protocol-sftp-0.4.jar -->
>>>        <protocolFactory
>>> class="org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory"/>
>>>    </protocol>
>>>
>>>    <protocol type="imaps">
>>>        <!--protocolFactory
>>> class="org.apache.oodt.cas.protocol.imaps.ImapsClientFactory"/ -->
>>>        <!-- cas-protocol-sftp-0.4.jar -->
>>>        <protocolFactory
>>> class="org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory"/>
>>>    </protocol>
>>>
>>>    <!--protocol type="file">
>>>        <protocolFactory
>>> class="org.apache.oodt.cas.protocol.local.LocalClientFactory"/>
>>>    </protocol>
>>>    -->
>>>
>>
>> Brian Foster -- any idea here?
>>
>>> But  up to now I am having a trouble to run the script
>>> $CAS_PP_HOME/bin/pushpull  after I changed the file
>>> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml".
>>>
>>> The following is the part of the error message when running the script.
>>> SEVERE: Failed to finish downloading per property files
>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>> : null
>>
>
>>
>>>
>>> I think that the error may occur the file RetrievalSetup.java in the
>>> following directory:
>>> ./apache-oodt-0.4-src/pushpull/src/main/java/org/apache/oodt/cas/pushpull/retrievalsystem/RetrievalSetup.java
>>
>> Yep might be -- the problem is that many remote FTP sites have their
>> own intricacies and some protocol libraries handle it better than others.
>> That's why we integrate many of these libraries and try to allow others
>> to plug in their own -- we may need a specific FTP library for this site
>> or hopefully we can find a protocol factory implementation that works.
>>
>>>
>>> Please let me know how to fix the problem.
>>>
>>>
>>> yhkang@yunhee-kisti:~/oodt/cas-pushpull/bin$ ./pushpull
>>> Jul 20, 2012 3:06:44 AM
>>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure
>>> INFO: Configuring DaemonLauncher. . .
>>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
>>> loadProtocolTypes
>>> INFO: Associating protocol types with ProtocolFactories . . .
>>> Jul 20, 2012 3:06:44 AM
>>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
>>> INFO: Assiging protocol 'ftp' with class 'class
>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory'
>>> Jul 20, 2012 3:06:44 AM
>>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
>>> INFO: Assiging protocol 'http' with class 'class
>>> org.apache.oodt.cas.protocol.http.HttpProtocolFactory'
>>> Jul 20, 2012 3:06:44 AM
>>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
>>> INFO: Assiging protocol 'sftp' with class 'class
>>> org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory'
>>> Jul 20, 2012 3:06:44 AM
>>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
>>> INFO: Assiging protocol 'imaps' with class 'class
>>> org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory'
>>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
>>> loadParserInfo
>>> INFO: Associating parsers with RetrievalMethods . . .
>>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
>>> loadParserInfo
>>> INFO: Assiging parser
>>> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser'
>>> with retrievalmethod
>>> 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler'
>>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
>>> loadParserInfo
>>> INFO: Assiging parser
>>> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.FileListParser'
>>> with retrievalmethod
>>> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever'
>>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
>>> loadParserInfo
>>> INFO: Assiging parser
>>> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.ClassNoaaEmailParser'
>>> with retrievalmethod
>>> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever'
>>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config loadIngester
>>> INFO: Configuring and building ingester:
>>> [org.apache.oodt.cas.filemgr.ingest.StdIngester]: data transfer:
>>> [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]:
>>> to ingest to file manager: [http://localhost:9000]
>>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config loadIngester
>>> INFO: Configuring Ingester cache: []
>>> TRANSFER: org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
>>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
>>> loadProductTypeDetection
>>> INFO: Loading product type detection file:
>>> /home/yhkang/oodt/cas-pushpull/policy/mimetypes.xml
>>> Jul 20, 2012 3:06:44 AM
>>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure
>>> INFO: Loading SiteInfo file
>>> '/home/yhkang/oodt/cas-pushpull/etc/examples/RemoteSpecsFiles/RemoteSpecs.xml'
>>> Jul 20, 2012 3:06:44 AM
>>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons
>>> INFO: Launching Daemons . . .
>>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>>> registerRMIServer
>>> INFO: Created Daemon ID = 91131 on RMI registry port 9113
>>> Jul 20, 2012 3:06:44 AM
>>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons
>>> INFO: Creating Daemon with ID = 1
>>> Jul 20, 2012 3:06:44 AM
>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning
>>> INFO: Daemon with ID = 91131 was given permission to run
>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>>> Jul 20, 2012 3:06:44 AM
>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>> retrieveFiles
>>> INFO: Creating 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler'
>>> to download data files
>>> Jul 20, 2012 3:06:49 AM
>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
>>> INFO: Testing protocol
>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this may
>>> take a few minutes . . .
>>> Jul 20, 2012 3:06:59 AM
>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>> username 'anonymous'
>>> Jul 20, 2012 3:06:59 AM
>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>> retrieveFiles
>>> SEVERE: Failed to finish downloading per property files
>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>> : null
>>> Jul 20, 2012 3:06:59 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to
>>> sleep until Fri Jul 20 03:09:00 KST 2012
>>>
>> Thanks let me know what you think above!
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>

Mime
View raw message