oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cameron Goodale <good...@apache.org>
Subject Re: Problem when running pushpull
Date Mon, 23 Jul 2012 00:16:39 GMT
YunHee,

Thank you for testing out the PushPull User Guide.  I have been waiting for
someone to run through it and have success (which it appears you have).  I
just added in a small block about how to Launch the PushPull Daemon, but it
seems that you got it running already.

If there is anything that you think should be added to the User Guide, or
if any part was confusing I would appreciate hearing from you.

Cheers!


Cameron

On Sat, Jul 21, 2012 at 10:15 PM, YunHee Kang <yunh.kang@gmail.com> wrote:

> HI chris,
>
> > Ahh gotcha. You've found a config file bug in oodt0.4. Can you please:
> >
> > 1. Create yourself an account at our JIRA issue tracker:
> > https://issues.apache.org/jira/browse/OODT
> >
> > 2. File an issue (create a new issue/bug) against oodt 0.4 (set the
> affects
> > version to 0.4)
> >
> I just created  the isse named "OODT-474"  related with config file
> bug in oodt0.4.
> It is the first time to write an issue report.  Can you check the
> issue that I wrote ?
>
>
> > Ahh, what FTP site are you trying to access? That way I can try and
> > test this myself in my own local checkout.
>
> I tried to test downloading a file  based on WIKI site
> "https://cwiki.apache.org/OODT/oodt-push-pull-user-guide.html". There
> is no problem to download the file  as follows:
>  $ ftp l4ftl01.larc.nasa.gov
> Connected to l4ftl01.larc.nasa.gov.
> 220-
> 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 220-
> 220- This US Government computer is for authorized users only.
> 220- By accessing this system you are consenting to complete
> 220- monitoring with no expectation of privacy. Unauthorized
> 220- access or use may subject you to disciplinary action and
> 220- criminal prosecution.
> 220-
> 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 220-
> 220-
> 220-
> 220 FTP server ready.
> Name (l4ftl01.larc.nasa.gov:yhkang): anonymous
> 331 Guest login ok, send your complete e-mail address as password.
> Password:
> 230 Guest login ok, access restrictions apply.
> Remote system type is UNIX.
> Using binary mode to transfer files.
> ftp> cd /TES/TL2CO2N.005
> 250 CWD command successful.
> ftp> cd 2004.09.20
> 250 CWD command successful.
> ftp> get TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5
> local: TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 remote:
> TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5
> 200 PORT command successful.
> 150 Opening BINARY mode data connection for
> TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 (92150280 bytes).
> 226 Transfer complete. Checksum matched.
> 92150280 bytes received in 171.84 secs (523.7 kB/s)
>
>
> Thanks,
>
> Yunhee
>
> 2012/7/21 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
> > Hi YunHee,
> >
> > On Jul 19, 2012, at 11:19 AM, YunHee Kang wrote:
> >
> >> I just found a problem that is in a file
> >> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml" in  oodt 0.4.
> >>
> >> The problem is caused by missing classes related with an element
> >> "protocolFactory" in the file described above.
> >> To fix this problem,  It needs to replace a wrong class name with  a
> >> correct class name.
> >> For example  there is no class named
> >> org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory in any jar
> >> file in the directory,  $CAS_PP_HOME/lib.  To run the script
> >> $CAS_PP_HOME/bin/pushpull,   CogJGlobusFtpClientFactory  needs to be
> >> replaced with CogJGlobusFtpProtocolFactory.
> >
> > Ahh gotcha. You've found a config file bug in oodt0.4. Can you please:
> >
> > 1. Create yourself an account at our JIRA issue tracker:
> > https://issues.apache.org/jira/browse/OODT
> >
> > 2. File an issue (create a new issue/bug) against oodt 0.4 (set the
> affects
> > version to 0.4)
> >
> > If you are willing to, we'd love to have a patch file from you that fixes
> > this config file. The patch process would be something like:
> >
> > 1. svn co http://svn.apache.org/repos/asf/oodt/trunk oodt0.5
> > 2. edit
> odot0.5/pushpull/src/main/resources/policy/ProtocolFactoryInfo.xml and
> > fix XML policy file.
> > 3. svn diff > OODT-xxx.ykang.yyMMdd.patch.txt
> >
> > where xxx is the JIRA issue ID from #2 above.
> > 4. attach your patch to the JIRA issue from #2.
> >
> >>
> >> But I don't know how to replace
> >> org.apache.oodt.cas.protocol.local.LocalClientFactory.
> >
> > I'm not sure you have to -- this is probably OK.
> >
> >> To be honest, I am not familiar with OODT.
> >>
> >>
> >> <protocol type="ftp">
> >>        <!--protocolFactory
> >> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory"/-->
> >>        <!-- cas-protocol-ftp-0.4.jar -->
> >>        <protocolFactory
> >> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpProtocolFactory"/>
> >>        <!--protocolFactory
> >> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/-->
> >>        <protocolFactory
> >> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/>
> >>    </protocol>
> >>
> >> <protocol type="http">
> >>        <!--protocolFactory
> >> class="org.apache.oodt.cas.protocol.http.HttpClientFactory"/-->
> >>        <!-- cas-protocol-http-0.4.jar-->
> >>        <protocolFactory
> >> class="org.apache.oodt.cas.protocol.http.HttpProtocolFactory"/>
> >>    </protocol>
> >>
> >>    <protocol type="sftp">
> >>        <!--protocolFactory
> >> class="org.apache.oodt.cas.protocol.sftp.JschSftpClientFactory"/-->
> >>        <!-- cas-protocol-sftp-0.4.jar -->
> >>        <protocolFactory
> >> class="org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory"/>
> >>    </protocol>
> >>
> >>    <protocol type="imaps">
> >>        <!--protocolFactory
> >> class="org.apache.oodt.cas.protocol.imaps.ImapsClientFactory"/ -->
> >>        <!-- cas-protocol-sftp-0.4.jar -->
> >>        <protocolFactory
> >> class="org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory"/>
> >>    </protocol>
> >>
> >>    <!--protocol type="file">
> >>        <protocolFactory
> >> class="org.apache.oodt.cas.protocol.local.LocalClientFactory"/>
> >>    </protocol>
> >>    -->
> >>
> >
> > Brian Foster -- any idea here?
> >
> >> But  up to now I am having a trouble to run the script
> >> $CAS_PP_HOME/bin/pushpull  after I changed the file
> >> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml".
> >>
> >> The following is the part of the error message when running the script.
> >> SEVERE: Failed to finish downloading per property files
> >>
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
> >> : null
> >
>
> >
> >>
> >> I think that the error may occur the file RetrievalSetup.java in the
> >> following directory:
> >>
> ./apache-oodt-0.4-src/pushpull/src/main/java/org/apache/oodt/cas/pushpull/retrievalsystem/RetrievalSetup.java
> >
> > Yep might be -- the problem is that many remote FTP sites have their
> > own intricacies and some protocol libraries handle it better than others.
> > That's why we integrate many of these libraries and try to allow others
> > to plug in their own -- we may need a specific FTP library for this site
> > or hopefully we can find a protocol factory implementation that works.
> >
> >>
> >> Please let me know how to fix the problem.
> >>
> >>
> >> yhkang@yunhee-kisti:~/oodt/cas-pushpull/bin$ ./pushpull
> >> Jul 20, 2012 3:06:44 AM
> >> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure
> >> INFO: Configuring DaemonLauncher. . .
> >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
> >> loadProtocolTypes
> >> INFO: Associating protocol types with ProtocolFactories . . .
> >> Jul 20, 2012 3:06:44 AM
> >> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
> >> INFO: Assiging protocol 'ftp' with class 'class
> >> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory'
> >> Jul 20, 2012 3:06:44 AM
> >> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
> >> INFO: Assiging protocol 'http' with class 'class
> >> org.apache.oodt.cas.protocol.http.HttpProtocolFactory'
> >> Jul 20, 2012 3:06:44 AM
> >> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
> >> INFO: Assiging protocol 'sftp' with class 'class
> >> org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory'
> >> Jul 20, 2012 3:06:44 AM
> >> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType
> >> INFO: Assiging protocol 'imaps' with class 'class
> >> org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory'
> >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
> >> loadParserInfo
> >> INFO: Associating parsers with RetrievalMethods . . .
> >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
> >> loadParserInfo
> >> INFO: Assiging parser
> >>
> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser'
> >> with retrievalmethod
> >> 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler'
> >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
> >> loadParserInfo
> >> INFO: Assiging parser
> >> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.FileListParser'
> >> with retrievalmethod
> >> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever'
> >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo
> >> loadParserInfo
> >> INFO: Assiging parser
> >>
> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.ClassNoaaEmailParser'
> >> with retrievalmethod
> >> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever'
> >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
> loadIngester
> >> INFO: Configuring and building ingester:
> >> [org.apache.oodt.cas.filemgr.ingest.StdIngester]: data transfer:
> >> [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]:
> >> to ingest to file manager: [http://localhost:9000]
> >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
> loadIngester
> >> INFO: Configuring Ingester cache: []
> >> TRANSFER:
> org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
> >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config
> >> loadProductTypeDetection
> >> INFO: Loading product type detection file:
> >> /home/yhkang/oodt/cas-pushpull/policy/mimetypes.xml
> >> Jul 20, 2012 3:06:44 AM
> >> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure
> >> INFO: Loading SiteInfo file
> >>
> '/home/yhkang/oodt/cas-pushpull/etc/examples/RemoteSpecsFiles/RemoteSpecs.xml'
> >> Jul 20, 2012 3:06:44 AM
> >> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons
> >> INFO: Launching Daemons . . .
> >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> >> registerRMIServer
> >> INFO: Created Daemon ID = 91131 on RMI registry port 9113
> >> Jul 20, 2012 3:06:44 AM
> >> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons
> >> INFO: Creating Daemon with ID = 1
> >> Jul 20, 2012 3:06:44 AM
> >> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning
> >> INFO: Daemon with ID = 91131 was given permission to run
> >> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
> >> Jul 20, 2012 3:06:44 AM
> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
> >> retrieveFiles
> >> INFO: Creating
> 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler'
> >> to download data files
> >> Jul 20, 2012 3:06:49 AM
> >> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
> >> INFO: Testing protocol
> >> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this may
> >> take a few minutes . . .
> >> Jul 20, 2012 3:06:59 AM
> >> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
> >> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
> >> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
> >> username 'anonymous'
> >> Jul 20, 2012 3:06:59 AM
> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
> >> retrieveFiles
> >> SEVERE: Failed to finish downloading per property files
> >>
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
> >> : null
> >> Jul 20, 2012 3:06:59 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> >> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to
> >> sleep until Fri Jul 20 03:09:00 KST 2012
> >>
> > Thanks let me know what you think above!
> >
> > Cheers,
> > Chris
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattmann@nasa.gov
> > WWW:   http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message