oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Foster <holeno...@mac.com>
Subject Fwd: Problem when running pushpull
Date Thu, 26 Jul 2012 03:32:00 GMT
hey Yunhee,

i just realized that your probably not on the dev list and my last email just went to the
list and didn't have you cc'ed

try the patch attached to this issue mentioned below...

-brian´╗┐

Begin forwarded message:

> From: Brian Foster <holenoter@mac.com>
> Date: July 25, 2012 12:43:10 AM
> To: dev@oodt.apache.org,dev@oodt.apache.org
> Subject: Re: Problem when running pushpull
>
> hey Yunhee,
>
> Try patching your pushpull code with the fix i've attached to issue:
>
> https://issues.apache.org/jira/browse/OODT-476
>
> -brian
>
> On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.kang@gmail.com> wrote:
>
>> Hi Brian,
>>
>> I changed the file RetrievalSetup.java to get the stacktrace.
>> So I got a new cas-pushpull-0.4.jar after rebuilt it.
>>
>> The following is a message including the stacktrace with information
>> about excpetion propation.
>>
>> SEVERE: Failed to finish downloading per property files
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>> : null
>> java.lang.NullPointerException
>> at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367)
>> at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311)
>> at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300)
>> at org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103)
>> at org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Please let me know what java programs I check frist.
>>
>> Thanks,
>>
>> Yunhee
>> 2012/7/25 Brian Foster <holenoter@mac.com>:
>> > YunHee,
>> >
>> > There is a NullPointerException being thrown... on line 118 of
>> > org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change it
>> >
>> > rebuild and deploy and that should give you the stacktrace which will answer
>> > what is going on here.
>> >
>> > -brian
>> >
>> >
>> > On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)"
>> > <chris.a.mattmann@jpl.nasa.gov> wrote:
>> >
>> > Hi YunHee,
>> >
>> > Thanks for your mail.
>> >
>> > Let me take a look at the configs again. Does anyone else using
>> > PushPull have any advice for YunHee here? That error message
>> > seems to indicate there is a config file error somewhere. I'm wondering
>> > if your etc/push_pull_framework.properties has all the right values
>> > in there. Did you go through Cameron's guide to getting Push Pull
>> > running on the wiki?
>> >
>> > Cheers,
>> > Chris
>> >
>> > On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote:
>> >
>> >> Hi Chris,
>> >>
>> >> There are some problems in the configuration files.
>> >> I found there is a dead link which does not connect two information
>> >> in two files :RemoteSpecs.xml and ExternalSources.xml.
>> >> For example an alias named "DewFtp" in RemoteSpecs.xml has no
>> >> associated source in "ExternalSources.xml.
>> >>
>> >> After I tailored those two XML files, I just tested one of
>> >> configurations, whose alias is PolarNcepNoaaFtp.
>> >> I also faced the same problem as follows:
>> >>
>> >> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with
>> >> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>> >> username 'anonymous'
>> >> Jul 25, 2012 2:21:04 AM
>> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>> >> retrieveFiles
>> >> SEVERE: Failed to finish downloading per property files
>> >>
>> >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>> >> : null
>> >> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>> >>
>> >>
>> >> Thanks,
>> >> Yunhee
>> >>
>> >> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>> >>> Hi YunHee,
>> >>>
>> >>> Thanks!
>> >>>
>> >>> Cheers,
>> >>> Chris
>> >>>
>> >>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote:
>> >>>
>> >>>> Hi Chris,
>> >>>>
>> >>>>> Thanks, can you try some of the other XML file configs.
>> >>>> Yes, I will try it and let you know its result.
>> >>>>
>> >>>>> Maybe it's being blocked in your country,
>> >>>> That site may be blocked.
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>> Yunhee
>> >>>>
>> >>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>> >>>>> Hi YunHee,
>> >>>>>
>> >>>>> Thanks, can you try some of the other XML file configs. Maybe
it's
>> >>>>> being
>> >>>>> blocked in your country, or something, but let's see if we can
get some
>> >>>>> of the other XML RemoteSpec configs to work?
>> >>>>>
>> >>>>> Cheers,
>> >>>>> Chris
>> >>>>>
>> >>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote:
>> >>>>>
>> >>>>>> Hi Chris,
>> >>>>>>
>> >>>>>>
>> >>>>>>> Try out the MODIS one there, does that work?
>> >>>>>> No, it doesn't.
>> >>>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection
>> >>>>>> problem.
>> >>>>>>
>> >>>>>> $ ftp e4ftl01u.ecs.nasa.gov
>> >>>>>> ftp: connect: Connection timed out
>> >>>>>> ftp>
>> >>>>>>
>> >>>>>> The message below also say the reason why the problem was
occurred.
>> >>>>>> WARNING: Error occurred while connecting to RemoteSite:
alias =
>> >>>>>> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username
=
>> >>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed
to
>> >>>>>> connect to server : Connection timed out
>> >>>>>> Jul 25, 2012 1:31:25 AM
>> >>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>> >>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp'
url =
>> >>>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir
=
>> >>>>>> 'null' maxConnections = '-1' in 5 seconds
>> >>>>>> Waiting . . . . .
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Yunhee
>> >>>>>>
>> >>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>> >>>>>>> Hi YunHee,
>> >>>>>>>
>> >>>>>>> OK, can you try some of the other files, e.g., the ones
in:
>> >>>>>>>
>> >>>>>>> http://s.apache.org/ks0
>> >>>>>>>
>> >>>>>>> Try out the MODIS one there, does that work?
>> >>>>>>>
>> >>>>>>> Cheers,
>> >>>>>>> Chris
>> >>>>>>>
>> >>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote:
>> >>>>>>>
>> >>>>>>>> Hi Chris,
>> >>>>>>>>
>> >>>>>>>> I added the header of XML as follows:
>> >>>>>>>> <?xml version="1.0"?>
>> >>>>>>>>
>> >>>>>>>> Next the value of elements was changed into specific
one.
>> >>>>>>>> The following is the output of the XML:
>> >>>>>>>> <?xml version="1.0"?>
>> >>>>>>>> <root>
>> >>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>> >>>>>>>> <nofiles/>
>> >>>>>>>> <!--dir name="\d{4}\.\d{2}\.\d{2}"-->
>> >>>>>>>> <dir name="2004\.09\.20">
>> >>>>>>>> <nodirs/>
>> >>>>>>>> <!--file
>> >>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/-->
>> >>>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/>
>> >>>>>>>> </dir>
>> >>>>>>>> </dirstruct>
>> >>>>>>>> </root>
>> >>>>>>>>
>> >>>>>>>> Unfortunately I got the same result before:
>> >>>>>>>> ...
>> >>>>>>>> INFO: Testing protocol
>> >>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
. . . this ma
>> >>>>>>>> y take a few minutes . . .
>> >>>>>>>> Jul 25, 2012 12:30:03 AM
>> >>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler
connect
>> >>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov
with
>> >>>>>>>> protocol 'org.apache.oodt.
>> >>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username
'anonymous'
>> >>>>>>>> Jul 25, 2012 12:30:03 AM
>> >>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
retrie
>> >>>>>>>> veFiles
>> >>>>>>>> SEVERE: Failed to finish downloading per property
files
>> >>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/
>> >>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml :
null
>> >>>>>>>> Jul 25, 2012 12:30:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>> >>>>>>>> sleep
>> >>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port
9113 is going to
>> >>>>>>>> sleep until Wed Jul 25 0
>> >>>>>>>> ...
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Thanks,
>> >>>>>>>> Yunhee
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>> >>>>>>>>> Hi YunHee,
>> >>>>>>>>>
>> >>>>>>>>> Can you try simply putting a specific <dir
name="..."> instead of
>> >>>>>>>>> \d{4}, etc., and
>> >>>>>>>>> then can you put a specific file name in the
<file name="..">. If
>> >>>>>>>>> that works and
>> >>>>>>>>> downloads, then we know there is something up
with the regex
>> >>>>>>>>> somehow.
>> >>>>>>>>> Also can you try adding the <?xml version="1.0"?>
start to your
>> >>>>>>>>> file?
>> >>>>>>>>>
>> >>>>>>>>> Cheers,
>> >>>>>>>>> Chris
>> >>>>>>>>>
>> >>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Hi Chris,
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> The following is the content of TESL2CO2.xml.
>> >>>>>>>>>> $ cat
>> >>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>> >>>>>>>>>> <root>
>> >>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>> >>>>>>>>>> <nofiles/>
>> >>>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}">
>> >>>>>>>>>> <nodirs/>
>> >>>>>>>>>> <file
>> >>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/>
>> >>>>>>>>>> </dir>
>> >>>>>>>>>> </dirstruct>
>> >>>>>>>>>> </root>
>> >>>>>>>>>>
>> >>>>>>>>>> Let me know if there is a problem in the
configuration file.
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks,
>> >>>>>>>>>> Yunhee
>> >>>>>>>>>>
>> >>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J)
>> >>>>>>>>>> <chris.a.mattmann@jpl.nasa.gov>:
>> >>>>>>>>>>> Hi YunHee,
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee
Kang wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hi Chris,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I am stuck on the problem related
with "pushpull" script
>> >>>>>>>>>>>> occurred
>> >>>>>>>>>>>> the following error message:
>> >>>>>>>>>>>> ....
>> >>>>>>>>>>>> INFO: Testing protocol
>> >>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
. . .
>> >>>>>>>>>>>> this may
>> >>>>>>>>>>>> take a few minutes . . .
>> >>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>> >>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler
connect
>> >>>>>>>>>>>> INFO: Successfully connected to
ftp://l4ftl01.larc.nasa.gov with
>> >>>>>>>>>>>> protocol
>> >>>>>>>>>>>> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol'
and
>> >>>>>>>>>>>> username 'anonymous'
>> >>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>> >>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>> >>>>>>>>>>>> retrieveFiles
>> >>>>>>>>>>>> SEVERE: Failed to finish downloading
per property files
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>> >>>>>>>>>>>> : null
>> >>>>>>>>>>>
>> >>>>>>>>>>> Can you do me a favor and simply run
this command:
>> >>>>>>>>>>>
>> >>>>>>>>>>> cat
>> >>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>> >>>>>>>>>>>
>> >>>>>>>>>>> What do you get?
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>> >>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon
sleep
>> >>>>>>>>>>>> INFO: Daemon with ID = 91131 on
RMI registry port 9113 is going
>> >>>>>>>>>>>> to
>> >>>>>>>>>>>> sleep until Tue Jul 24 16:06:00
KST 2012
>> >>>>>>>>>>>> Jul 24, 2012 4:06:00 PM
>> >>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
>> >>>>>>>>>>>> INFO: Daemon with ID = 91131 was
given permission to run
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> First of all, I would like test
the pushpull framework to move
>> >>>>>>>>>>>> data
>> >>>>>>>>>>>> written in NetCDF from a remote
site by a network protocols
>> >>>>>>>>>>>> (like ftp,
>> >>>>>>>>>>>> http, etc).
>> >>>>>>>>>>>
>> >>>>>>>>>>> Yep, sounds good.
>> >>>>>>>>>>>
>> >>>>>>>>>>>> I think that there is a test suite
which can be used to check
>> >>>>>>>>>>>> the
>> >>>>>>>>>>>> pushpull framework.
>> >>>>>>>>>>>> Can you give configuration files(a
set of XML files) that I can
>> >>>>>>>>>>>> test
>> >>>>>>>>>>>> downloading files from a FTP site
(public anonymous site) ?
>> >>>>>>>>>>>
>> >>>>>>>>>>> The examples in the resources directory
that you found should
>> >>>>>>>>>>> work
>> >>>>>>>>>>> so we'll just need to figure out why
they aren't working on your
>> >>>>>>>>>>> system.
>> >>>>>>>>>>>
>> >>>>>>>>>>>> or
>> >>>>>>>>>>>> Please let me know a reference implementation
of pushpull
>> >>>>>>>>>>>> framework
>> >>>>>>>>>>>> that I can use to move NetCDF files.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks we'll figure this out.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Cheers,
>> >>>>>>>>>>> Chris
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>>>>>> Chris Mattmann, Ph.D.
>> >>>>>>>>>>> Senior Computer Scientist
>> >>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena,
CA 91109 USA
>> >>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>> >>>>>>>>>>> Email: chris.a.mattmann@nasa.gov
>> >>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>> >>>>>>>>>>>
>> >>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>>>>>> Adjunct Assistant Professor, Computer
Science Department
>> >>>>>>>>>>> University of Southern California, Los
Angeles, CA 90089 USA
>> >>>>>>>>>>>
>> >>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>>>> Chris Mattmann, Ph.D.
>> >>>>>>>>> Senior Computer Scientist
>> >>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA
91109 USA
>> >>>>>>>>> Office: 171-266B, Mailstop: 171-246
>> >>>>>>>>> Email: chris.a.mattmann@nasa.gov
>> >>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>> >>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>>>> Adjunct Assistant Professor, Computer Science
Department
>> >>>>>>>>> University of Southern California, Los Angeles,
CA 90089 USA
>> >>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>> Chris Mattmann, Ph.D.
>> >>>>>>> Senior Computer Scientist
>> >>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>>>>>> Office: 171-266B, Mailstop: 171-246
>> >>>>>>> Email: chris.a.mattmann@nasa.gov
>> >>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>> >>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>> Adjunct Assistant Professor, Computer Science Department
>> >>>>>>> University of Southern California, Los Angeles, CA 90089
USA
>> >>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>>
>> >>>>>
>> >>>>>
>> >>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>> Chris Mattmann, Ph.D.
>> >>>>> Senior Computer Scientist
>> >>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>>>> Office: 171-266B, Mailstop: 171-246
>> >>>>> Email: chris.a.mattmann@nasa.gov
>> >>>>> WWW: http://sunset.usc.edu/~mattmann/
>> >>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>> Adjunct Assistant Professor, Computer Science Department
>> >>>>> University of Southern California, Los Angeles, CA 90089 USA
>> >>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>
>> >>>
>> >>>
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> Chris Mattmann, Ph.D.
>> >>> Senior Computer Scientist
>> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>> Office: 171-266B, Mailstop: 171-246
>> >>> Email: chris.a.mattmann@nasa.gov
>> >>> WWW: http://sunset.usc.edu/~mattmann/
>> >>> Phone: +1 (818) 354-8810
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> Adjunct Assistant Professor, Computer Science Department
>> >>> University of Southern California, Los Angeles, CA 90089 USA
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>
>> >
>> >
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Chris Mattmann, Ph.D.
>> > Senior Computer Scientist
>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > Office: 171-266B, Mailstop: 171-246
>> > Email: chris.a.mattmann@nasa.gov
>> > WWW: http://sunset.usc.edu/~mattmann/
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Adjunct Assistant Professor, Computer Science Department
>> > University of Southern California, Los Angeles, CA 90089 USA
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
    • Unnamed multipart/related (inline, None, 0 bytes)
View raw message