oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Problem when running pushpull
Date Tue, 24 Jul 2012 17:51:34 GMT
Hi YunHee,

Thanks for your mail. 

Let me take a look at the configs again. Does anyone else using
PushPull have any advice for YunHee here? That error message
seems to indicate there is a config file error somewhere. I'm wondering
if your etc/push_pull_framework.properties has all the right values
in there. Did you go through Cameron's guide to getting Push Pull
running on the wiki?

Cheers,
Chris

On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote:

> Hi Chris,
> 
> There are some problems in the configuration files.
> I found there is a dead link which does not  connect two information
> in  two files :RemoteSpecs.xml and ExternalSources.xml.
> For example  an alias named  "DewFtp" in  RemoteSpecs.xml has no
> associated source in "ExternalSources.xml.
> 
> After I tailored those two XML files,  I just tested one of
> configurations, whose alias is PolarNcepNoaaFtp.
> I also faced the same problem as follows:
> 
> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with
> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
> username 'anonymous'
> Jul 25, 2012 2:21:04 AM
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
> retrieveFiles
> SEVERE: Failed to finish downloading per property files
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
> : null
> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> 
> 
> Thanks,
> Yunhee
> 
> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>> Hi YunHee,
>> 
>> Thanks!
>> 
>> Cheers,
>> Chris
>> 
>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote:
>> 
>>> Hi Chris,
>>> 
>>>> Thanks, can you try some of the other XML file configs.
>>> Yes,  I will try it and let you know its result.
>>> 
>>>> Maybe it's being blocked in your country,
>>> That site may be blocked.
>>> 
>>> 
>>> Thanks,
>>> Yunhee
>>> 
>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>>>> Hi YunHee,
>>>> 
>>>> Thanks, can you try some of the other XML file configs. Maybe it's being
>>>> blocked in your country, or something, but let's see if we can get some
>>>> of the other XML RemoteSpec configs to work?
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote:
>>>> 
>>>>> Hi Chris,
>>>>> 
>>>>> 
>>>>>> Try out the MODIS one there, does that work?
>>>>> No, it doesn't.
>>>>> I think  that the ftp site  e4ftl01u.ecs.nasa.gov has a connection problem.
>>>>> 
>>>>> $ ftp e4ftl01u.ecs.nasa.gov
>>>>> ftp: connect: Connection timed out
>>>>> ftp>
>>>>> 
>>>>> The message below also say the reason why the problem was occurred.
>>>>> WARNING: Error occurred while connecting to RemoteSite: alias =
>>>>> 'ModisFtp'  url = 'ftp://e4ftl01u.ecs.nasa.gov'  username =
>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to
>>>>> connect to server : Connection timed out
>>>>> Jul 25, 2012 1:31:25 AM
>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp'  url =
>>>>> 'ftp://e4ftl01u.ecs.nasa.gov'  username = 'anonymous' cdTestDir =
>>>>> 'null' maxConnections = '-1' in 5 seconds
>>>>> Waiting . . . . .
>>>>> 
>>>>> Thanks,
>>>>> Yunhee
>>>>> 
>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>>>>>> Hi YunHee,
>>>>>> 
>>>>>> OK, can you try some of the other files, e.g., the ones in:
>>>>>> 
>>>>>> http://s.apache.org/ks0
>>>>>> 
>>>>>> Try out the MODIS one there, does that work?
>>>>>> 
>>>>>> Cheers,
>>>>>> Chris
>>>>>> 
>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote:
>>>>>> 
>>>>>>> Hi Chris,
>>>>>>> 
>>>>>>> I added the header of XML as follows:
>>>>>>> <?xml version="1.0"?>
>>>>>>> 
>>>>>>> Next the value of elements was changed into specific one.
>>>>>>> The following is the output of the XML:
>>>>>>> <?xml version="1.0"?>
>>>>>>> <root>
>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>     <nofiles/>
>>>>>>>     <!--dir name="\d{4}\.\d{2}\.\d{2}"-->
>>>>>>>     <dir name="2004\.09\.20">
>>>>>>>         <nodirs/>
>>>>>>>         <!--file
>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/-->
>>>>>>>         <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/>
>>>>>>>     </dir>
>>>>>>> </dirstruct>
>>>>>>> </root>
>>>>>>> 
>>>>>>> Unfortunately I got the same result before:
>>>>>>> ...
>>>>>>> INFO: Testing protocol
>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . .
this ma
>>>>>>>    y take a few minutes . . .
>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>>>>>> protocol 'org.apache.oodt.
>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous'
>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup retrie
>>>>>>>  veFiles
>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/
>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null
>>>>>>> Jul 25, 2012 12:30:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon
sleep
>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going
to
>>>>>>> sleep until Wed Jul 25 0
>>>>>>> ...
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Yunhee
>>>>>>> 
>>>>>>> 
>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>>>>>>>> Hi YunHee,
>>>>>>>> 
>>>>>>>> Can you try simply putting a specific <dir name="...">
instead of \d{4}, etc., and
>>>>>>>> then can you put a specific file name in the <file name="..">.
If that works and
>>>>>>>> downloads, then we know there is something up with the regex
somehow.
>>>>>>>> Also can you try adding the <?xml version="1.0"?> start
to your file?
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote:
>>>>>>>> 
>>>>>>>>> Hi Chris,
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> The following is the content of  TESL2CO2.xml.
>>>>>>>>> $ cat /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>> <root>
>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>>>    <nofiles/>
>>>>>>>>>    <dir name="\d{4}\.\d{2}\.\d{2}">
>>>>>>>>>        <nodirs/>
>>>>>>>>>        <file name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/>
>>>>>>>>>    </dir>
>>>>>>>>> </dirstruct>
>>>>>>>>> </root>
>>>>>>>>> 
>>>>>>>>> Let me know if there is a problem in the configuration
file.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Yunhee
>>>>>>>>> 
>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>>>>>>>>>> Hi YunHee,
>>>>>>>>>> 
>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Chris,
>>>>>>>>>>> 
>>>>>>>>>>> I am stuck on the problem related with  "pushpull"
script  occurred
>>>>>>>>>>> the following error message:
>>>>>>>>>>> ....
>>>>>>>>>>> INFO: Testing protocol
>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
. . . this may
>>>>>>>>>>> take a few minutes . . .
>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler
connect
>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov
with
>>>>>>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol'
and
>>>>>>>>>>> username 'anonymous'
>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>>>>>> retrieveFiles
>>>>>>>>>>> SEVERE: Failed to finish downloading per property
files
>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>> : null
>>>>>>>>>> 
>>>>>>>>>> Can you do me a favor and simply run this command:
>>>>>>>>>> 
>>>>>>>>>> cat /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>> 
>>>>>>>>>> What do you get?
>>>>>>>>>> 
>>>>>>>>>>> Jul 24, 2012 4:05:02 PM org.apache.oodt.cas.pushpull.daemon.Daemon
sleep
>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry
port 9113 is going to
>>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012
>>>>>>>>>>> Jul 24, 2012 4:06:00 PM
>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
>>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission
to run
>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>>>>>>>>>>> 
>>>>>>>>>>> First of all, I would like test the pushpull
framework  to move data
>>>>>>>>>>> written in NetCDF from a remote site by a network
protocols (like ftp,
>>>>>>>>>>> http, etc).
>>>>>>>>>> 
>>>>>>>>>> Yep, sounds good.
>>>>>>>>>> 
>>>>>>>>>>> I think  that there is a test suite which can
be used to  check the
>>>>>>>>>>> pushpull framework.
>>>>>>>>>>> Can you give  configuration files(a set of XML
files)  that I can test
>>>>>>>>>>> downloading  files from a  FTP site (public anonymous
site) ?
>>>>>>>>>> 
>>>>>>>>>> The examples in the resources directory that you
found should work
>>>>>>>>>> so we'll just need to figure out why they aren't
working on your system.
>>>>>>>>>> 
>>>>>>>>>>> or
>>>>>>>>>>> Please let me know  a reference implementation
of pushpull framework
>>>>>>>>>>> that I can use to move NetCDF files.
>>>>>>>>>> 
>>>>>>>>>> Thanks we'll figure this out.
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> Chris
>>>>>>>>>> 
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109
USA
>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>>>>> WWW:   http://sunset.usc.edu/~mattmann/
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>> University of Southern California, Los Angeles, CA
90089 USA
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>> Senior Computer Scientist
>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>>> WWW:   http://sunset.usc.edu/~mattmann/
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>> University of Southern California, Los Angeles, CA 90089
USA
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Chris Mattmann, Ph.D.
>>>>>> Senior Computer Scientist
>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>> WWW:   http://sunset.usc.edu/~mattmann/
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> 
>>>> 
>>>> 
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Senior Computer Scientist
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 171-266B, Mailstop: 171-246
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW:   http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Assistant Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> 
>> 
>> 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> Phone: +1 (818) 354-8810
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mime
View raw message