oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Foster <holeno...@me.com>
Subject Fwd: Problem when running pushpull
Date Sun, 29 Jul 2012 23:59:11 GMT
hey YunHee,

I keep forgetting to add you to the email cc... are you on the dev list?... see email below, explains your commented out filemgr url making the pushpull work

-brian

Begin forwarded message:

> From: Brian Foster <holenoter@me.com>
> Date: July 29, 2012 10:12:48 AM PDT
> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Cc: Brian Foster <holenoter@mac.com>, "dev@oodt.apache.org" <dev@oodt.apache.org>, "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
> Subject: Re: Problem when running pushpull
> Reply-To: dev@oodt.apache.org
> 
> Hey YunHee,
> 
> You need to configure you mime-types.xml file in your policy directory to map your file names to filemgr ProductTypes... if you want to just download the files for now just comment out the filemgr 
> URL property in your pushpull properties file... if you want to configure the mime-types.xml file, take a look at: 
> 
> https://svn.apache.org/repos/asf/oodt/trunk/pushpull/src/main/resources/policy/mimetypes.xml
> 
> Here is basically how the file works:
> 
> <mime-type type="product/tes">
>  <_comment>ProductType=MyTesProductType</_comment>
>  <glob pattern="any valid Pattern.java regex" is regex="true"/>
> </mime-type>
> 
> the mime-type "type" must have a / and is just a name being used to title the mapping... fill in an actual filemgr ProductType in place of MyTesProdectType... and the pattern should be the regex describing the filenames of the files that belong to the give ProductType.
> 
> then make sure that org.apache.oodt.cas.pushpull.config.type.dectection.file property points at your mime-types.xml file.
> 
> -Brian 
> On Jul 29, 2012, at 6:45 AM, YunHee Kang <yunh.kang@gmail.com> wrote:
> 
>> Hi Brian,
>> 
>> I checked out the latest trunk of oodt 0.5 and installed it.
>> After I set up the file manger and pushpull framework,  I ran the
>> pushpull script.
>> As I described below, I think  that the trunk  is useful to fix the
>> problem, which was caused by handling ftp protocol.
>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>> username 'anonymous'
>> Jul 29, 2012 10:09:30 PM
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
>> initializeStagingArea
>> INFO: Preparing staging area /home/yhkang/oodt-0.5/cas-pushpull/staging/TESL2CO2
>> Jul 29, 2012 10:09:30 PM org.apache.commons.httpclient.HttpClient <clinit>
>> FINE: Java version: 1.6.0_31
>> 
>> But I got the difference problem that is related with  filemgr.
>> The following is part of error message in the file cas-pushpull0.log :
>> Jul 29, 2012 10:09:30 PM
>> org.apache.oodt.cas.filemgr.ingest.StdIngester hasProduct
>> WARNING: Unable to check for existance of product: [null]: Message:
>> Failure writing request
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Failure writing request
>>       at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606)
>>       at org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.java:284)
>>       at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlreadyInDatabase(FileRetrievalSystem.java:254)
>>       at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDownloadQueue(FileRetrievalSystem.java:463)
>>       at org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:138)
>> at org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>       at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>       at java.lang.Thread.run(Thread.java:662)
>> Jul 29, 2012 10:09:30 PM
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>> retrieveFiles
>> SEVERE: Failed to finish downloading per property files
>> /home/yhkang/oodt-0.5/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>> : [RemoteCrawler] Failed to communicate with database : Failure
>> writing request
>> org.apache.oodt.cas.pushpull.exceptions.RetrievalMethodException:
>> [RemoteCrawler] Failed to communicate with database : Failure writing
>> request
>>       at org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:150)
>>       at org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>       at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>       at java.lang.Thread.run(Thread.java:662)
>> Jul 29, 2012 10:09:30 PM
>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
>> INFO: Disconnecting protocol
>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
>> Jul 29, 2012 10:09:30 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>> INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to
>> sleep until Sun Jul 29 22:12:00 KST 2012
>> 
>> Please let me know how to fix this problem.
>> 
>> Thanks,
>> Yunhee
>> 
>> 
>> 
>> 2012/7/28 Brian Foster <holenoter@me.com>:
>>> Hey YunHee,
>>> 
>>> Doesn't look like the patch took this time, but don't worry about that now, i have put the fixes into the trunk... if you just check out the latest trunk and mvn install it (make sure you check out more than just cas-pushpull, because if protocol-ftp doesn't get reinstalled your cas-pushpull mvn install will pick up the last mvn installed version of the protocol-ftp 0.5 in your local maven repository... another thing to note, you probably should utilize the environment variables in place in the pushpull config files because if you ever decide to move your install you are going to have to change all the paths in your configuration files instead of just being able to change environment variables (if you have any question about that let me know)... again if you run into any road blocks just hit up the list and sorry about the bumpy road of getting pushpull up and running... it should be good to go now though (it downloads the files on my machine at least... lol).
>>> 
>>> -brian
>>> 
>>> On Jul 27, 2012, at 10:56 AM, YunHee Kang wrote:
>>> 
>>>> Hi Brain,
>>>> 
>>>> Unfortunately  your patch didn't fix the problem.   I got the same
>>>> error message before:
>>>> SEVERE: Failed to finish downloading per property files
>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>> : null
>>>> java.lang.NullPointerException
>>>>      at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368)
>>>>      at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312)
>>>>      at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301)
>>>>      at org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104)
>>>>      at org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>>>      at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>>>      at java.lang.Thread.run(Thread.java:662)
>>>> Jul 28, 2012 2:31:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>>> 
>>>> First of all,  I replaced  CommonsNetFtpProtocol.java with new one
>>>> that you modified as follows:
>>>> ftp.retrieveFile(fromFile.getName(), os);
>>>> /*
>>>> if (ftp.retrieveFile(fromFile.getName(), os))// {
>>>>       throw new ProtocolException("Failed to download file "
>>>>                             + fromFile.getName());
>>>>        // }
>>>> 
>>>>  */
>>>> Next  I rebuild a new jar file  cas-protocol-ftp-0.4.jar and then
>>>> copied it to  lib directory:
>>>> -rw-r--r-- 1 yhkang yhkang 17279 2012-07-28 02:09 cas-protocol-ftp-0.4.jar
>>>> ls -al  cas-protocol-ftp-0.4.jar
>>>> 
>>>> I am wondering that there is difference between  the following codes(1 and 2) :
>>>> 1)  ftp.retrieveFile(fromFile.getName(), os);
>>>> 2)    if (ftp.retrieveFile(fromFile.getName(), os))// {
>>>>       throw new ProtocolException("Failed to download file "
>>>>                             + fromFile.getName());
>>>>        // }
>>>> 
>>>> Will you let me know  if the method get()  in the
>>>> CommonsNetFtpProtocol.java was called when pushpull was run ?
>>>> 
>>>> Thanks,
>>>> Yunhee
>>>> 
>>>> 
>>>> 2012/7/27 Brian Foster <holenoter@mac.com>:
>>>>> Hey YunHee,
>>>>> 
>>>>> Okay the latest patch attached to the issue now (the on with todays date)
>>>>> works as far as pushpull is concerned, however FTPClient.retrieveFile being
>>>>> used in CommonsNetFtpProtocol is returning false for some reason... so i
>>>>> would recommend trying the following... first unpatch the first patch from
>>>>> your cas-pushpull and repatch with the new patch i've uploaded... then check
>>>>> out https://svn.apache.org/repos/asf/oodt/trunk/protocol/ftp/ and modify
>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol line 160 from this:
>>>>> 
>>>>> 
>>>>> if (ftp.retrieveFile(fromFile.getName(), os))// {
>>>>> 
>>>>> throw new ProtocolException("Failed to download file "
>>>>> 
>>>>> + fromFile.getName());
>>>>> 
>>>>> // }
>>>>> 
>>>>> 
>>>>> to this:
>>>>> 
>>>>> 
>>>>> ftp.retrieveFile(fromFile.getName(), os);
>>>>> 
>>>>> 
>>>>> then mvn compile jar:jar and replace ftp protocol jar with that one in your
>>>>> pushpull lib directory and give pushpull another run and see what happens
>>>>> (validate the files downloaded to make sure they are complete)... hope this
>>>>> helps and let me know if you need any further help!
>>>>> 
>>>>> 
>>>>> -brian
>>>>> 
>>>>> 
>>>>> On Jul 26, 2012, at 03:37 PM, YunHee Kang <yunh.kang@gmail.com> wrote:
>>>>> 
>>>>> Hi Brian,
>>>>> 
>>>>> I attach a tar file including etc and policy directories.
>>>>> 
>>>>> Thanks,
>>>>> Yunhee
>>>>> 
>>>>> 2012/7/27 Brian Foster <holenoter@mac.com>:
>>>>>> Hey YunHee,
>>>>>> 
>>>>>> send me a tarball with your etc and policy directories i will spend some
>>>>>> time today figuring out what is going on here
>>>>>> 
>>>>>> -brian
>>>>>> 
>>>>>> 
>>>>>> On Jul 26, 2012, at 01:26 AM, YunHee Kang <yunh.kang@gmail.com> wrote:
>>>>>> 
>>>>>> Hi Brian,
>>>>>> 
>>>>>> I just applied the patch you made according to the following steps:
>>>>>> cd apache-oodt-0.4-src/pushpull
>>>>>> patch -p0 < OODT-476.2012-07-25.txt
>>>>>> mvn clean
>>>>>> mvn install
>>>>>> cp target/cas-pushpull-0.4.jar ~/oodt/cas-pushpull/lib/
>>>>>> 
>>>>>> After I got a new cas-pushpull-0.4.jar,
>>>>>> yhkang@yunhee-kisti:~/oodt/cas-pushpull/lib$ ls -l cas-pushpull-0.4.jar
>>>>>> -rw-r--r-- 1 yhkang yhkang 422602 2012-07-26 16:58 cas-pushpull-0.4.jar
>>>>>> I ran the script pushpull.
>>>>>> 
>>>>>> 
>>>>>> But the same error message was generated:
>>>>>> 
>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>>>>> username 'anonymous'
>>>>>> Jul 26, 2012 5:12:56 PM
>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>> retrieveFiles
>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>> 
>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>> : null
>>>>>> java.lang.NullPointerException
>>>>>> at
>>>>>> 
>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368)
>>>>>> at
>>>>>> 
>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312)
>>>>>> at
>>>>>> 
>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301)
>>>>>> at
>>>>>> 
>>>>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104)
>>>>>> at
>>>>>> 
>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>>>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> 
>>>>>> Could you check a class name for handling ftp protocol ?
>>>>>> I used "CommonsNetFtpProtocolFactory.class" in the jar named
>>>>>> "cas-protocol-ftp-0.4.jar".
>>>>>> The class file was described in the file ProtocolFactoryInfo.xml as
>>>>>> follows:
>>>>>> <protocol type="ftp">
>>>>>> ....
>>>>>> <!-- cas-protocol-ftp-0.4.jar -->
>>>>>> <!--protocolFactory
>>>>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/-->
>>>>>> <protocolFactory
>>>>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/>
>>>>>> </protocol>
>>>>>> 
>>>>>> Thanks,
>>>>>> Yunhee
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 2012/7/26 Brian Foster <holenoter@mac.com>:
>>>>>>> hey Yunhee,
>>>>>>> 
>>>>>>> i just realized that your probably not on the dev list and my last email
>>>>>>> just went to the list and didn't have you cc'ed
>>>>>>> 
>>>>>>> try the patch attached to this issue mentioned below...
>>>>>>> 
>>>>>>> -brian
>>>>>>> 
>>>>>>> Begin forwarded message:
>>>>>>> 
>>>>>>> From: Brian Foster <holenoter@mac.com>
>>>>>>> Date: July 25, 2012 12:43:10 AM
>>>>>>> To: dev@oodt.apache.org,dev@oodt.apache.org
>>>>>>> Subject: Re: Problem when running pushpull
>>>>>>> 
>>>>>>> hey Yunhee,
>>>>>>> 
>>>>>>> Try patching your pushpull code with the fix i've attached to issue:
>>>>>>> 
>>>>>>> https://issues.apache.org/jira/browse/OODT-476
>>>>>>> 
>>>>>>> -brian
>>>>>>> 
>>>>>>> On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.kang@gmail.com> wrote:
>>>>>>> 
>>>>>>> Hi Brian,
>>>>>>> 
>>>>>>> I changed the file RetrievalSetup.java to get the stacktrace.
>>>>>>> So I got a new cas-pushpull-0.4.jar after rebuilt it.
>>>>>>> 
>>>>>>> The following is a message including the stacktrace with information
>>>>>>> about excpetion propation.
>>>>>>> 
>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>> 
>>>>>>> 
>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>>>>>>> : null
>>>>>>> java.lang.NullPointerException
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>>>>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>> 
>>>>>>> Please let me know what java programs I check frist.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Yunhee
>>>>>>> 2012/7/25 Brian Foster <holenoter@mac.com>:
>>>>>>>> YunHee,
>>>>>>>> 
>>>>>>>> There is a NullPointerException being thrown... on line 118 of
>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change
>>>>>>>> it
>>>>>>>> 
>>>>>>>> rebuild and deploy and that should give you the stacktrace which will
>>>>>>>> answer
>>>>>>>> what is going on here.
>>>>>>>> 
>>>>>>>> -brian
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)"
>>>>>>> 
>>>>>>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>>>> 
>>>>>>>> Hi YunHee,
>>>>>>>> 
>>>>>>>> Thanks for your mail.
>>>>>>>> 
>>>>>>>> Let me take a look at the configs again. Does anyone else using
>>>>>>>> PushPull have any advice for YunHee here? That error message
>>>>>>>> seems to indicate there is a config file error somewhere. I'm wondering
>>>>>>>> if your etc/push_pull_framework.properties has all the right values
>>>>>>>> in there. Did you go through Cameron's guide to getting Push Pull
>>>>>>>> running on the wiki?
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>> 
>>>>>>>> 
>>>>>>>> On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote:
>>>>>>>> 
>>>>>>>>> Hi Chris,
>>>>>>>>> 
>>>>>>>>> There are some problems in the configuration files.
>>>>>>>>> I found there is a dead link which does not connect two information
>>>>>>>>> in two files :RemoteSpecs.xml and ExternalSources.xml.
>>>>>>>>> For example an alias named "DewFtp" in RemoteSpecs.xml has no
>>>>>>>>> associated source in "ExternalSources.xml.
>>>>>>>>> 
>>>>>>>>> After I tailored those two XML files, I just tested one of
>>>>>>>>> configurations, whose alias is PolarNcepNoaaFtp.
>>>>>>>>> I also faced the same problem as follows:
>>>>>>>>> 
>>>>>>>>> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with
>>>>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>>>>>>>> username 'anonymous'
>>>>>>>>> Jul 25, 2012 2:21:04 AM
>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>>>> retrieveFiles
>>>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>>>>>>>>> : null
>>>>>>>>> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>>>>>>>>> sleep
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Yunhee
>>>>>>>>> 
>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>>>>>>>>>> Hi YunHee,
>>>>>>>>>> 
>>>>>>>>>> Thanks!
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> Chris
>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Chris,
>>>>>>>>>>> 
>>>>>>>>>>>> Thanks, can you try some of the other XML file configs.
>>>>>>>>>>> Yes, I will try it and let you know its result.
>>>>>>>>>>> 
>>>>>>>>>>>> Maybe it's being blocked in your country,
>>>>>>>>>>> That site may be blocked.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Yunhee
>>>>>>>>>>> 
>>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>>>>>>>>>>>> Hi YunHee,
>>>>>>>>>>>> 
>>>>>>> 
>>>>>>>>>>>> Thanks, can you try some of the other XML file configs. Maybe it's
>>>>>>>>>>>> being
>>>>>>>>>>>> blocked in your country, or something, but let's see if we can get
>>>>>>>>>>>> some
>>>>>>>>>>>> of the other XML RemoteSpec configs to work?
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Chris
>>>>>>>>>>>> 
>>>>>>>>>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Try out the MODIS one there, does that work?
>>>>>>>>>>>>> No, it doesn't.
>>>>>>>>>>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection
>>>>>>>>>>>>> problem.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> $ ftp e4ftl01u.ecs.nasa.gov
>>>>>>>>>>>>> ftp: connect: Connection timed out
>>>>>>>>>>>>> ftp>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The message below also say the reason why the problem was occurred.
>>>>>>>>>>>>> WARNING: Error occurred while connecting to RemoteSite: alias =
>>>>>>>>>>>>> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username =
>>>>>>>>>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to
>>>>>>>>>>>>> connect to server : Connection timed out
>>>>>>>>>>>>> Jul 25, 2012 1:31:25 AM
>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url =
>>>>>>>>>>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir =
>>>>>>>>>>>>> 'null' maxConnections = '-1' in 5 seconds
>>>>>>>>>>>>> Waiting . . . . .
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Yunhee
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>>>>>>>>>>>>>> Hi YunHee,
>>>>>>>>>>>>>> 
>>>>>>> 
>>>>>>>>>>>>>> OK, can you try some of the other files, e.g., the ones in:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> http://s.apache.org/ks0
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Try out the MODIS one there, does that work?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I added the header of XML as follows:
>>>>>>>>>>>>>>> <?xml version="1.0"?>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Next the value of elements was changed into specific one.
>>>>>>>>>>>>>>> The following is the output of the XML:
>>>>>>>>>>>>>>> <?xml version="1.0"?>
>>>>>>>>>>>>>>> <root>
>>>>>>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>>>>>>>>> <nofiles/>
>>>>>>>>>>>>>>> <!--dir name="\d{4}\.\d{2}\.\d{2}"-->
>>>>>>>>>>>>>>> <dir name="2004\.09\.20">
>>>>>>>>>>>>>>> <nodirs/>
>>>>>>>>>>>>>>> <!--file
>>>>>>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/-->
>>>>>>>>>>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/>
>>>>>>>>>>>>>>> </dir>
>>>>>>>>>>>>>>> </dirstruct>
>>>>>>>>>>>>>>> </root>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Unfortunately I got the same result before:
>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>> INFO: Testing protocol
>>>>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this
>>>>>>>>>>>>>>> ma
>>>>>>>>>>>>>>> y take a few minutes . . .
>>>>>>>>>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>>>>>>>>>>>>>> protocol 'org.apache.oodt.
>>>>>>>>>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous'
>>>>>>>>>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>>>>>>>>>> retrie
>>>>>>>>>>>>>>> veFiles
>>>>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/
>>>>>>>>>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null
>>>>>>>>>>>>>>> Jul 25, 2012 12:30:04 AM
>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon
>>>>>>>>>>>>>>> sleep
>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> sleep until Wed Jul 25 0
>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Yunhee
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J)
>>>>>>>>>>>>>>> <chris.a.mattmann@jpl.nasa.gov>:
>>>>>>>>>>>>>>>> Hi YunHee,
>>>>>>>>>>>>>>>> 
>>>>>>> 
>>>>>>>>>>>>>>>> Can you try simply putting a specific <dir name="..."> instead
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> \d{4}, etc., and
>>>>>>>>>>>>>>>> then can you put a specific file name in the <file name="..">.
>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>> that works and
>>>>>>>>>>>>>>>> downloads, then we know there is something up with the regex
>>>>>>>>>>>>>>>> somehow.
>>>>>>>>>>>>>>>> Also can you try adding the <?xml version="1.0"?> start to your
>>>>>>>>>>>>>>>> file?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The following is the content of TESL2CO2.xml.
>>>>>>>>>>>>>>>>> $ cat
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>>>>>>> <root>
>>>>>>>>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>>>>>>>>>>> <nofiles/>
>>>>>>>>>>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}">
>>>>>>>>>>>>>>>>> <nodirs/>
>>>>>>>>>>>>>>>>> <file
>>>>>>>>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/>
>>>>>>>>>>>>>>>>> </dir>
>>>>>>>>>>>>>>>>> </dirstruct>
>>>>>>>>>>>>>>>>> </root>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Let me know if there is a problem in the configuration file.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Yunhee
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J)
>>>>>>>>>>>>>>>>> <chris.a.mattmann@jpl.nasa.gov>:
>>>>>>>>>>>>>>>>>> Hi YunHee,
>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I am stuck on the problem related with "pushpull" script
>>>>>>>>>>>>>>>>>>> occurred
>>>>>>>>>>>>>>>>>>> the following error message:
>>>>>>>>>>>>>>>>>>> ....
>>>>>>>>>>>>>>>>>>> INFO: Testing protocol
>>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . .
>>>>>>>>>>>>>>>>>>> this may
>>>>>>>>>>>>>>>>>>> take a few minutes . . .
>>>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov
>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>>> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>>>>>>>>>>>>>>>>>> username 'anonymous'
>>>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>>>>>>>>>>>>>> retrieveFiles
>>>>>>>>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>>>>>>>>> : null
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Can you do me a favor and simply run this command:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> cat
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> What do you get?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>>>>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is
>>>>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012
>>>>>>>>>>>>>>>>>>> Jul 24, 2012 4:06:00 PM
>>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager
>>>>>>>>>>>>>>>>>>> setAsRunning
>>>>>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission to run
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> First of all, I would like test the pushpull framework to
>>>>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>> written in NetCDF from a remote site by a network protocols
>>>>>>>>>>>>>>>>>>> (like ftp,
>>>>>>>>>>>>>>>>>>> http, etc).
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Yep, sounds good.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I think that there is a test suite which can be used to check
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> pushpull framework.
>>>>>>>>>>>>>>>>>>> Can you give configuration files(a set of XML files) that I
>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> test
>>>>>>>>>>>>>>>>>>> downloading files from a FTP site (public anonymous site) ?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The examples in the resources directory that you found should
>>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>> so we'll just need to figure out why they aren't working on
>>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>> system.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> Please let me know a reference implementation of pushpull
>>>>>>>>>>>>>>>>>>> framework
>>>>>>>>>>>>>>>>>>> that I can use to move NetCDF files.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks we'll figure this out.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>>>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>> Phone: +1 (818) 354-8810
>>>>>>> 
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>> Senior Computer Scientist
>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> 
>>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message