oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From YunHee Kang <yunh.k...@gmail.com>
Subject Re: Problem when running pushpull
Date Wed, 01 Aug 2012 02:27:03 GMT
Hi Brian,


> Also, if you want to turn off filemgr communication you should only comment
> out the following property:
>
> org.apache.oodt.cas.filemgr.url

If I turned off the property that is related with filemgr mentioned
above, there was no problem to download a file from remote site. I
also checked the file was in the staging area:
$ ls
TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5
TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5.info.tmp

I found an extra file named
"TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5.info.tmp". I think the
extra file was generated by oodt.
Please let me know what the file is.

I also have a question about pushpull script.
The pushpull script has been running without termination even if the
file was completely downloaded.
Is it working well ?


> Let's go this route instead... don't worry about the mime-types.xml files...
> in your push_pull_framework.properties file just modify the following
> property to the following:
>
> org.apache.oodt.cas.pushpull.allow.only.defined.types=false
I modified the value of the property mentioned above to false.
It didn't give an answer to fix  my problem.

> does your filemgr have any products ingested?
Yes it does.
I  tried to test that my filemgr have a text file that was ingested.

 $./filemgr-client.sh
Aug 1, 2012 10:26:54 AM
org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from:
[../etc/filemgr.properties]
ingestProduct: Result: fb44d9a8-db77-11e1-a6d8-abc525326b84
I used the filemgr-client.sh as follows:
#!/bin/sh
./filemgr-client --url http://localhost:8000 --operation  --ingestProduct --prod
uctName blahProduct --productStructure Flat  --productTypeName GenericFile  --me
tadataFile file:///home/yhkang/oodt-0.5/cas-filemgr/bin/blah.txt.met
--refs file://
/home/yhkang/oodt/cas-filemgr/bin/blah.txt


But I am not sure that the the file was ingested well.  Because I
can't get a result by the following query:
./query_tool --url http://localhost:8000  --lucene -query
CAS.ProductType:GenericFile
As you can see, I used  8000 as the port number of my filemgr.
Let me know if there is a problem in the query.

I am sorry to give you too many questions.

Thanks,
Yunee
2012/7/31 Brian Foster <holenoter@me.com>:
> Hey YunHee,
>
> Let's go this route instead... don't worry about the mime-types.xml files...
> in your push_pull_framework.properties file just modify the following
> property to the following:
>
> org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>
> That turns off the mime-type stuff...
>
> Also, if you want to turn off filemgr communication you should only comment
> out the following property:
>
> org.apache.oodt.cas.filemgr.url
>
> However you should be fine with just turning off the mime-type stuff like
> described above... another thing, does your filemgr have any products
> ingested? i know there was a bug (not sure if it had been fixed yet) where
> the filemgr would throw an exception if you tried to query it when there
> were no products ingested (This is when using Lucene for the catalog
> store)... you may want to just make sure there is a dummy file ingested
>
> -brian
>
>
> On Jul 29, 2012, at 11:50 PM, YunHee Kang <yunh.kang@gmail.com> wrote:
>
> Hi Brain,
>
> I found two mime-types.xml files:
> - pushpull/policy/mime-types.xml
> - cas-filemgr/etc/mimet-ypes.xml
>
> In the directory pushpull/policy, I put the element "mine-type" like
> the following into mimetypes.xml:
> <mime-type type="product/tes">
> <_comment>ProductType=MyTesProductType</_comment>
> <glob
> pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
> isregex="true"/>
> </mime-type>
>
>> fill in an actual filemgr ProductType in place of
>> MyTesProdectType... and the pattern should be the regex describing the
>> filenames of the files that belong to the give ProductType.
>
> I also put the same "mime-type" into mime-types.xml in the directory
> cas-filemgr/etc.
> <mime-type type="product/tes">
> <_comment>ProductType=MyTesProductType</_comment>
> <glob
> pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
> isregex="true"/>
> </mime-type>
>
> Am I doing well ?
>
> Thanks,
> Yunhee
>
> 2012/7/30 Brian Foster <holenoter@me.com>:
>> hey YunHee,
>>
>> I keep forgetting to add you to the email cc... are you on the dev
>> list?...
>> see email below, explains your commented out filemgr url making the
>> pushpull
>> work
>>
>> -brian
>>
>> Begin forwarded message:
>>
>> From: Brian Foster <holenoter@me.com>
>> Date: July 29, 2012 10:12:48 AM PDT
>> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>> Cc: Brian Foster <holenoter@mac.com>, "dev@oodt.apache.org"
>> <dev@oodt.apache.org>, "Mattmann, Chris A (388J)"
>> <chris.a.mattmann@jpl.nasa.gov>
>> Subject: Re: Problem when running pushpull
>> Reply-To: dev@oodt.apache.org
>>
>> Hey YunHee,
>>
>> You need to configure you mime-types.xml file in your policy directory to
>> map your file names to filemgr ProductTypes... if you want to just
>> download
>> the files for now just comment out the filemgr
>> URL property in your pushpull properties file... if you want to configure
>> the mime-types.xml file, take a look at:
>>
>>
>> https://svn.apache.org/repos/asf/oodt/trunk/pushpull/src/main/resources/policy/mimetypes.xml
>>
>> Here is basically how the file works:
>>
>> <mime-type type="product/tes">
>> <_comment>ProductType=MyTesProductType</_comment>
>> <glob pattern="any valid Pattern.java regex" is regex="true"/>
>> </mime-type>
>>
>> the mime-type "type" must have a / and is just a name being used to title
>> the mapping... fill in an actual filemgr ProductType in place of
>> MyTesProdectType... and the pattern should be the regex describing the
>> filenames of the files that belong to the give ProductType.
>>
>> then make sure that
>> org.apache.oodt.cas.pushpull.config.type.dectection.file
>> property points at your mime-types.xml file.
>>
>> -Brian
>> On Jul 29, 2012, at 6:45 AM, YunHee Kang <yunh.kang@gmail.com> wrote:
>>
>> Hi Brian,
>>
>>
>> I checked out the latest trunk of oodt 0.5 and installed it.
>>
>> After I set up the file manger and pushpull framework, I ran the
>>
>> pushpull script.
>>
>> As I described below, I think that the trunk is useful to fix the
>>
>> problem, which was caused by handling ftp protocol.
>>
>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>
>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>
>> username 'anonymous'
>>
>> Jul 29, 2012 10:09:30 PM
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
>>
>> initializeStagingArea
>>
>> INFO: Preparing staging area
>> /home/yhkang/oodt-0.5/cas-pushpull/staging/TESL2CO2
>>
>> Jul 29, 2012 10:09:30 PM org.apache.commons.httpclient.HttpClient <clinit>
>>
>> FINE: Java version: 1.6.0_31
>>
>>
>> But I got the difference problem that is related with filemgr.
>>
>> The following is part of error message in the file cas-pushpull0.log :
>>
>> Jul 29, 2012 10:09:30 PM
>>
>> org.apache.oodt.cas.filemgr.ingest.StdIngester hasProduct
>>
>> WARNING: Unable to check for existance of product: [null]: Message:
>>
>> Failure writing request
>>
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>
>> Failure writing request
>>
>> at
>>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606)
>>
>> at
>>
>> org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.java:284)
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlreadyInDatabase(FileRetrievalSystem.java:254)
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDownloadQueue(FileRetrievalSystem.java:463)
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:138)
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>
>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Jul 29, 2012 10:09:30 PM
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>
>> retrieveFiles
>>
>> SEVERE: Failed to finish downloading per property files
>>
>>
>> /home/yhkang/oodt-0.5/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>
>> : [RemoteCrawler] Failed to communicate with database : Failure
>>
>> writing request
>>
>> org.apache.oodt.cas.pushpull.exceptions.RetrievalMethodException:
>>
>> [RemoteCrawler] Failed to communicate with database : Failure writing
>>
>> request
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:150)
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>
>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Jul 29, 2012 10:09:30 PM
>>
>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
>>
>> INFO: Disconnecting protocol
>>
>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
>>
>> Jul 29, 2012 10:09:30 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>
>> INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to
>>
>> sleep until Sun Jul 29 22:12:00 KST 2012
>>
>>
>> Please let me know how to fix this problem.
>>
>>
>> Thanks,
>>
>> Yunhee
>>
>>
>>
>>
>> 2012/7/28 Brian Foster <holenoter@me.com>:
>>
>> Hey YunHee,
>>
>>
>> Doesn't look like the patch took this time, but don't worry about that
>> now,
>> i have put the fixes into the trunk... if you just check out the latest
>> trunk and mvn install it (make sure you check out more than just
>> cas-pushpull, because if protocol-ftp doesn't get reinstalled your
>> cas-pushpull mvn install will pick up the last mvn installed version of
>> the
>> protocol-ftp 0.5 in your local maven repository... another thing to note,
>> you probably should utilize the environment variables in place in the
>> pushpull config files because if you ever decide to move your install you
>> are going to have to change all the paths in your configuration files
>> instead of just being able to change environment variables (if you have
>> any
>> question about that let me know)... again if you run into any road blocks
>> just hit up the list and sorry about the bumpy road of getting pushpull up
>> and running... it should be good to go now though (it downloads the files
>> on
>> my machine at least... lol).
>>
>>
>> -brian
>>
>>
>> On Jul 27, 2012, at 10:56 AM, YunHee Kang wrote:
>>
>>
>> Hi Brain,
>>
>>
>> Unfortunately your patch didn't fix the problem. I got the same
>>
>> error message before:
>>
>> SEVERE: Failed to finish downloading per property files
>>
>>
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>
>> : null
>>
>> java.lang.NullPointerException
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368)
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312)
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301)
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104)
>>
>> at
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>
>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Jul 28, 2012 2:31:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>
>>
>> First of all, I replaced CommonsNetFtpProtocol.java with new one
>>
>> that you modified as follows:
>>
>> ftp.retrieveFile(fromFile.getName(), os);
>>
>> /*
>>
>> if (ftp.retrieveFile(fromFile.getName(), os))// {
>>
>> throw new ProtocolException("Failed to download file "
>>
>> + fromFile.getName());
>>
>> // }
>>
>>
>> */
>>
>> Next I rebuild a new jar file cas-protocol-ftp-0.4.jar and then
>>
>> copied it to lib directory:
>>
>> -rw-r--r-- 1 yhkang yhkang 17279 2012-07-28 02:09 cas-protocol-ftp-0.4.jar
>>
>> ls -al cas-protocol-ftp-0.4.jar
>>
>>
>> I am wondering that there is difference between the following codes(1 and
>> 2) :
>>
>> 1) ftp.retrieveFile(fromFile.getName(), os);
>>
>> 2) if (ftp.retrieveFile(fromFile.getName(), os))// {
>>
>> throw new ProtocolException("Failed to download file "
>>
>> + fromFile.getName());
>>
>> // }
>>
>>
>> Will you let me know if the method get() in the
>>
>> CommonsNetFtpProtocol.java was called when pushpull was run ?
>>
>>
>> Thanks,
>>
>> Yunhee
>>
>>
>>
>> 2012/7/27 Brian Foster <holenoter@mac.com>:
>>
>> Hey YunHee,
>>
>>
>> Okay the latest patch attached to the issue now (the on with todays date)
>>
>> works as far as pushpull is concerned, however FTPClient.retrieveFile
>> being
>>
>> used in CommonsNetFtpProtocol is returning false for some reason... so i
>>
>> would recommend trying the following... first unpatch the first patch from
>>
>> your cas-pushpull and repatch with the new patch i've uploaded... then
>> check
>>
>> out https://svn.apache.org/repos/asf/oodt/trunk/protocol/ftp/ and modify
>>
>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol line 160 from this:
>>
>>
>>
>> if (ftp.retrieveFile(fromFile.getName(), os))// {
>>
>>
>> throw new ProtocolException("Failed to download file "
>>
>>
>> + fromFile.getName());
>>
>>
>> // }
>>
>>
>>
>> to this:
>>
>>
>>
>> ftp.retrieveFile(fromFile.getName(), os);
>>
>>
>>
>> then mvn compile jar:jar and replace ftp protocol jar with that one in
>> your
>>
>> pushpull lib directory and give pushpull another run and see what happens
>>
>> (validate the files downloaded to make sure they are complete)... hope
>> this
>>
>> helps and let me know if you need any further help!
>>
>>
>>
>> -brian
>>
>>
>>
>> On Jul 26, 2012, at 03:37 PM, YunHee Kang <yunh.kang@gmail.com> wrote:
>>
>>
>> Hi Brian,
>>
>>
>> I attach a tar file including etc and policy directories.
>>
>>
>> Thanks,
>>
>> Yunhee
>>
>>
>> 2012/7/27 Brian Foster <holenoter@mac.com>:
>>
>> Hey YunHee,
>>
>>
>> send me a tarball with your etc and policy directories i will spend some
>>
>> time today figuring out what is going on here
>>
>>
>> -brian
>>
>>
>>
>> On Jul 26, 2012, at 01:26 AM, YunHee Kang <yunh.kang@gmail.com> wrote:
>>
>>
>> Hi Brian,
>>
>>
>> I just applied the patch you made according to the following steps:
>>
>> cd apache-oodt-0.4-src/pushpull
>>
>> patch -p0 < OODT-476.2012-07-25.txt
>>
>> mvn clean
>>
>> mvn install
>>
>> cp target/cas-pushpull-0.4.jar ~/oodt/cas-pushpull/lib/
>>
>>
>> After I got a new cas-pushpull-0.4.jar,
>>
>> yhkang@yunhee-kisti:~/oodt/cas-pushpull/lib$ ls -l cas-pushpull-0.4.jar
>>
>> -rw-r--r-- 1 yhkang yhkang 422602 2012-07-26 16:58 cas-pushpull-0.4.jar
>>
>> I ran the script pushpull.
>>
>>
>>
>> But the same error message was generated:
>>
>>
>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>
>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>
>> username 'anonymous'
>>
>> Jul 26, 2012 5:12:56 PM
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>
>> retrieveFiles
>>
>> SEVERE: Failed to finish downloading per property files
>>
>>
>>
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>
>> : null
>>
>> java.lang.NullPointerException
>>
>> at
>>
>>
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368)
>>
>> at
>>
>>
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312)
>>
>> at
>>
>>
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301)
>>
>> at
>>
>>
>>
>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104)
>>
>> at
>>
>>
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>
>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>
>> at java.lang.Thread.run(Thread.java:662)
>>
>>
>> Could you check a class name for handling ftp protocol ?
>>
>> I used "CommonsNetFtpProtocolFactory.class" in the jar named
>>
>> "cas-protocol-ftp-0.4.jar".
>>
>> The class file was described in the file ProtocolFactoryInfo.xml as
>>
>> follows:
>>
>> <protocol type="ftp">
>>
>> ....
>>
>> <!-- cas-protocol-ftp-0.4.jar -->
>>
>> <!--protocolFactory
>>
>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/-->
>>
>> <protocolFactory
>>
>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/>
>>
>> </protocol>
>>
>>
>> Thanks,
>>
>> Yunhee
>>
>>
>>
>>
>>
>> 2012/7/26 Brian Foster <holenoter@mac.com>:
>>
>> hey Yunhee,
>>
>>
>> i just realized that your probably not on the dev list and my last email
>>
>> just went to the list and didn't have you cc'ed
>>
>>
>> try the patch attached to this issue mentioned below...
>>
>>
>> -brian
>>
>>
>> Begin forwarded message:
>>
>>
>> From: Brian Foster <holenoter@mac.com>
>>
>> Date: July 25, 2012 12:43:10 AM
>>
>> To: dev@oodt.apache.org,dev@oodt.apache.org
>>
>> Subject: Re: Problem when running pushpull
>>
>>
>> hey Yunhee,
>>
>>
>> Try patching your pushpull code with the fix i've attached to issue:
>>
>>
>> https://issues.apache.org/jira/browse/OODT-476
>>
>>
>> -brian
>>
>>
>> On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.kang@gmail.com> wrote:
>>
>>
>> Hi Brian,
>>
>>
>> I changed the file RetrievalSetup.java to get the stacktrace.
>>
>> So I got a new cas-pushpull-0.4.jar after rebuilt it.
>>
>>
>> The following is a message including the stacktrace with information
>>
>> about excpetion propation.
>>
>>
>> SEVERE: Failed to finish downloading per property files
>>
>>
>>
>>
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>>
>> : null
>>
>> java.lang.NullPointerException
>>
>> at
>>
>>
>>
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367)
>>
>> at
>>
>>
>>
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311)
>>
>> at
>>
>>
>>
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300)
>>
>> at
>>
>>
>>
>>
>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103)
>>
>> at
>>
>>
>>
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>
>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>
>> at java.lang.Thread.run(Thread.java:662)
>>
>>
>> Please let me know what java programs I check frist.
>>
>>
>> Thanks,
>>
>>
>> Yunhee
>>
>> 2012/7/25 Brian Foster <holenoter@mac.com>:
>>
>> YunHee,
>>
>>
>> There is a NullPointerException being thrown... on line 118 of
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change
>>
>> it
>>
>>
>> rebuild and deploy and that should give you the stacktrace which will
>>
>> answer
>>
>> what is going on here.
>>
>>
>> -brian
>>
>>
>>
>> On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)"
>>
>>
>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>
>>
>> Hi YunHee,
>>
>>
>> Thanks for your mail.
>>
>>
>> Let me take a look at the configs again. Does anyone else using
>>
>> PushPull have any advice for YunHee here? That error message
>>
>> seems to indicate there is a config file error somewhere. I'm wondering
>>
>> if your etc/push_pull_framework.properties has all the right values
>>
>> in there. Did you go through Cameron's guide to getting Push Pull
>>
>> running on the wiki?
>>
>>
>> Cheers,
>>
>> Chris
>>
>>
>>
>> On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote:
>>
>>
>> Hi Chris,
>>
>>
>> There are some problems in the configuration files.
>>
>> I found there is a dead link which does not connect two information
>>
>> in two files :RemoteSpecs.xml and ExternalSources.xml.
>>
>> For example an alias named "DewFtp" in RemoteSpecs.xml has no
>>
>> associated source in "ExternalSources.xml.
>>
>>
>> After I tailored those two XML files, I just tested one of
>>
>> configurations, whose alias is PolarNcepNoaaFtp.
>>
>> I also faced the same problem as follows:
>>
>>
>> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with
>>
>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>
>> username 'anonymous'
>>
>> Jul 25, 2012 2:21:04 AM
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>
>> retrieveFiles
>>
>> SEVERE: Failed to finish downloading per property files
>>
>>
>>
>>
>>
>>
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>>
>> : null
>>
>> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>>
>> sleep
>>
>>
>>
>> Thanks,
>>
>> Yunhee
>>
>>
>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>>
>> Hi YunHee,
>>
>>
>> Thanks!
>>
>>
>> Cheers,
>>
>> Chris
>>
>>
>>
>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote:
>>
>>
>> Hi Chris,
>>
>>
>> Thanks, can you try some of the other XML file configs.
>>
>> Yes, I will try it and let you know its result.
>>
>>
>> Maybe it's being blocked in your country,
>>
>> That site may be blocked.
>>
>>
>>
>> Thanks,
>>
>> Yunhee
>>
>>
>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>>
>> Hi YunHee,
>>
>>
>>
>> Thanks, can you try some of the other XML file configs. Maybe it's
>>
>> being
>>
>> blocked in your country, or something, but let's see if we can get
>>
>> some
>>
>> of the other XML RemoteSpec configs to work?
>>
>>
>> Cheers,
>>
>> Chris
>>
>>
>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote:
>>
>>
>> Hi Chris,
>>
>>
>>
>> Try out the MODIS one there, does that work?
>>
>> No, it doesn't.
>>
>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection
>>
>> problem.
>>
>>
>> $ ftp e4ftl01u.ecs.nasa.gov
>>
>> ftp: connect: Connection timed out
>>
>> ftp>
>>
>>
>> The message below also say the reason why the problem was occurred.
>>
>> WARNING: Error occurred while connecting to RemoteSite: alias =
>>
>> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username =
>>
>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to
>>
>> connect to server : Connection timed out
>>
>> Jul 25, 2012 1:31:25 AM
>>
>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>
>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url =
>>
>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir =
>>
>> 'null' maxConnections = '-1' in 5 seconds
>>
>> Waiting . . . . .
>>
>>
>> Thanks,
>>
>> Yunhee
>>
>>
>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov>:
>>
>> Hi YunHee,
>>
>>
>>
>> OK, can you try some of the other files, e.g., the ones in:
>>
>>
>> http://s.apache.org/ks0
>>
>>
>> Try out the MODIS one there, does that work?
>>
>>
>> Cheers,
>>
>> Chris
>>
>>
>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote:
>>
>>
>> Hi Chris,
>>
>>
>> I added the header of XML as follows:
>>
>> <?xml version="1.0"?>
>>
>>
>> Next the value of elements was changed into specific one.
>>
>> The following is the output of the XML:
>>
>> <?xml version="1.0"?>
>>
>> <root>
>>
>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>
>> <nofiles/>
>>
>> <!--dir name="\d{4}\.\d{2}\.\d{2}"-->
>>
>> <dir name="2004\.09\.20">
>>
>> <nodirs/>
>>
>> <!--file
>>
>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/-->
>>
>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/>
>>
>> </dir>
>>
>> </dirstruct>
>>
>> </root>
>>
>>
>> Unfortunately I got the same result before:
>>
>> ...
>>
>> INFO: Testing protocol
>>
>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this
>>
>> ma
>>
>> y take a few minutes . . .
>>
>> Jul 25, 2012 12:30:03 AM
>>
>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>
>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>
>> protocol 'org.apache.oodt.
>>
>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous'
>>
>> Jul 25, 2012 12:30:03 AM
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>
>> retrie
>>
>> veFiles
>>
>> SEVERE: Failed to finish downloading per property files
>>
>> /home/yhkang/oodt/cas-pushpull/etc/
>>
>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null
>>
>> Jul 25, 2012 12:30:04 AM
>>
>> org.apache.oodt.cas.pushpull.daemon.Daemon
>>
>> sleep
>>
>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going
>>
>> to
>>
>> sleep until Wed Jul 25 0
>>
>> ...
>>
>>
>>
>> Thanks,
>>
>> Yunhee
>>
>>
>>
>> 2012/7/25 Mattmann, Chris A (388J)
>>
>> <chris.a.mattmann@jpl.nasa.gov>:
>>
>> Hi YunHee,
>>
>>
>>
>> Can you try simply putting a specific <dir name="..."> instead
>>
>> of
>>
>> \d{4}, etc., and
>>
>> then can you put a specific file name in the <file name="..">.
>>
>> If
>>
>> that works and
>>
>> downloads, then we know there is something up with the regex
>>
>> somehow.
>>
>> Also can you try adding the <?xml version="1.0"?> start to your
>>
>> file?
>>
>>
>> Cheers,
>>
>> Chris
>>
>>
>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote:
>>
>>
>> Hi Chris,
>>
>>
>>
>> The following is the content of TESL2CO2.xml.
>>
>> $ cat
>>
>>
>>
>>
>>
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>
>> <root>
>>
>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>
>> <nofiles/>
>>
>> <dir name="\d{4}\.\d{2}\.\d{2}">
>>
>> <nodirs/>
>>
>> <file
>>
>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/>
>>
>> </dir>
>>
>> </dirstruct>
>>
>> </root>
>>
>>
>> Let me know if there is a problem in the configuration file.
>>
>>
>> Thanks,
>>
>> Yunhee
>>
>>
>> 2012/7/24 Mattmann, Chris A (388J)
>>
>> <chris.a.mattmann@jpl.nasa.gov>:
>>
>> Hi YunHee,
>>
>>
>>
>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote:
>>
>>
>> Hi Chris,
>>
>>
>> I am stuck on the problem related with "pushpull" script
>>
>> occurred
>>
>> the following error message:
>>
>> ....
>>
>> INFO: Testing protocol
>>
>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . .
>>
>> this may
>>
>> take a few minutes . . .
>>
>> Jul 24, 2012 4:05:02 PM
>>
>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>
>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov
>>
>> with
>>
>> protocol
>>
>> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>
>> username 'anonymous'
>>
>> Jul 24, 2012 4:05:02 PM
>>
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>
>> retrieveFiles
>>
>> SEVERE: Failed to finish downloading per property files
>>
>>
>>
>>
>>
>>
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>
>> : null
>>
>>
>> Can you do me a favor and simply run this command:
>>
>>
>> cat
>>
>>
>>
>>
>>
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>
>>
>> What do you get?
>>
>>
>> Jul 24, 2012 4:05:02 PM
>>
>> org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>
>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is
>>
>> going
>>
>> to
>>
>> sleep until Tue Jul 24 16:06:00 KST 2012
>>
>> Jul 24, 2012 4:06:00 PM
>>
>> org.apache.oodt.cas.pushpull.daemon.DaemonManager
>>
>> setAsRunning
>>
>> INFO: Daemon with ID = 91131 was given permission to run
>>
>>
>>
>>
>>
>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>>
>>
>> First of all, I would like test the pushpull framework to
>>
>> move
>>
>> data
>>
>> written in NetCDF from a remote site by a network protocols
>>
>> (like ftp,
>>
>> http, etc).
>>
>>
>> Yep, sounds good.
>>
>>
>> I think that there is a test suite which can be used to check
>>
>> the
>>
>> pushpull framework.
>>
>> Can you give configuration files(a set of XML files) that I
>>
>> can
>>
>> test
>>
>> downloading files from a FTP site (public anonymous site) ?
>>
>>
>> The examples in the resources directory that you found should
>>
>> work
>>
>> so we'll just need to figure out why they aren't working on
>>
>> your
>>
>> system.
>>
>>
>> or
>>
>> Please let me know a reference implementation of pushpull
>>
>> framework
>>
>> that I can use to move NetCDF files.
>>
>>
>> Thanks we'll figure this out.
>>
>>
>> Cheers,
>>
>> Chris
>>
>>
>>
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Chris Mattmann, Ph.D.
>>
>> Senior Computer Scientist
>>
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>
>> Office: 171-266B, Mailstop: 171-246
>>
>> Email: chris.a.mattmann@nasa.gov
>>
>> WWW: http://sunset.usc.edu/~mattmann/
>>
>>
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Adjunct Assistant Professor, Computer Science Department
>>
>> University of Southern California, Los Angeles, CA 90089 USA
>>
>>
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Chris Mattmann, Ph.D.
>>
>> Senior Computer Scientist
>>
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>
>> Office: 171-266B, Mailstop: 171-246
>>
>> Email: chris.a.mattmann@nasa.gov
>>
>> WWW: http://sunset.usc.edu/~mattmann/
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Adjunct Assistant Professor, Computer Science Department
>>
>> University of Southern California, Los Angeles, CA 90089 USA
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Chris Mattmann, Ph.D.
>>
>> Senior Computer Scientist
>>
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>
>> Office: 171-266B, Mailstop: 171-246
>>
>> Email: chris.a.mattmann@nasa.gov
>>
>> WWW: http://sunset.usc.edu/~mattmann/
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Adjunct Assistant Professor, Computer Science Department
>>
>> University of Southern California, Los Angeles, CA 90089 USA
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Chris Mattmann, Ph.D.
>>
>> Senior Computer Scientist
>>
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>
>> Office: 171-266B, Mailstop: 171-246
>>
>> Email: chris.a.mattmann@nasa.gov
>>
>> WWW: http://sunset.usc.edu/~mattmann/
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Adjunct Assistant Professor, Computer Science Department
>>
>> University of Southern California, Los Angeles, CA 90089 USA
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Chris Mattmann, Ph.D.
>>
>> Senior Computer Scientist
>>
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>
>> Office: 171-266B, Mailstop: 171-246
>>
>> Email: chris.a.mattmann@nasa.gov
>>
>> WWW: http://sunset.usc.edu/~mattmann/
>>
>> Phone: +1 (818) 354-8810
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Adjunct Assistant Professor, Computer Science Department
>>
>> University of Southern California, Los Angeles, CA 90089 USA
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Chris Mattmann, Ph.D.
>>
>> Senior Computer Scientist
>>
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>
>> Office: 171-266B, Mailstop: 171-246
>>
>> Email: chris.a.mattmann@nasa.gov
>>
>> WWW: http://sunset.usc.edu/~mattmann/
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Adjunct Assistant Professor, Computer Science Department
>>
>> University of Southern California, Los Angeles, CA 90089 USA
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>

Mime
View raw message