oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: pushPull component problem
Date Thu, 28 Mar 2013 14:15:07 GMT
Hey Chintu,

This totally helped, thanks.

Can you take a look at the plugins here?

https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html


Try installing those, and let me know if either of those FTP
plugins works better and actually downloads the filesŠ

In the meanwhile I'll look at the logs and see what I can come up
with.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++




-----Original Message-----
From: <Mistry>, "Chintu [SGT INC]  (GSFC-586.0)" <chintu.mistry@nasa.gov>
Date: Thursday, March 28, 2013 7:12 AM
To: jpluser <chris.a.mattmann@jpl.nasa.gov>, "dev@oodt.apache.org"
<dev@oodt.apache.org>
Cc: "Ramirez, Paul M (388J)" <paul.m.ramirez@jpl.nasa.gov>
Subject: Re: pushPull component problem

>I have attached verbose log. I have added extra print statements and
>compiled to see what is returned from remote site. Hope this helps.
>
>
>-- 
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
>
>
>
>
>
>From: <Mattmann>, Chris A <chris.a.mattmann@jpl.nasa.gov>
>Date: Thursday, March 28, 2013 10:05 AM
>To: Chintu Mistry <chintu.mistry@nasa.gov>, "dev@oodt.apache.org"
><dev@oodt.apache.org>
>Cc: "Ramirez, Paul M" <paul.m.ramirez@jpl.nasa.gov>
>Subject: Re: pushPull component problem
>
>
>
>Hey Chintu,
>
>
>Thanks for sending this ‹ really helped verify it's something else.
>
>
>Can you capture the log of push pull, with verbose logging turned on, and
>maybe do a pastebin?
>
>
>Cheers,
>Chris
>
>
>
>
>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <chintu.mistry@nasa.gov>
>Date: Thursday, March 28, 2013 7:03 AM
>To: jpluser <chris.a.mattmann@jpl.nasa.gov>, "dev@oodt.apache.org"
><dev@oodt.apache.org>
>Cc: "Ramirez, Paul M (388J)" <paul.m.ramirez@jpl.nasa.gov>
>Subject: Re: pushPull component problem
>
>
>
>>
>>
>>
>>
>>Something else is wrong. I know that the regex is fine. I don't even
>>have regex, it is straight up file name. It should match.
>>
>>
>>Thanks for looking into this.
>>-- 
>>Chintu Mistry
>>NASA Goddard Space Flight Center
>>Bldg L40B, Room S776
>>Office: 240 684 0477
>>Mobile: 770 310 1047
>>
>>
>>
>>
>>
>>From: <Mattmann>, Chris A <chris.a.mattmann@jpl.nasa.gov>
>>Date: Thursday, March 28, 2013 9:01 AM
>>To: Chintu Mistry <chintu.mistry@nasa.gov>, "dev@oodt.apache.org"
>><dev@oodt.apache.org>
>>Cc: "Ramirez, Paul M" <paul.m.ramirez@jpl.nasa.gov>
>>Subject: Re: pushPull component problem
>>
>>
>>
>>Hey Chintu,
>>
>>
>>Sorry that you have been having trouble and thanks for pasting the
>>relevant portions
>>of push pull config to diagnose this.
>>
>>
>>I think you have an issue in your RemoteFileSpecs file
>>(TEST_AVHRR_ONLY.xml) --
>>one quick way to debug this is to use the tool that Brian Foster wrote:
>>
>>
>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
>>$ cd bin
>>$ java ­Djava.ext.dirs=../lib
>>org.apache.oodt.cas.pushpull.util.ExpressionValidator
>>
>>
>>It's a simple GUI that will allow you to put in regular expressions from
>>your push pull
>>RmoteFileSpecs files, and then put in file names from the server to see
>>if they match.
>>
>>
>>Can you run the above and let me know if it's matching? If not, we've
>>found your issue.
>>If it is matching, we'll move on to the next set of things to check.
>>
>>
>>Thanks!
>>
>>
>>Cheers,
>>Chris
>>
>>
>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <chintu.mistry@nasa.gov>
>>Date: Thursday, March 28, 2013 5:47 AM
>>To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>>Cc: jpluser <chris.a.mattmann@jpl.nasa.gov>, "Ramirez, Paul M (388J)"
>><paul.m.ramirez@jpl.nasa.gov>
>>Subject: pushPull component problem
>>
>>
>>
>>>Hi,
>>>
>>>
>>>I have been playing around with push pull for last two days and trying
>>>to download some data from remote location without success.
>>>I would appreciate if you guys can shed some light on what is going on
>>>and why isn't push pull find files to download even though its clearly
>>>there at remote location.
>>>
>>>
>>>Here is my configuration. I am trying to download
>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc
>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and
>>>it lists all the files as well. However for some reason it
>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am
>>>doing wrong. I have tried almost all combinations and still could not
>>>get it to work.
>>>
>>>
>>>In my /etc/pushpull.properties file, I have types=false so that it
>>>downloads the file even if its not specified in mime-types.xml file.
>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>>>
>>>
>>>ExternalSources.xml
>>><sources>
>>>    <source host="eclipse.ncdc.noaa.gov">
>>>        <login type="ftp" alias="eclipseClass">
>>>            <username>anonymous</username>
>>>            <password>user@host.com</password>
>>>        </login>
>>>    </source>
>>></sources>
>>>
>>>
>>>
>>>RemoteSpecs.xml
>>>        <daemon alias="eclipseClass" active="yes">
>>>            <runInfo firstRunDateTime="2013-03-25T00:00:00Z"
>>>period="1m" runOnReboot="yes"/>
>>>            <propInfo
>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
>>>                <propFiles regExp="TEST_AVHRR_ONLY\.xml"
>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructX
>>>mlParser"/>
>>>            </propInfo>
>>>            <dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no"
>>>queryElement="Filename"/>
>>>        </daemon>
>>>
>>>
>>>
>>>TEST_AVHRR_ONLY.xml
>>><root>
>>>    <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
>>>        <nodirs/>
>>>        <!-- <file
>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
>>>        <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
>>>    </dirstruct>
>>></root>
>>>
>>>
>>>
>>>
>>>
>>>Regards
>>>-- 
>>>Chintu Mistry
>>>NASA Goddard Space Flight Center
>>>Bldg L40B, Room S776
>>>Office: 240 684 0477
>>>Mobile: 770 310 1047
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>


Mime
View raw message