oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Verma, Rishi (388J)" <Rishi.Ve...@jpl.nasa.gov>
Subject Re: pushPull component problem
Date Thu, 28 Mar 2013 23:57:54 GMT
Hey Chintu,

I took a look at your output file, and it seems you must still be using a RegEx somewhere?

PushPull seems to be scanning through a bunch of files, not only one:
---
FileList size: 86
Filename : [avhrr-only-v2.20130101.nc.gz]
Filename : [avhrr-only-v2.20130102.nc.gz]
Filename : [avhrr-only-v2.20130103.nc.gz]
Filename : [avhrr-only-v2.20130104.nc.gz]
Filename : [avhrr-only-v2.20130105.nc.gz]
---

Can you try the following? Specify an exact filename within TEST_AVHRR_ONLY.xml (you may have
already done this), and specify the exact same filename within mime-types.xml. Make sure types=true
wtihin pushpull.properties and that the alias for your product type matches the type name
within mime-types.xml. You should see PushPull trying to scan for only one file.

Another thing you can try is to recreating the tutorial example from [1] and slowly adjusting
it to your desired FTP server and product details. I recently set up PushPull using this method,
and got it working.

Rishi

--
[1] https://cwiki.apache.org/OODT/oodt-push-pull-user-guide.html

On Mar 28, 2013, at 7:12 AM, Mistry, Chintu (GSFC-586.0)[SGT INC] wrote:

I have attached verbose log. I have added extra print statements and compiled to see what
is returned from remote site. Hope this helps.

--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047

From: <Mattmann>, Chris A <chris.a.mattmann@jpl.nasa.gov<mailto:chris.a.mattmann@jpl.nasa.gov>>
Date: Thursday, March 28, 2013 10:05 AM
To: Chintu Mistry <chintu.mistry@nasa.gov<mailto:chintu.mistry@nasa.gov>>, "dev@oodt.apache.org<mailto:dev@oodt.apache.org>"
<dev@oodt.apache.org<mailto:dev@oodt.apache.org>>
Cc: "Ramirez, Paul M" <paul.m.ramirez@jpl.nasa.gov<mailto:paul.m.ramirez@jpl.nasa.gov>>
Subject: Re: pushPull component problem

Hey Chintu,

Thanks for sending this — really helped verify it's something else.

Can you capture the log of push pull, with verbose logging turned on, and maybe do a pastebin?

Cheers,
Chris


From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <chintu.mistry@nasa.gov<mailto:chintu.mistry@nasa.gov>>
Date: Thursday, March 28, 2013 7:03 AM
To: jpluser <chris.a.mattmann@jpl.nasa.gov<mailto:chris.a.mattmann@jpl.nasa.gov>>,
"dev@oodt.apache.org<mailto:dev@oodt.apache.org>" <dev@oodt.apache.org<mailto:dev@oodt.apache.org>>
Cc: "Ramirez, Paul M (388J)" <paul.m.ramirez@jpl.nasa.gov<mailto:paul.m.ramirez@jpl.nasa.gov>>
Subject: Re: pushPull component problem


Something else is wrong. I know that the regex is fine. I don't even have regex, it is straight
up file name. It should match.

Thanks for looking into this.
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047

From: <Mattmann>, Chris A <chris.a.mattmann@jpl.nasa.gov<mailto:chris.a.mattmann@jpl.nasa.gov>>
Date: Thursday, March 28, 2013 9:01 AM
To: Chintu Mistry <chintu.mistry@nasa.gov<mailto:chintu.mistry@nasa.gov>>, "dev@oodt.apache.org<mailto:dev@oodt.apache.org>"
<dev@oodt.apache.org<mailto:dev@oodt.apache.org>>
Cc: "Ramirez, Paul M" <paul.m.ramirez@jpl.nasa.gov<mailto:paul.m.ramirez@jpl.nasa.gov>>
Subject: Re: pushPull component problem

Hey Chintu,

Sorry that you have been having trouble and thanks for pasting the relevant portions
of push pull config to diagnose this.

I think you have an issue in your RemoteFileSpecs file (TEST_AVHRR_ONLY.xml) --
one quick way to debug this is to use the tool that Brian Foster wrote:

(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
$ cd bin
$ java –Djava.ext.dirs=../lib org.apache.oodt.cas.pushpull.util.ExpressionValidator

It's a simple GUI that will allow you to put in regular expressions from your push pull
RmoteFileSpecs files, and then put in file names from the server to see if they match.

Can you run the above and let me know if it's matching? If not, we've found your issue.
If it is matching, we'll move on to the next set of things to check.

Thanks!

Cheers,
Chris

From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <chintu.mistry@nasa.gov<mailto:chintu.mistry@nasa.gov>>
Date: Thursday, March 28, 2013 5:47 AM
To: "dev@oodt.apache.org<mailto:dev@oodt.apache.org>" <dev@oodt.apache.org<mailto:dev@oodt.apache.org>>
Cc: jpluser <chris.a.mattmann@jpl.nasa.gov<mailto:chris.a.mattmann@jpl.nasa.gov>>,
"Ramirez, Paul M (388J)" <paul.m.ramirez@jpl.nasa.gov<mailto:paul.m.ramirez@jpl.nasa.gov>>
Subject: pushPull component problem

Hi,

I have been playing around with push pull for last two days and trying to download some data
from remote location without success.
I would appreciate if you guys can shed some light on what is going on and why isn't push
pull find files to download even though its clearly there at remote location.

Here is my configuration. I am trying to download  /pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz
file from eclipse.ncdc.noaa.gov<http://eclipse.ncdc.noaa.gov>. It logs-in to the server
fine and it lists all the files as well. However for some reason it does not match to the
rule in TEST_AVHRR_ONLY.xml. Not sure what I am doing wrong. I have tried almost all combinations
and still could not get it to work.

In my /etc/pushpull.properties file, I have types=false so that it downloads the file even
if its not specified in mime-types.xml file.
org.apache.oodt.cas.pushpull.allow.only.defined.types=false

ExternalSources.xml
<sources>
    <source host="eclipse.ncdc.noaa.gov<http://eclipse.ncdc.noaa.gov>">
        <login type="ftp" alias="eclipseClass">
            <username>anonymous</username>
            <password>user@host.com<mailto:user@host.com></password>
        </login>
    </source>
</sources>

RemoteSpecs.xml
        <daemon alias="eclipseClass" active="yes">
            <runInfo firstRunDateTime="2013-03-25T00:00:00Z" period="1m" runOnReboot="yes"/>
            <propInfo dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
                <propFiles regExp="TEST_AVHRR_ONLY\.xml" parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser"/>
            </propInfo>
            <dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no" queryElement="Filename"/>
        </daemon>

TEST_AVHRR_ONLY.xml
<root>
    <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
        <nodirs/>
        <!-- <file name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> -->
        <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
    </dirstruct>
</root>


Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047
<typescript.txt>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message