oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mistry, Chintu (GSFC-586.0)[SGT INC]" <chintu.mis...@nasa.gov>
Subject Re: pushPull component problem
Date Thu, 28 Mar 2013 16:28:55 GMT
Sorry about not explaining properly. When I said "failed to run", I meant
class not found. Did not really look deep enough what the problem is.
Apparently I did not include all jar files properly.

Anyway, I managed to run both ftp client (of course I did change
ProtocolFactor.xml file). But still no luck. Attaching files with output.

Ftp4CheFtp => Completely barfed
jvFTP => runs fine but same output as the other plugins.



Regards
-- 
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047






On 3/28/13 12:13 PM, "Mattmann, Chris A" <chris.a.mattmann@jpl.nasa.gov>
wrote:

>Hey Chintu,
>
>Thanks -- sorry to be asking all these questions, but what does "failed to
>run" mean?
>
>Did you update the protocol plugins config to ref the new plugins and
>their FQCN (fully
>qualified class names?) Do you have a log file for that?
>
>Cheers,
>Chris
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW:  http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>-----Original Message-----
>From: <Mistry>, "Chintu [SGT INC]  (GSFC-586.0)" <chintu.mistry@nasa.gov>
>Date: Thursday, March 28, 2013 9:12 AM
>To: jpluser <chris.a.mattmann@jpl.nasa.gov>, "dev@oodt.apache.org"
><dev@oodt.apache.org>
>Cc: "Ramirez, Paul M (388J)" <paul.m.ramirez@jpl.nasa.gov>
>Subject: Re: pushPull component problem
>
>>Since we are not using OODT 0.6, it did not compile straight away.
>>However, I did manage to compile it with "0.5" version of
>>cas-protocol-api. But when I was running it, It failed to run.
>>
>>--
>>Chintu Mistry
>>NASA Goddard Space Flight Center
>>Bldg L40B, Room S776
>>Office: 240 684 0477
>>Mobile: 770 310 1047
>>
>>
>>
>>
>>
>>
>>On 3/28/13 12:04 PM, "Mattmann, Chris A" <chris.a.mattmann@jpl.nasa.gov>
>>wrote:
>>
>>>Hey Chintu,
>>>
>>>Thanks for identifying this!
>>>
>>>Have you tried the alternative plugins? They use a diff set of FTP
>>>protocol
>>>plugins (that unfortunately are not ALv2 licensed or Category A
>>>approved,
>>>so we can't ship with Apache OODT).
>>>
>>>They may do better at handling that symlink than the default version of
>>>commons-ftp
>>>we're using.
>>>
>>>Let me know. We'll investigate.
>>>
>>>Cheers,
>>>Chris
>>>
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>Chris Mattmann, Ph.D.
>>>Senior Computer Scientist
>>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>Office: 171-266B, Mailstop: 171-246
>>>Email: chris.a.mattmann@nasa.gov
>>>WWW:  http://sunset.usc.edu/~mattmann/
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>Adjunct Assistant Professor, Computer Science Department
>>>University of Southern California, Los Angeles, CA 90089 USA
>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>>
>>>
>>>
>>>-----Original Message-----
>>>From: <Mistry>, "Chintu [SGT INC]  (GSFC-586.0)"
>>><chintu.mistry@nasa.gov>
>>>Date: Thursday, March 28, 2013 9:02 AM
>>>To: jpluser <chris.a.mattmann@jpl.nasa.gov>, "dev@oodt.apache.org"
>>><dev@oodt.apache.org>
>>>Cc: "Ramirez, Paul M (388J)" <paul.m.ramirez@jpl.nasa.gov>
>>>Subject: Re: pushPull component problem
>>>
>>>>Chris,
>>>>
>>>>I figured out the problem. Don't know the solution.
>>>>
>>>>So the problem is the symbolic links on FTP server.
>>>>
>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR  is actually
>>>>/san1/oisst/NetCDF/2013/AVHRR
>>>>Where pub -> san1
>>>>  And OI-daily-v2 -> oisst
>>>>
>>>>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it
>>>>works.
>>>>Otherwise it does not work.
>>>>
>>>>Another example is : ftp://aftp.cmdl.noaa.gov
>>>>
>>>>/data/radiation/surfrad/dra/2013
>>>>->/data/radiation/surfrad/Desert_Rock_NV/2013
>>>>
>>>>
>>>>
>>>>I don't know how to fix this. Well, not exactly sure where to look. It
>>>>appears that the filename filters are being applied based on the path
>>>>specified in DirStructParserFiles instead of just the filename. Because
>>>>I
>>>>can see the correct path when I print RemoteFile (path =
>>>>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz
>>>>'
>>>>)
>>>>on STDOUT.
>>>>
>>>>
>>>>Hopefully you guys can fix this fast or let me know where to look.
>>>>
>>>>Regards
>>>>--
>>>>Chintu Mistry
>>>>NASA Goddard Space Flight Center
>>>>Bldg L40B, Room S776
>>>>Office: 240 684 0477
>>>>Mobile: 770 310 1047
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>On 3/28/13 10:15 AM, "Mattmann, Chris A"
>>>><chris.a.mattmann@jpl.nasa.gov>
>>>>wrote:
>>>>
>>>>>Hey Chintu,
>>>>>
>>>>>This totally helped, thanks.
>>>>>
>>>>>Can you take a look at the plugins here?
>>>>>
>>>>>https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html
>>>>>
>>>>>
>>>>>Try installing those, and let me know if either of those FTP
>>>>>plugins works better and actually downloads the filesŠ
>>>>>
>>>>>In the meanwhile I'll look at the logs and see what I can come up
>>>>>with.
>>>>>
>>>>>Cheers,
>>>>>Chris
>>>>>
>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>Chris Mattmann, Ph.D.
>>>>>Senior Computer Scientist
>>>>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>Office: 171-266B, Mailstop: 171-246
>>>>>Email: chris.a.mattmann@nasa.gov
>>>>>WWW:  http://sunset.usc.edu/~mattmann/
>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>Adjunct Assistant Professor, Computer Science Department
>>>>>University of Southern California, Los Angeles, CA 90089 USA
>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>-----Original Message-----
>>>>>From: <Mistry>, "Chintu [SGT INC]  (GSFC-586.0)"
>>>>><chintu.mistry@nasa.gov>
>>>>>Date: Thursday, March 28, 2013 7:12 AM
>>>>>To: jpluser <chris.a.mattmann@jpl.nasa.gov>, "dev@oodt.apache.org"
>>>>><dev@oodt.apache.org>
>>>>>Cc: "Ramirez, Paul M (388J)" <paul.m.ramirez@jpl.nasa.gov>
>>>>>Subject: Re: pushPull component problem
>>>>>
>>>>>>I have attached verbose log. I have added extra print statements and
>>>>>>compiled to see what is returned from remote site. Hope this helps.
>>>>>>
>>>>>>
>>>>>>--
>>>>>>Chintu Mistry
>>>>>>NASA Goddard Space Flight Center
>>>>>>Bldg L40B, Room S776
>>>>>>Office: 240 684 0477
>>>>>>Mobile: 770 310 1047
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>From: <Mattmann>, Chris A <chris.a.mattmann@jpl.nasa.gov>
>>>>>>Date: Thursday, March 28, 2013 10:05 AM
>>>>>>To: Chintu Mistry <chintu.mistry@nasa.gov>, "dev@oodt.apache.org"
>>>>>><dev@oodt.apache.org>
>>>>>>Cc: "Ramirez, Paul M" <paul.m.ramirez@jpl.nasa.gov>
>>>>>>Subject: Re: pushPull component problem
>>>>>>
>>>>>>
>>>>>>
>>>>>>Hey Chintu,
>>>>>>
>>>>>>
>>>>>>Thanks for sending this ‹ really helped verify it's something else.
>>>>>>
>>>>>>
>>>>>>Can you capture the log of push pull, with verbose logging turned
on,
>>>>>>and
>>>>>>maybe do a pastebin?
>>>>>>
>>>>>>
>>>>>>Cheers,
>>>>>>Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>>><chintu.mistry@nasa.gov>
>>>>>>Date: Thursday, March 28, 2013 7:03 AM
>>>>>>To: jpluser <chris.a.mattmann@jpl.nasa.gov>, "dev@oodt.apache.org"
>>>>>><dev@oodt.apache.org>
>>>>>>Cc: "Ramirez, Paul M (388J)" <paul.m.ramirez@jpl.nasa.gov>
>>>>>>Subject: Re: pushPull component problem
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Something else is wrong. I know that the regex is fine. I don't
even
>>>>>>>have regex, it is straight up file name. It should match.
>>>>>>>
>>>>>>>
>>>>>>>Thanks for looking into this.
>>>>>>>--
>>>>>>>Chintu Mistry
>>>>>>>NASA Goddard Space Flight Center
>>>>>>>Bldg L40B, Room S776
>>>>>>>Office: 240 684 0477
>>>>>>>Mobile: 770 310 1047
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>From: <Mattmann>, Chris A <chris.a.mattmann@jpl.nasa.gov>
>>>>>>>Date: Thursday, March 28, 2013 9:01 AM
>>>>>>>To: Chintu Mistry <chintu.mistry@nasa.gov>, "dev@oodt.apache.org"
>>>>>>><dev@oodt.apache.org>
>>>>>>>Cc: "Ramirez, Paul M" <paul.m.ramirez@jpl.nasa.gov>
>>>>>>>Subject: Re: pushPull component problem
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Hey Chintu,
>>>>>>>
>>>>>>>
>>>>>>>Sorry that you have been having trouble and thanks for pasting
the
>>>>>>>relevant portions
>>>>>>>of push pull config to diagnose this.
>>>>>>>
>>>>>>>
>>>>>>>I think you have an issue in your RemoteFileSpecs file
>>>>>>>(TEST_AVHRR_ONLY.xml) --
>>>>>>>one quick way to debug this is to use the tool that Brian Foster
>>>>>>>wrote:
>>>>>>>
>>>>>>>
>>>>>>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
>>>>>>>$ cd bin
>>>>>>>$ java ­Djava.ext.dirs=../lib
>>>>>>>org.apache.oodt.cas.pushpull.util.ExpressionValidator
>>>>>>>
>>>>>>>
>>>>>>>It's a simple GUI that will allow you to put in regular expressions
>>>>>>>from
>>>>>>>your push pull
>>>>>>>RmoteFileSpecs files, and then put in file names from the server
to
>>>>>>>see
>>>>>>>if they match.
>>>>>>>
>>>>>>>
>>>>>>>Can you run the above and let me know if it's matching? If not,
>>>>>>>we've
>>>>>>>found your issue.
>>>>>>>If it is matching, we'll move on to the next set of things to
check.
>>>>>>>
>>>>>>>
>>>>>>>Thanks!
>>>>>>>
>>>>>>>
>>>>>>>Cheers,
>>>>>>>Chris
>>>>>>>
>>>>>>>
>>>>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)"
>>>>>>><chintu.mistry@nasa.gov>
>>>>>>>Date: Thursday, March 28, 2013 5:47 AM
>>>>>>>To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>>>>>>>Cc: jpluser <chris.a.mattmann@jpl.nasa.gov>, "Ramirez, Paul
M
>>>>>>>(388J)"
>>>>>>><paul.m.ramirez@jpl.nasa.gov>
>>>>>>>Subject: pushPull component problem
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>I have been playing around with push pull for last two days
and
>>>>>>>>trying
>>>>>>>>to download some data from remote location without success.
>>>>>>>>I would appreciate if you guys can shed some light on what
is going
>>>>>>>>on
>>>>>>>>and why isn't push pull find files to download even though
its
>>>>>>>>clearly
>>>>>>>>there at remote location.
>>>>>>>>
>>>>>>>>
>>>>>>>>Here is my configuration. I am trying to download
>>>>>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_prelimina
>>>>>>>>r
>>>>>>>>y
>>>>>>>>.
>>>>>>>>n
>>>>>>>>c
>>>>>>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server
fine
>>>>>>>>and
>>>>>>>>it lists all the files as well. However for some reason it
>>>>>>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure
what I
>>>>>>>>am
>>>>>>>>doing wrong. I have tried almost all combinations and still
could
>>>>>>>>not
>>>>>>>>get it to work.
>>>>>>>>
>>>>>>>>
>>>>>>>>In my /etc/pushpull.properties file, I have types=false so
that it
>>>>>>>>downloads the file even if its not specified in mime-types.xml
>>>>>>>>file.
>>>>>>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>>>>>>>>
>>>>>>>>
>>>>>>>>ExternalSources.xml
>>>>>>>><sources>
>>>>>>>>    <source host="eclipse.ncdc.noaa.gov">
>>>>>>>>        <login type="ftp" alias="eclipseClass">
>>>>>>>>            <username>anonymous</username>
>>>>>>>>            <password>user@host.com</password>
>>>>>>>>        </login>
>>>>>>>>    </source>
>>>>>>>></sources>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>RemoteSpecs.xml
>>>>>>>>        <daemon alias="eclipseClass" active="yes">
>>>>>>>>            <runInfo firstRunDateTime="2013-03-25T00:00:00Z"
>>>>>>>>period="1m" runOnReboot="yes"/>
>>>>>>>>            <propInfo
>>>>>>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
>>>>>>>>                <propFiles regExp="TEST_AVHRR_ONLY\.xml"
>>>>>>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirSt
>>>>>>>>r
>>>>>>>>u
>>>>>>>>c
>>>>>>>>t
>>>>>>>>X
>>>>>>>>mlParser"/>
>>>>>>>>            </propInfo>
>>>>>>>>            <dataInfo stagingArea="AVHRR-ONLY"
>>>>>>>>deleteFromServer="no"
>>>>>>>>queryElement="Filename"/>
>>>>>>>>        </daemon>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>TEST_AVHRR_ONLY.xml
>>>>>>>><root>
>>>>>>>>    <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR">
>>>>>>>>        <nodirs/>
>>>>>>>>        <!-- <file
>>>>>>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/>
-->
>>>>>>>>        <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/>
>>>>>>>>    </dirstruct>
>>>>>>>></root>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Regards
>>>>>>>>--
>>>>>>>>Chintu Mistry
>>>>>>>>NASA Goddard Space Flight Center
>>>>>>>>Bldg L40B, Room S776
>>>>>>>>Office: 240 684 0477
>>>>>>>>Mobile: 770 310 1047
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message