oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (3980)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: catalog queries in pge config files and filemgr-client problem
Date Mon, 31 Aug 2015 20:02:19 GMT
Val, I would inspect your file manager catalog. Can you use the OPSUI
or some other browser (Lucene) to see the metadata that is recorded
for your files? I think this is part of the problem - from the query
the only reason that $FileLocation/$Filename wouldn’t be resolved in
your queries is that they aren’t recorded as metadata fields for your
products.

Can you check?

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++





-----Original Message-----
From: Lewis John Mcgibbney <lewis.mcgibbney@gmail.com>
Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Date: Thursday, August 27, 2015 at 5:07 PM
To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Subject: Re: catalog queries in pge config files and filemgr-client problem

>Hi Val,
>Wise idea to start debugging this from the filemgr-client.
>Can you please try changing logging levels in the following file to DEBUG
>
>https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/loggi
>ng.properties#L40-L74
>
>If you try your queries again then hopefully you should see some
>patterns/indications within FileMgr logs to indicate what is going on.
>
>
>On Wednesday, August 26, 2015, Mallder, Valerie
><Valerie.Mallder@jhuapl.edu>
>wrote:
>
>> Hi All,
>>
>> So, I'm phasing off of the New Horizons project for now and I am back to
>> my science data pipeline work.  I am in the process of trying to pick up
>> where I left off a few months ago - but with a twist J  I graduated from
>> RADiX and now I am trying to get my pipeline system set up to run with
>>the
>> 'regular' OODT.  I have made extremely good progress so far. I have
>>pretty
>> much replicated all of the functionality of filemgr and workflow manager
>> that I had been using with RADiX, with the exception of one thing that
>> doesn't seem to be working the way it used too.  And I'm hoping someone
>>out
>> there can give me some ideas on how to fix it.
>>
>> I have a catalog query in one of my PGE config files. I copied the
>>syntax
>> for the query statement directly from one of Chris' PGE config files
>> (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
>>
>> Here is Chris' statement:
>>
>>     <metadata key="InputFiles"
>> 
>>val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTim
>>e'){SELECT
>> FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}"/>
>>
>> Here is my statement:
>>
>>      <metadata key="InitialDirList"
>> 
>>val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTim
>>e'){SELECT
>> FileLocation,Filename,CAS.ProductReceivedTime FROM ScienceFile}"/>
>>
>> And here are the results from the query:
>>
>>
>> 
>>$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$
>>FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$F
>>ileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fi
>>leLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fil
>>eLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$File
>>Location/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
>>ocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLo
>>cation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
>>ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoca
>>tion/$Filename,
>>
>> And those results are not exactly what I wanted J
>>
>> So I ran some experiments with ./filemgr-client on the command line to
>> make sure the files I expected to be in the catalog were in fact in the
>> catalog.
>>
>> The following operation worked just as expected.  The expected number of
>> products were listed.
>> ./filemgr-client --url http://localhost:9016 --operation --getFirstPage
>> --productTypeName ScienceFile
>>
>> However, this query gave some unexpected results. And this may be
>>related
>> the problem with the query in my PGE config file, but I am not sure.
>> ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
>>-query
>> "SELECT FileLocation,Filename FROM ScienceFile"
>>
>> That query only outputs several blank lines to the screen and nothing
>>else.
>>
>> I am not very familiar with all of the catalog related code yet so I am
>> not sure where to start looking for the problem.  If you know this area
>>of
>> the code well and have some debugging ideas, please let me know.
>>
>> Thanks so much!
>>
>> Val
>>
>>

Mime
View raw message