oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mallder, Valerie" <Valerie.Mall...@jhuapl.edu>
Subject RE: catalog queries in pge config files and filemgr-client problem
Date Mon, 31 Aug 2015 22:15:27 GMT
No problem!


Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory


> -----Original Message-----
> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> Sent: Monday, August 31, 2015 6:02 PM
> To: dev@oodt.apache.org
> Subject: Re: catalog queries in pge config files and filemgr-client problem
> 
> Grand
> Apologies for incorrect advice on logging Level. Log4j Vs.
> java.util.Logging I think!
> 
> On Mon, Aug 31, 2015 at 2:41 PM, Mallder, Valerie < Valerie.Mallder@jhuapl.edu>
> wrote:
> 
> > Hi Lewis,
> >
> > Thanks Lewis, I tried setting it to DEBUG, that it told me it wasn't a
> > recognized level. But, I used FINEST and added more log message and
> > eventually found that I had run into the issue that has already been
> > documented in OODT-854 which is supposed to be fixed in 0.10.  It has
> > to do with where I was putting my policy files for the filemgr.  It
> > sounds like the fix for OODT-854 will solve my problem.  Once I put
> > all of my policy files in FILEMGR_HOME/policy/core, the catalog query started
> working.
> >
> > Thanks,
> > Val
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> > > Sent: Thursday, August 27, 2015 8:07 PM
> > > To: dev@oodt.apache.org
> > > Subject: Re: catalog queries in pge config files and filemgr-client
> > problem
> > >
> > > Hi Val,
> > > Wise idea to start debugging this from the filemgr-client.
> > > Can you please try changing logging levels in the following file to
> > > DEBUG
> > >
> > >
> > https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/l
> > ogging.pro
> > > perties#L40-L74
> > >
> > > If you try your queries again then hopefully you should see some
> > > patterns/indications within FileMgr logs to indicate what is going on.
> > >
> > >
> > > On Wednesday, August 26, 2015, Mallder, Valerie <
> > Valerie.Mallder@jhuapl.edu>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > So, I'm phasing off of the New Horizons project for now and I am
> > > > back to my science data pipeline work.  I am in the process of
> > > > trying to pick up where I left off a few months ago - but with a
> > > > twist J  I graduated from RADiX and now I am trying to get my
> > > > pipeline system set up to run with the 'regular' OODT.  I have
> > > > made extremely good progress so far. I have pretty much replicated
> > > > all of the functionality of filemgr and workflow manager that I
> > > > had been using with RADiX, with the exception of one thing that
> > > > doesn't seem to be working the way it used too.  And I'm hoping
> > > > someone out there can
> > give me
> > > some ideas on how to fix it.
> > > >
> > > > I have a catalog query in one of my PGE config files. I copied the
> > > > syntax for the query statement directly from one of Chris' PGE
> > > > config files
> > > > (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
> > > >
> > > > Here is Chris' statement:
> > > >
> > > >     <metadata key="InputFiles"
> > > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductRece
> > > > ived Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime
> > > > FROM RatLog}"/>
> > > >
> > > > Here is my statement:
> > > >
> > > >      <metadata key="InitialDirList"
> > > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductRece
> > > > ived Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime
> > > > FROM ScienceFile}"/>
> > > >
> > > > And here are the results from the query:
> > > >
> > > >
> > > > $FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Fil
> > > > enam
> > > > e,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$F
> > > > ilen
> > > > ame,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/
> > > > $Fil
> > > > ename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocatio
> > > > n/$F
> > > > ilename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocat
> > > > ion/
> > > > $Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
> > > > atio
> > > > n/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
> > > > ocat
> > > > ion/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fil
> > > > eLoc
> > > > ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$F
> > > > ileL ocation/$Filename,$FileLocation/$Filename,
> > > >
> > > > And those results are not exactly what I wanted J
> > > >
> > > > So I ran some experiments with ./filemgr-client on the command
> > > > line to make sure the files I expected to be in the catalog were
> > > > in fact in the catalog.
> > > >
> > > > The following operation worked just as expected.  The expected
> > > > number of products were listed.
> > > > ./filemgr-client --url http://localhost:9016 --operation
> > > > --getFirstPage --productTypeName ScienceFile
> > > >
> > > > However, this query gave some unexpected results. And this may be
> > > > related the problem with the query in my PGE config file, but I am
> > > > not
> > sure.
> > > > ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
> > > > -query "SELECT FileLocation,Filename FROM ScienceFile"
> > > >
> > > > That query only outputs several blank lines to the screen and
> > > > nothing
> > else.
> > > >
> > > > I am not very familiar with all of the catalog related code yet so
> > > > I am not sure where to start looking for the problem.  If you know
> > > > this area of the code well and have some debugging ideas, please
> > > > let me
> > know.
> > > >
> > > > Thanks so much!
> > > >
> > > > Val
> > > >
> > > >
> >
> 
> 
> 
> --
> *Lewis*
Mime
View raw message