oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mallder, Valerie" <Valerie.Mall...@jhuapl.edu>
Subject RE: file staging question
Date Tue, 29 Sep 2015 17:45:11 GMT
Awesome. Thanks Chris. I'm actually already using the cp command in my exe block to do something
similar. I had been wondering if file staging was a better (or more elegant) way to do what
I needed to do. But, I will just continue doing what I'm doing :)  Thanks!


Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory


> -----Original Message-----
> From: Chris Mattmann [mailto:chris.mattmann@gmail.com]
> Sent: Tuesday, September 29, 2015 12:59 PM
> To: dev@oodt.apache.org
> Subject: Re: file staging question
> 
> No problem Val.
> 
> The fileStaging feature was sort of half baked, however it does work and here is
> the basic concept behind it.
> If you have files in your CAS-PGE job that need to be copied from controlled
> access storage and staged into the job directory for the running CAS-PGE job, the
> fileStager is the way to do it. You give it a FM query that defines what files to
> select, and then it uses the File Manager configured data transferer to copy the
> file(s) from controlled access storage to the PGE job working dir.
> 
> Another way to accomplish this is simply in your CAS-PGE exe block and cmd
> blocks is to add a cp or remote staging command (rsync, scp, etc.) that
> accomplishes the same thing.
> 
> Cheers,
> Chris
> 
> —
> Chris Mattmann
> chris.mattmann@gmail.com
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: "Mallder, Valerie" <Valerie.Mallder@jhuapl.edu>
> Reply-To: <dev@oodt.apache.org>
> Date: Tuesday, September 29, 2015 at 9:51 AM
> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Subject: file staging question
> 
> >Hi All,
> >
> >I have a question related to the file staging feature in a PGE Task.
> >Basically, how are you using this feature and what exactly does it do
> >for you?
> >
> >Does it just move some files from one place to another?  If so, does
> >the destination directory have to exist or does it get created?  And,
> >does the destination directory structure need to be the same as the
> >source directory structure?
> >
> >When does the 'staging' take place?  Does it execute before the <exe>
> >section of the PGE?
> >
> >
> >Thanks,
> >Val
> >
> >P.S. I have read every line of ever file in OODT that contains the word
> >'staging' and it is still not clear to me what this feature actually does
> >or how I can configure my PGE config file to use it.   But, I did find
> >some totally awesome documentation in the curator J  But, it made it
> >sound like the directory you set up as your 'staging' directory is the
> >directory that files get ingested from. Whereas, I was under the
> >impression that the directory you specify as your <output> directory is
> >the directory that files get ingested from.  So, it was confusing. L
> >Wonderfully thorough documentation, just confusing for my case.
> >
> >
> >Valerie A. Mallder
> >
> >New Horizons Deputy Mission System Engineer The Johns Hopkins
> >University/Applied Physics Laboratory
> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
> >240-228-7846 (Office) 410-504-2233 (Blackberry)
> >
> 

Mime
View raw message