Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@oodt.apache.org
User-Agent: Microsoft-MacOutlook/14.5.5.150821
Date: Tue, 29 Sep 2015 09:58:41 -0700
Subject: Re: file staging question
From: Chris Mattmann <chris.mattmann@gmail.com>
To: <dev@oodt.apache.org>
Message-ID: <D23010F5.4645A%chris.mattmann@gmail.com>
Thread-Topic: file staging question
References: <49ddae7c5bc14dcaaf57cc5f40f2756f@aplex01.dom1.jhuapl.edu>
In-Reply-To: <49ddae7c5bc14dcaaf57cc5f40f2756f@aplex01.dom1.jhuapl.edu>
Mime-version: 1.0
Content-type: text/plain;
	charset="UTF-8"
Content-transfer-encoding: quoted-printable

No problem Val.

The fileStaging feature was sort of half baked, however
it does work and here is the basic concept behind it.
If you have files in your CAS-PGE job that need to be
copied from controlled access storage and staged into the
job directory for the running CAS-PGE job, the fileStager
is the way to do it. You give it a FM query that defines
what files to select, and then it uses the File Manager
configured data transferer to copy the file(s) from controlled
access storage to the PGE job working dir.

Another way to accomplish this is simply in your CAS-PGE
exe block and cmd blocks is to add a cp or remote staging
command (rsync, scp, etc.) that accomplishes the same thing.

Cheers,
Chris

=E2=80=94
Chris Mattmann
chris.mattmann@gmail.com


-----Original Message-----
From: "Mallder, Valerie" <Valerie.Mallder@jhuapl.edu>
Reply-To: <dev@oodt.apache.org>
Date: Tuesday, September 29, 2015 at 9:51 AM
To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Subject: file staging question

>Hi All,
>
>I have a question related to the file staging feature in a PGE Task.
>Basically, how are you using this feature and what exactly does it do for
>you?
>
>Does it just move some files from one place to another?  If so, does the
>destination directory have to exist or does it get created?  And, does
>the destination directory structure need to be the same as the source
>directory structure?
>
>When does the 'staging' take place?  Does it execute before the <exe>
>section of the PGE?
>
>
>Thanks,
>Val
>
>P.S. I have read every line of ever file in OODT that contains the word
>'staging' and it is still not clear to me what this feature actually does
>or how I can configure my PGE config file to use it.   But, I did find
>some totally awesome documentation in the curator J  But, it made it
>sound like the directory you set up as your 'staging' directory is the
>directory that files get ingested from. Whereas, I was under the
>impression that the directory you specify as your <output> directory is
>the directory that files get ingested from.  So, it was confusing. L
>Wonderfully thorough documentation, just confusing for my case.
>
>
>Valerie A. Mallder
>
>New Horizons Deputy Mission System Engineer
>The Johns Hopkins University/Applied Physics Laboratory
>11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
>240-228-7846 (Office) 410-504-2233 (Blackberry)
>