airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miller, Mark" <>
Subject RE: "input" and "output" subdirs in working directories
Date Tue, 23 Dec 2014 20:04:14 GMT
I believe this would work. Perhaps others can correct if there are cases where it is wrong.

-----Original Message-----
From: Marlon Pierce [] 
Sent: Tuesday, December 23, 2014 12:02 PM
Subject: Re: "input" and "output" subdirs in working directories

So let's take $working_dir as the base and upload all inputs to $working_dir/input. Then in
the job script we set the working directory to $working_dir/output and run the application.

Would this be reasonably guaranteed to put all outputs in the $working_dir/output directory,
or are there codes for which this would not work?


On 12/23/14, 2:54 PM, Miller, Mark wrote:
> We do use a flat system. However, to return files in a very controlled way, since one
has to know precisely the names of input and output files, and maybe even restrict the user's
ability to name them. The codes we use produce files conditionally, depending on the command
line, and so unless you know the code super well, using explicit names can cause certain output
files to now be returned at all, an obvious bummer for user and developer alike. As a result,
I have elected in most cases to just return all files. This can be confusing for users, and
just binning input and output as coarse categories is something I wish I could do currently.
> That was my motivation in responding....
> Mark
> -----Original Message-----
> From: Marlon Pierce []
> Sent: Tuesday, December 23, 2014 11:50 AM
> To:
> Subject: Re: "input" and "output" subdirs in working directories
> Thanks, Mark. What's your approach in CIPRES?  Do you use a flat structure for your working
directories, or do you do something else?
> Marlon
> On 12/23/14, 2:45 PM, Miller, Mark wrote:
>> I am not sure of the reasoning behind the design, but off-hand it seems to me it
would simplify the job of returning to "input" and "output" files as discrete entities, so
if you don't know which files are which, you would have at least two coarse bins that can
be returned under separate banners.
>> Mark
>> -----Original Message-----
>> From: Marlon Pierce []
>> Sent: Tuesday, December 23, 2014 11:01 AM
>> To: Airavata Dev
>> Subject: "input" and "output" subdirs in working directories
>> When Airavata executes a remote command (launching a SLURM job, for example), it
creates a working directory on the target machine's scratch space and two subdirectories,
"input" and "output".  Is there a good reason for creating these two subdirectories?  Why
not just do all the work in the top level of the working directory?  It seems unnecessary.
>> Also, I don't understand why these are in the GFAC module, as these should be constructed
from Registry information.
>> Below is background information.
>> --------------
>> Below is an example working directory.
>> $ cd
>> /oasis/scratch/trestles/ogce/temp_project/gta-work-dirs/TEST_8b10aa04
>> -
>> 95c3-4695-af77-d3b3987c7ef9/
>> $ ls -tlr
>> total 20
>> drwxr-xr-x 2 ogce sds128 4096 Dec 23 07:17 output
>> -rw-r--r-- 1 ogce sds128  831 Dec 23 07:39 1203922204.pbs
>> -rw------- 1 ogce sds128   28 Dec 23 07:40 Gaussian.stdout
>> -rw------- 1 ogce sds128  663 Dec 23 07:40 Gaussian.stderr drwxr-xr-x
>> 2 ogce sds128 4096 Dec 23 07:47 input
>> The names of these subdirectories are specified in (as
>> files in the GFAC module that use these two constants.
>> $ find ./modules/gfac -type f -exec grep -il "OUTPUT_DATA_DIR_VAR" {} 
>> \;
>> | grep java|grep -v target
>> ./modules/gfac/gfac-core/src/main/java/org/apache/airavata/gfac/Const
>> a
>> ./modules/gfac/gfac-core/src/main/java/org/apache/airavata/gfac/core/
>> c
>> pi/
>> ./modules/gfac/gfac-gram/src/main/java/org/apache/airavata/gfac/gram/
>> u
>> til/
>> ./modules/gfac/gfac-local/src/main/java/org/apache/airavata/gfac/loca
>> l
>> /provider/impl/
>> ./modules/gfac/gfac-ssh/src/main/java/org/apache/airavata/gfac/ssh/pr
>> o
>> vider/impl/
>> So we would need to clean these up if we remove the constants.
>> Marlon

View raw message