airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: [DISCUSS] Enhance Airavata support for CLI's
Date Wed, 30 Oct 2013 18:00:01 GMT
Hi All,

We let this topic live in the archives for 17 months, its probably has
baked enough. How about we revisit this thread and startup application use
cases and make it into a list of features Airavata should support for
Command Line Applications.

You can browse through the thread at -
http://markmail.org/thread/hd7azhp7w7o7eqyq

As we brainstorm, I will stratup a wiki document and capture the outcomes
and finally we can review the document and vote on it.

Suresh


On Sun, May 13, 2012 at 9:49 AM, Suresh Marru <smarru@apache.org> wrote:

> Hi All,
>
> I am trying to revisit the Airavata support for all command line options
> we pass to applications. Airavata's goal is to make end users oblivious to
> any application execution details, but application service providers need
> flexibility to configure all possible application options.
>
> Some terminology like arguments vs parameters vs attributes get ambiguous.
> They differ by definition but in practice they are often used
> interchangeably. For Airavata, we should avoid a confusion between whats
> exposed in wsdl's vs whats passed to application. This matches the
> semantics as well, for instance, an argument is an instance of parameter.
> This discussion is about what Airavata passes to the command line
> applications. I am not suggesting any changes to wsdl's and schemas which
> use xml definitions. For applications I am suggesting to use the
> terminology per POSIX standard definitions [1]. I also propose that we
> should try and follow the utility syntax guidelines [2]. If an application
> does not follow these guidelines, we suggest it be wrapped by a shell
> script so we can pass arguments and flags confirming to standard practices.
>
> Application refers to the commands airavata executes on computational
> resources.
>
> Working directory. Airavata should insist on executing each invocation in
> a unique working directory. Some applications try and change to a static
> directory, but if proper uniqueness is not followed for output and log
> files, we risk overwriting executions producing unintended outputs. Also,
> avoid writing to home directories and source directories. This might have
> side effects and a overrun log file might fill the disk space and freeze
> further usage of that account.
>
> Arguments:
> *  should support application arguments and provide a way to specify both
> required and optional.
> In the case of optional parameters, the resulting wsdl's attributes should
> have minOccurs=0 and airavata should skip passing that value to application
> (if not specified).
>
> * Airavata *should not* support arguments with operands followed by
> commands. These additional commands get forked without having control over
> the process id and monitoring and exit status of these series of commands
> gets tricky. More over, the underlying grid job managers do not like
> treating a chain of commands as one executable. Rather encourage explicitly
> specifying the execution chain and associated I/O.
>
> * Airavata should also support flags only ( they serve different purpose
> than option flags). Flags normally prefix with '--'. These flags control
> the execution of the application like --verbose, --fast, --use-fft, e.t.c
>
> * Arguments can be passed to the application as standardinput (with
> redirector operator) or as name-value pairs or with option flags. The
> option flags should always prefix with the POSIX standard of '-'.
>
> * If the arguments are preceded by an option flag they do not need to be
> ordered. But if the arguments are passed just as values, applications are
> sensitive to the order the arguments are passed. In this case, optional
> arguments have to carefully handled, as missing an argument in between will
> mislead.
>
> * If an argument is a file type, and if the file has a remote supported
> protocols of (http, ftp, gsiftp, s3) then the file has to be staged first
> and only local path passed to the application. Application should be able
> to consume the full local path and if only basename is required, it should
> be able to handle it internally.
>
> * If an application requires a remove ftp url as an argument, then it
> should be specified as a string, in which case Airavata will skip staging
> that url and will pass the url as is to the application.
>
> * Implicit Parameters: As much as possible, Airavata should insist on
> one-on-one match between inputs specified in service description to whats
> passed to application. But there will be exceptions like fortran
> applications which uses NAMELIST standard to specify all inputs in a config
> file and pass only this file to the application. In these cases, the
> application still needs to stage some data files to the remote compute
> server but these file names or implicitly specified in the application. The
> application typically looks for these files relative to working directory
> or to input namelist file.
>
> Outputs:
> * Airavata should support standard outputs and errors and optionally
> provide a way to specify the names of stdout and stderr.
> * All outputs required to be staged out of the compute machine or scratch
> working directory be explicitly specified.
> * If the output file name(s) are predetermined or specified at in a config
> file, then the name should be specified in application description. In the
> cases, where output file names are not deterministic, a regular expression
> or a containing directory should be specified.
> * If the application requires the output file name be passed at command
> line like -out output.txt, then airavata should provide support for these
> outputs flags.
> * Airavata should support outputs which can be optionally produced. If an
> optional output is not generated but application exits with exit code 0,
> then the application should be marked as success. (A different discussion
> on application execution success criteria is needed).
> * A default output data directory should be created on the remote compute
> resource. The application description should be able to specific an
> overriding name for this directory.
> * Airavata should support applications/shell script wrappers which print
> name-value pairs of output content or file paths to standard out.
>
> Once we discuss this topic, we should raise JIRAs for any missing features
> and also add these on website/wiki.
>
> Cheers,
> Suresh
>
> [1] -
> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html
> [2] -
> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html#tag_12_02
>
>
>

Mime
View raw message