airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shameera Rathnayaka <shameerai...@gmail.com>
Subject Re: Improvements to Experiment input data model in order to support Gaussian application
Date Thu, 11 Dec 2014 08:02:28 GMT
Hi Amila,

According to my understanding, what this handle does is, read the user
given configuration at run time. I have no idea this will effect to qsub or
aprun or other parameter. It is better if someone explain it to me too.

We already have a way to provide the these configuration parameters with
the experiment itself by defining ComputeResourceScheduling. But there are
some use cases like gaussian, where gaussian users provide these
configurations with the input file it self.  IMO here we have two options,
either we can  ask gaussian users to adopt to the airavata way but still
those configuration in input file is required for gaussian application(I
guess , correct me if I am wrong here) or use airavata extension points to
support this scenario. Here the handler address the second option.

Thanks,
Shameera.

On Thu, Dec 11, 2014 at 12:00 AM, Amila Jayasekara <thejaka.amila@gmail.com>
wrote:

> Also, regarding the handler that Shameera is working on ...
> I guess that handler is going to change mainly "qsub" parameters or
> "aprun" parameters (Correct me if i am wrong). I think it would be more
> useful to write a handler which changes any parameter in "qsub", "aprun" or
> "mpiexec".
>
> In implementation wise I would imagine there is an abstract handler with
> concrete implementation for each job scheduling command.
>
> Thanks
> -Amila
>
> On Wed, Dec 10, 2014 at 9:17 AM, Marlon Pierce <marpierc@iu.edu> wrote:
>
>> +1 for more generalization.
>>
>> We are collecting more raw material for chemistry application use cases
>> at https://cwiki.apache.org/confluence/display/AIRAVATA/Use+Cases. We'll
>> review them (and bio apps that we also collected previously) in a wiki
>> document to see if our API mappings are correct.
>>
>> Preliminarily, we see the command line arguments don't contain the full
>> list of input and output files.  Additional required inputs may be passed
>> via control files, environment variables, etc.  Examples include data
>> libraries for basis functions, names of checkpoint files, names of output
>> files, and so forth.  So we need a way to say the application may take 4
>> inputs, but only 1 is needed to construct a valid command line, for example.
>>
>> On the other hand, I don't think we need the InputMetadataType that
>> Chathuri introduces below. This overlaps with what is already in the
>> compute resource description fields.
>>
>>
>> Marlon
>>
>>
>> On 12/8/14, 10:17 PM, Amila Jayasekara wrote:
>>
>>> Hi Chathuri,
>>>
>>> I do not know anything about Gaussian. So its kind of hard for me to
>>> understand what exactly is the meaning of the structures you introduced
>>> and
>>> why you exactly need those structures.
>>>
>>> A more important question is how to come up with a more abstract and
>>> generic thrift IDLS so that you dont need to change it every time we add
>>> a
>>> new application. Going through many example applications is certainly a
>>> good way to understand broad requirements and helps to abstract out many
>>> features.
>>>
>>> Thanks
>>> -Thejaka
>>>
>>> On Mon, Dec 8, 2014 at 10:22 AM, Chathuri Wimalasena <
>>> kamalasini@gmail.com>
>>> wrote:
>>>
>>>  Hi Devs,
>>>>
>>>> We are trying to add Gaussian application using airavata-appcatalog.
>>>> While
>>>> doing that, we face some limitations of the current design.
>>>>
>>>> In Gaussian there are several input files, some input files should used
>>>> when the job run command is generated, but some does not.  Those which
>>>> are
>>>> not involved with job run command also need to be staged to working
>>>> directory. Such flags are not supported in current design.
>>>>
>>>> Another interesting feature that in Gaussian is, in input file, we can
>>>> specify the values for memory, cpu like options. If input file includes
>>>> those parameters, we need to give priority to those values instead of
>>>> the
>>>> values specified in the request.
>>>>
>>>> To support these features, we need to slightly modify our thrift IDLS,
>>>> specially to InputDataObjectType struct.
>>>>
>>>> Current struct is below.
>>>>
>>>> struct InputDataObjectType {
>>>>      1: required string name,
>>>>      2: optional string value,
>>>>      3: optional DataType type,
>>>>      4: optional string applicationArgument,
>>>>      5: optional bool standardInput = 0,
>>>>      6: optional string userFriendlyDescription,
>>>>      7: optional string metaData
>>>> }
>>>>
>>>> In order to support 1st requirement, we introduce 2 enums.
>>>>
>>>> enum InputValidityType{
>>>> REQUIRED,
>>>> OPTIONAL
>>>> }
>>>>
>>>> enum CommandLineType{
>>>> INCLUSIVE,
>>>> EXCLUSIVE
>>>> }
>>>>
>>>> Please excuse me for names. You are welcome to suggest better names.
>>>>
>>>> To support 2nd requirement, we change metaData field to a map with
>>>> another
>>>> enum where we define all the metadata types that can have.
>>>>
>>>> enum InputMetadataType {
>>>>      MEMORY,
>>>>      CPU
>>>> }
>>>>
>>>> So the new InputDataObjectType would be as below.
>>>>
>>>> struct InputDataObjectType {
>>>>      1: required string name,
>>>>      2: optional string value,
>>>>      3: optional DataType type,
>>>>      4: optional string applicationArgument,
>>>>      5: optional bool standardInput = 0,
>>>>      6: optional string userFriendlyDescription,
>>>>    *  7: optional map<InputMetadataType, string> metaData,*
>>>> *    8: optional InputValidityType inputValid;*
>>>> *    9: optional CommandLineType addedToCommandLine;*
>>>> *    10: optional bool dataStaged = 0;*
>>>> }
>>>>
>>>> Suggestions are welcome.
>>>>
>>>> Thanks,
>>>> Chathuri
>>>>
>>>>
>>>>
>>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Mime
View raw message