airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sachith Withana <swsach...@gmail.com>
Subject Re: Orchestration Component implementation review
Date Mon, 20 Jan 2014 16:38:44 GMT
Okay. Will do. Will send you a draft asap.


On Mon, Jan 20, 2014 at 11:17 AM, Marlon Pierce <marpierc@iu.edu> wrote:

> I think we want to make everything explicit.  The Airavata API is
> intended for external clients, not communications between Airavata
> components (the SPI).  Your figure at
>
> https://cwiki.apache.org/confluence/display/AIRAVATA/Simple+Gateway+Developer+Guide
> summarizes this nicely.  You'll need to explain both the API (what a
> gateway does) and the SPI (how Airavata components work together) to do
> this.
>
>
> Marlon
>
> On 1/20/14 11:09 AM, Sachith Withana wrote:
> > We are using internal SPIs which are not reflected in the Airavata API.
> > should it be explained or just make it a higher level diagram which won't
> > show the SPIs?
> >
> >
> > On Mon, Jan 20, 2014 at 11:06 AM, Marlon Pierce <marpierc@iu.edu> wrote:
> >
> >> Thanks, Sachith.  Can you explain your question about APIs and SPIs a
> >> little more?
> >>
> >>
> >> Marlon
> >>
> >> On 1/20/14 10:53 AM, Sachith Withana wrote:
> >>> Hi All,
> >>>
> >>> I will go ahead and create the Wiki on the Orchestrator. Will send you
> >> all
> >>> a draft as soon as I can.
> >>>
> >>> One question though, Do we have to explicitly show the SPIs and APIs
> >> both?
> >>>
> >>> On Mon, Jan 20, 2014 at 9:46 AM, Marlon Pierce <marpierc@iu.edu>
> wrote:
> >>>
> >>>> +1 for real use cases first. We have at least 3.  But I'm sure we will
> >>>> want to make it as easy as possible for developers to pass back the
> >>>> correct, created experimentID when invoking launchExperiment.
> >>>>
> >>>>
> >>>> Marlon
> >>>>
> >>>> On 1/17/14 2:57 PM, Saminda Wijeratne wrote:
> >>>>> Marlon, I think until we put this to real use we wont get much
> feedback
> >>>> on
> >>>>> what aspects we should focus on more and in what features we should
> >>>> expand
> >>>>> or prioritize on. So how about having a test plan for the
> Orchestrator.
> >>>>> Expose it to real usecases and see how it will survive. WDYT?
> >>>>>
> >>>>> It might be a little confusing to return a "JobRequest" object from
> the
> >>>>> Orchestrator (since its a response). Or perhaps it should be renamed?
> >>>>>
> >>>>> Sachith, I think we should have a google hangout or a separate mail
> >>>> thread
> >>>>> (or both) to discuss muti-threaded support. Could you organize this
> >>>> please?
> >>>>> On Fri, Jan 17, 2014 at 10:29 AM, Amila Jayasekara
> >>>>> <thejaka.amila@gmail.com>wrote:
> >>>>>
> >>>>>> On Fri, Jan 17, 2014 at 10:32 AM, Saminda Wijeratne <
> >> samindaw@gmail.com
> >>>>> wrote:
> >>>>>>> Following are few thoughts I had during my review of the
component,
> >>>>>>>
> >>>>>>> *Multi-threaded vs single threaded*
> >>>>>>> If we are going to have multi-threaded job submission the
> >>>> implementation
> >>>>>>> should work on handling race conditions. Essentially JobSubmitter
> >>>> should be
> >>>>>>> able to "lock" an experiment request before continuing processing
> >> that
> >>>>>>> request so that other JobSubmitters accessing the experiment
> requests
> >>>> a the
> >>>>>>> same time would skip it.
> >>>>>>>
> >>>>>> +1. These are implementation details.
> >>>>>>
> >>>>>>
> >>>>>>> *Orchestrator service*
> >>>>>>> We might want to think of the possibility in future where
we will
> be
> >>>>>>> having multiple deployments of an Airavata service. This
could
> >>>> particularly
> >>>>>>> be true for SciGaP. We may have to think how some of the
internal
> >> data
> >>>>>>> structures/SPIs should be updated to accomodate such requirements
> in
> >>>> future.
> >>>>>> +1.
> >>>>>>
> >>>>>>
> >>>>>>> *Orchestrator Component configurations*
> >>>>>>> I see alot of places where the orchestrator can have
> configurations.
> >> I
> >>>>>>> think its too early finalize them, but I think we can start
> >> refactoring
> >>>>>>> them out perhaps to the airavata-server.properties. I'm
also seeing
> >> the
> >>>>>>> orchestrator is now hardcoded to use default/admin gateway
and
> >>>> username. I
> >>>>>>> think it should come from the request itself.
> >>>>>>>
> >>>>>> +1. But in overall we may need to change the way we handle
> >>>> configurations
> >>>>>> within Airavata. Currently we have multiple configuration files
and
> >>>>>> multiple places where we read configurations. IMO we should
have a
> >>>> separate
> >>>>>> module to handle configurations. Only this module should be
aware
> how
> >> to
> >>>>>> intepret configurations in the file and provide a component
> interface
> >> to
> >>>>>> access those configuration values.
> >>>>>>
> >>>>> +1 we tried this once with "ServerSettings" and
> "ApplicationSettings",
> >>>> but
> >>>>> apparently again more configuration files seems to have spawned.
So
> far
> >>>>> however they seemed to be localized for their component now.
> >>>>>
> >>>>>>> *Visibility of API functions*
> >>>>>>> I think initialize(), shutdown() and startJobSubmitter()
functions
> >>>> should
> >>>>>>> not be part of the API because I don't see a scenario where
the
> >> gateway
> >>>>>>> developer would be responsible for using them. They serve
a more
> >>>> internal
> >>>>>>> purpose of managing the orchestrator component IMO. As Amila
> pointed
> >>>> out so
> >>>>>>> long ago (wink) functions that do not concern outside parties
> should
> >>>> not be
> >>>>>>> used as part of the API.
> >>>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>>
> >>>>>>> *Return values of Orchestrator API*
> >>>>>>> IMO unless it is specifically required to do so I think
the
> functions
> >>>>>>> does not necessarily need to return anything other than
throw
> >>>> exceptions
> >>>>>>> when needed. For example the launchExperiment can simply
return
> void
> >>>> if all
> >>>>>>> is succesful and return an exception if something fails.
Handling
> >>>> issues
> >>>>>>> with a try catch is not only simpler but also the explanations
are
> >>>> readily
> >>>>>>> available for the user.
> >>>>>>>
> >>>>>> +1. Also try to have different exception for different scenarios.
> For
> >>>>>> example if persistence (hypothetical) fails,
> >>>> DatabasePersistenceException,
> >>>>>> if validation fails, ValidationFailedException etc ... Then
the
> >>>> developer
> >>>>>> who uses the API can catch these different exceptions and act
on
> them
> >>>>>> appropriately.
> >>>>>>
> >>>>> +1. What needs to be understood here is that the Exception should
be
> a
> >>>>> Gateway friendly exception. i.e. it should not expose internal
> details
> >> of
> >>>>> Airavata at the top-level exception and exception message should
be
> >> self
> >>>>> explanatory enough for the gateway developer not to remain scratching
> >>>>> his/her head after reading the exception. A feedback from Sudhakar
> >>>> sometime
> >>>>> back was to provide suggestions in the exception message on how
to
> >>>> resolve
> >>>>> the issue.
> >>>>>
> >>>>>>> *Data persisted in registry*
> >>>>>>> ExperimentRequest.getUsername() : I think we should clarify
what
> this
> >>>>>>> username denotes. In current API, in experiment submission
we
> >> consider
> >>>> two
> >>>>>>> types of users. Submission user (the user who submits the
> experiment
> >>>> to the
> >>>>>>> Airavata Server - this is inferred by the request itself)
and the
> >>>> execution
> >>>>>>> user (the user who corelates to the application executions
of the
> >>>> gateway -
> >>>>>>> thus this user can be a different user for different gateway,
eg:
> >>>> community
> >>>>>>> user, gateway user).
> >>>>>>> I think we should persist the date/time of the experiment
request
> as
> >>>>>>> well.
> >>>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>>>  Also when retrying of API functions in the case of a failure
in an
> >>>>>>> previous attempt there should be a way to not to repeat
already
> >>>> performed
> >>>>>>> steps or gracefully roleback and redo those required steps
as
> >>>> necessary.
> >>>>>>> While such actions could be transparent to the user sometimes
it
> >> might
> >>>> make
> >>>>>>> sense to allow user to be notified of success/failure of
a retry.
> >>>> However
> >>>>>>> this might mean keeping additional records at the registry
level.
> >>>>>>>
> >>>>>> In addition we should also have a way of cleaning up unsubmitted
> >>>>>> experiment ids. (But not sure whether you want to address this
right
> >>>> now).
> >>>>>> The way I see this is to have a periodic thread which goes through
> the
> >>>>>> table and clear up experiments which are not submitted for a
defined
> >>>> time.
> >>>>> +1. Something else we may have to think of later is the data
> archiving
> >>>>> capabilities. We keep running in to performance issues when the
> >> database
> >>>>> grows with experiment results. Unless we become experts of
> distributed
> >>>>> database management we should have a way better way to manage our
db
> >>>>> performance issues.
> >>>>>
> >>>>>
> >>>>>> BTW, nice review notes, Saminda.
> >>>>>>
> >>>>>> Thanks
> >>>>>> Amila
> >>>>>>
> >>>>>>
> >>>>>>
> >>
> >
>
>


-- 
Thanks,
Sachith Withana

Mime
View raw message