airavata-architecture mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miller, Mark" <>
Subject RE: Experiment Summary retrieval
Date Fri, 14 Mar 2014 18:42:46 GMT
I think that would be useful, personally. 

-----Original Message-----
From: Borries Demeler [] 
Sent: Friday, March 14, 2014 11:37 AM
Subject: Re: Experiment Summary retrieval

Mark, Lahiru and others:

If you guys aren't averse to yet another online conference, I would be happy to give an on-screen
demo of our system. It is a bit difficult to explain per email how this rather complicated
system is organized, but it may be worthwhile to get some insights of one possible solution
to how this can be handled. I wouldn't say that the gateway is completely separated from the
task of processing output data, but the task is abstracted to a very basic and uncomplicated
level, and then the more complex issues are handled in a custom fashion by our software. This
way we can on the one hand have robust communications and on the other fine-grained and customized
data handling for the specific needs of the user. I would probably need about 20-30 mins of
your time. Perhaps during the next SciGap meeting?
It would be best if everyone had a vnc viewer on their desktop to follow along.


On Fri, Mar 14, 2014 at 04:49:15PM +0000, Miller, Mark wrote:
> Very interesting Borries. So if I read that right, delivery of results for your Gateway
takes place by a completely separate system.
> I think the RESTful access we provide will be similar in some ways.  We will be passing
results directly to the user's application, which will be in effect like the UltraScan package,
I think. In that case, what the user extracts from the data will be up to their application.
Of course we will track the relevant usage parameters just as we do currently on our side.
> The access to intermediate results for ongoing jobs is also critically important for
us as well, and very much outside of the main Gateway software. But we don't currently support
access to status of a list of running jobs. Although the idea of being able to provide users
with a window into the status of all their tasks is certainly a nice feature, and we would
likely consume it and present it to users if it was easy to access.
> Mark
> -----Original Message-----
> From: Borries Demeler []
> Sent: Friday, March 14, 2014 9:12 AM
> To:
> Subject: Re: Experiment Summary retrieval
> I am not sure if this relates at all to the getAllExperiments() question, but I thought
it may help to see how we deal with retrieving expt. data from the gateway in UltraScan.
> In the case of UltraScan we have the HPC output very much separated from the information
of the results that are useful to the user. All data are getting tar.gzipped into a single
archive which is transported back out and automatically parsed into a relational database
that is part of the UltraScan software, and not a function of the gateway. The user will then
access the database with secondary software and create a type of meta-analysis and visualization
of the results that ultimately are then useful to the user. Having said this, while data are
being calculated on the HPC resource, the user can monitor the process of the calculation
by reviewing a queue viewer that shows all pending and active jobs and their actual state.
This state include a lot of information keeping the user apprised of the state. This information
is continually being sent via UDP from the HPC resource and monitored by a daemon on our backend
LIMS server, again, this is semi-independent from the gateway software. This may be very specialized
for our use case, but it is one example of how the problem can be solved.
> -borries
> On Fri, Mar 14, 2014 at 11:37:02AM -0400, Saminda Wijeratne wrote:
> > Thanks for identifying the problem Sachith. To be precise 
> > potentially there are many subset variations of interested fields in 
> > the Experiment Data which different gateways will be interested in 
> > the same usecase or different usecase. We cannot scale by providing 
> > different functions for each of these scenarios.
> > 
> > 
> > On Fri, Mar 14, 2014 at 11:25 AM, Sachith Withana <>wrote:
> > 
> > > Hi all,
> > >
> > > Almost all gateways have a requirement of retrieving the 
> > > experiment summaries of all the experiments . The fields that are 
> > > required differ based on the gateway.
> > >
> > > For Example:
> > > CIPRES requires : Experiment name and status Some gateways 
> > > requires the Experiment name with the inputs only.
> > >
> > > But right now the getAllExperiments() method returns the list of 
> > > all the Experiments with all the experiment related attributes 
> > > filled ( the whole Experiment Model).
> > >
> > > It's costly to get the whole Experiment objects from the API 
> > > rather than getting the required few attributes.
> > >
> > > Any suggestions on how we could achieve this?
> > >
> > > One suggestion would be to have a getAllExperiments method with 
> > > the parameters as a list of required fields of the Experiments and 
> > > return only those fields.
> > >
> > > --
> > > Thanks,
> > > Sachith Withana
> > >

View raw message