airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saminda Wijeratne <samin...@gmail.com>
Subject Re: Persisting GFac job data
Date Tue, 21 May 2013 16:42:13 GMT
+1. Can we use the registry migration tool to copy the data from the
Gram_data table to GFac_job_data table? we can keep null to the new fields.
and then delete the gram_data table? I can deprecate the functions related
to gram table in the registry API and change the implementation of it to
point to gfac_job_data table.


On Tue, May 21, 2013 at 11:14 AM, Chathuri Wimalasena
<kamalasini@gmail.com>wrote:

> If we are going to get rid of the Gram_Data table later, we should find
> possible ways to migrate data from that table to new table (GFac_Job_Data).
> Since Gram_Data table does not have all the details that are specified in
> the new table, there is no way we can retrieve  submitted and completed
> time related information.
>
> Regards,
> Chathuri
>
>
> On Tue, May 21, 2013 at 11:04 AM, Saminda Wijeratne <samindaw@gmail.com
> >wrote:
>
> > It has being apparent more and more that saving the data related to
> > executing a jobs from the GFac can be useful for many reasons such as,
> >
> > debugging
> > retrying
> > to make smart decisions on reliability/cost etc.
> > statistical analysis
> >
> > Thus we thought of saving the data related to GFac jobs in the registry
> in
> > order to facilitate feature such as above in the future.
> >
> > However a GFac job is potentially any sort of computing resource access
> > (GRAM/UNICORE/EC2 etc.). Therefore we need to come up with a generalized
> > data structure that can hold the data of any type of resource. Following
> > are the suggested data to save for a single GFac job execution,
> >
> > *experiment id, workflow instance id, node id* - pinpoint the node
> > execution
> > *service, host, application description ids *- pinpoint the descriptors
> > responsible
> > *local job id* - the unique job id retrieved/generated per execution
> > [PRIMARY KEY]
> > *job data* - data related executing the job (eg: the rsl in GRAM)
> > *submitted, completed time*
> > *completed status* - whether the job was successfull or ran in to errors
> > etc.
> > *metadata* - custom field to add anything user wants
> >
> > Your feedback is most welcome. The API related changes will also be
> > discussed once we have a proper data structure. We are hoping to
> implement
> > this within next few days.
> >
> > Thanks,
> > Saminda
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message