Mailing-List: contact dev-help@airavata.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airavata.apache.org
Received-SPF: pass (athena.apache.org: domain of
 danushka.menikkumbura@gmail.com designates 209.85.128.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANotcz5WbdE=beTxGRPqFQ6mirD5fb8OPrgmJfvR5tyZ--oj4g@mail.gmail.com>
References: 
 <CANotcz5WbdE=beTxGRPqFQ6mirD5fb8OPrgmJfvR5tyZ--oj4g@mail.gmail.com>
From: Danushka Menikkumbura <danushka.menikkumbura@gmail.com>
Date: Tue, 21 May 2013 20:54:35 +0530
Message-ID: 
 <CAF4V=3GhccTXdgJo9RdYtZYus5U0i9gu8CiH5iEnwJeNz4bjmA@mail.gmail.com>
Subject: Re: Persisting GFac job data
To: dev@airavata.apache.org
Content-Type: multipart/alternative; boundary=047d7bea317627e47d04dd3c0d19

--047d7bea317627e47d04dd3c0d19
Content-Type: text/plain; charset=ISO-8859-1

Saminda,

I think the data container does not need to have a generic format. We can
have a base class that facilitate object serialization/deserialization and
let specific meta data structure implement them as required. We get the
Registry API to serialize objects and save them in a meta data table (with
just two columns?) and to deserialize as they are loaded off the registry.

Danushka


On Tue, May 21, 2013 at 8:34 PM, Saminda Wijeratne <samindaw@gmail.com>wrote:

> It has being apparent more and more that saving the data related to
> executing a jobs from the GFac can be useful for many reasons such as,
>
> debugging
> retrying
> to make smart decisions on reliability/cost etc.
> statistical analysis
>
> Thus we thought of saving the data related to GFac jobs in the registry in
> order to facilitate feature such as above in the future.
>
> However a GFac job is potentially any sort of computing resource access
> (GRAM/UNICORE/EC2 etc.). Therefore we need to come up with a generalized
> data structure that can hold the data of any type of resource. Following
> are the suggested data to save for a single GFac job execution,
>
> *experiment id, workflow instance id, node id* - pinpoint the node
> execution
> *service, host, application description ids *- pinpoint the descriptors
> responsible
> *local job id* - the unique job id retrieved/generated per execution
> [PRIMARY KEY]
> *job data* - data related executing the job (eg: the rsl in GRAM)
> *submitted, completed time*
> *completed status* - whether the job was successfull or ran in to errors
> etc.
> *metadata* - custom field to add anything user wants
>
> Your feedback is most welcome. The API related changes will also be
> discussed once we have a proper data structure. We are hoping to implement
> this within next few days.
>
> Thanks,
> Saminda
>

--047d7bea317627e47d04dd3c0d19--