Return-Path: X-Original-To: apmail-airavata-dev-archive@www.apache.org Delivered-To: apmail-airavata-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0D1D110A5E for ; Tue, 27 May 2014 09:30:25 +0000 (UTC) Received: (qmail 90924 invoked by uid 500); 27 May 2014 09:30:24 -0000 Delivered-To: apmail-airavata-dev-archive@airavata.apache.org Received: (qmail 90875 invoked by uid 500); 27 May 2014 09:30:24 -0000 Mailing-List: contact dev-help@airavata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airavata.apache.org Delivered-To: mailing list dev@airavata.apache.org Received: (qmail 90868 invoked by uid 99); 27 May 2014 09:30:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 May 2014 09:30:24 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of swsachith@gmail.com designates 209.85.212.182 as permitted sender) Received: from [209.85.212.182] (HELO mail-wi0-f182.google.com) (209.85.212.182) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 May 2014 09:30:19 +0000 Received: by mail-wi0-f182.google.com with SMTP id r20so1293282wiv.9 for ; Tue, 27 May 2014 02:29:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=5y2LHe03fax2Kdw7aHstplyFU2ppdfve6AfxYb1mXnI=; b=mz2kGZugbDCwwe5OXb+/B1rVrWk/3SQrO0KGfKxrNT7J+ynyxjD/IwefcpTpClC0Mo 3Vopj1g3L7zkJLXenxYzq31S8fmSg1ZaKrhJCJu17Oq197PWvqFXt1gpuhkwwD+GMqsK Cc8Rh0w1/s9Qp9avRMq6JPlbAo2aqLuZjJ+aIy+YTMUlfm6AnntwlI4Wo0Bh0ZWMM9f6 YJNFlJXNcfzHH+IDAFov7TrTfGNfxQIFsWgeZ5Fy6fzu8UTxG4v6F1wALOuZWPLtLML2 xpWRZ4ufvga7sIySwttG31ECBfn628lpaSDuMGloRHVDDa18bRFBcdIVQbelw2N2PRMC joTA== MIME-Version: 1.0 X-Received: by 10.194.82.9 with SMTP id e9mr38069669wjy.45.1401182998562; Tue, 27 May 2014 02:29:58 -0700 (PDT) Received: by 10.217.94.201 with HTTP; Tue, 27 May 2014 02:29:58 -0700 (PDT) In-Reply-To: References: Date: Tue, 27 May 2014 14:59:58 +0530 Message-ID: Subject: Re: Application Catalog CPI design From: Sachith Withana To: dev@airavata.apache.org Content-Type: multipart/alternative; boundary=047d7bb04260e517dc04fa5e5647 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bb04260e517dc04fa5e5647 Content-Type: text/plain; charset=UTF-8 So from what we discussed both offline and in this thread, these are the changes I propose to the current design, 1. Application Object will be passed as a whole to the GFac so that it can deal with it. ( It does not include deployment specific information) 2. Those Application Objects would be cached ( minimize the data retrieval overhead) 3. When the GFac wants to get the deployments for a particular application, the CPI would provide a method that looks like getDeployments( String appcalitionID, HashMap filters), the hashmap would contain the filters that Gfac wants to filter application deployments with, ex: get deployments for the application that involves trestles or stampede This would prevent the Gfac having to go through the deployments and filtering them out. WDYT? On Tue, May 27, 2014 at 8:58 AM, Saminda Wijeratne wrote: > > > > On Sun, May 25, 2014 at 4:18 AM, Sachith Withana wrote: > >> Thanks Saminda for the reply. >> Please refer to the inline comments >> >> >> On Thu, May 22, 2014 at 10:25 AM, Saminda Wijeratne wrote: >> >>> Looking what generally GFac requires I'd say something along the lines >>> of following would be useful. >>> >>> - Functions required to retrieve all the access data of a given >>> application id (eg: inputs/outputs/security data/validation data/host >>> properties/application properties/job properties etc.) >>> >>> We can just return the whole Application Object which contains all these >> details right? Since Gfac would be getting all these data at the same time, >> it would be a better option to minimize the calls between the Gfac and the >> App catalog and just return the object? >> > It would make sense to do so if GFac wants everything related to that > application. But IMO at launch GFac will not need any user metadata, > description fields, validation data (if validation done at orchestrator > already), alias info, irrelevant deployment data etc. > >> >> >>> >>> - Functions to help gfac scheduling (eg: get all SSH supported >>> deployments, get all trestles deployments) >>> >>> Thanks for pointing this out Saminda. But I failed to understand the use >> case here? >> > eg: GFac will have its own criteria to filter-out and select the > deployment its going to use. However IMO we can provide basic filtration > through App Catalog CPI to only send through a filtered set of deployments > so that GFac doesn't have to deal with all the deployments of an > application. Saves db query time, network turnaround time and memory usage. > >> >> >>> >>> - Functions validating permissions to use the resources (eg: >>> disabled resources, inactive job managers etc) >>> >>> Do you mind elaborating the line above? Disabled resources on what >> level? from the gateway level or the Airavata level? >> > This can be one of the functions provided by App Catalog CPI. But gateways > might be interested in managing these information themselves and during > experiment creation would imply such permission information. Disabled > resources can be drilled down to application, deployment, host. Like I > mentioned above it could be at the gateway side if gateway wants to manage > it. In which case AppCatalog do not have to worry about it. > >> >> >>> >>> >>> On Wed, May 21, 2014 at 1:15 AM, Sachith Withana wrote: >>> >>>> Hi folks, >>>> >>>> Here's[1] the initial Application Catalog Design with the basic >>>> functions regarding the application, deployment, host and gatewayProfile >>>> Objects. >>>> >>>> In terms of the users of the App Catalog like the GFac, what kind of >>>> fine-grained CPI methods are required ? >>>> >>>> ex: should we pass the whole Application Object and expect the GFac to >>>> deal with it or allow more fine-grained methods to get the inputs, hosts >>>> ...etc? >>>> >>>> Please note that I haven't included the search functionality in this >>>> phase of the CPI design. >>>> >>>> >>>> [1] >>>> https://docs.google.com/document/d/1FfAT0kRFUJR78o9Pj58sNcUJJzPTojmD3zjldiWfIik/edit >>>> >>>> -- >>>> Thanks, >>>> Sachith Withana >>>> >>>> >>> >> >> >> -- >> Thanks, >> Sachith Withana >> >> > -- Thanks, Sachith Withana --047d7bb04260e517dc04fa5e5647 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
So from what we discussed both offline and in this thread,= these are the changes I propose to the current design,

=
  1. Application Object will be passed as a whole to the GFac so that it= can deal with it. ( It does not include deployment specific information)
  2. Those Application Objects would be cached ( minimize the data retrieval= overhead)
  3. When the GFac wants to get the deployments for a particu= lar application, the CPI would provide a method that looks like
    getDeplo= yments( String appcalitionID, HashMap filters), the hashmap would contain t= he filters that Gfac wants to filter application deployments with,
    ex: get deployments for the application that involves trestles or stampede<= br>
    This would prevent the Gfac having to go through the deployments and= filtering them out.=C2=A0

WDYT?



On Tue, May 27, 2014 at 8:58 AM, Saminda Wijeratne <= ;samindaw@gmail.com= > wrote:



On Sun, May 25, 2014= at 4:18 AM, Sachith Withana <swsachith@gmail.com> wrote:<= br>
Thanks Saminda for the= reply.=C2=A0
Please refer to the inline comments


On Thu, May 22, 2014 at 10:25 AM, S= aminda Wijeratne <samindaw@gmail.com> wrote:
Looking what= generally GFac requires I'd say something along the lines of following= would be useful.
  • Functions required to retrieve all the access data of a given= application id (eg: inputs/outputs/security data/validation data/host prop= erties/application properties/job properties etc.)
We can just return the whole Application Obje= ct which contains all these details right? Since Gfac would be getting all = these data at the same time, it would be a better option to minimize the ca= lls between the Gfac and the App catalog and just return the object?
It would make sense to do so if G= Fac wants everything related to that application. But IMO at launch GFac wi= ll not need any user metadata, description fields, validation data (if vali= dation done at orchestrator already), alias info, irrelevant deployment dat= a etc.
=C2=A0
=
  • Functions to help gfac scheduling (eg: get all SSH supported deployments, g= et all trestles deployments)
=
Thanks for pointing this out Saminda. But I failed to understand the u= se case here?
eg: GFac will have its own criter= ia to filter-out and select the deployment its going to use.=C2=A0 However = IMO we can provide basic filtration through App Catalog CPI to only send th= rough a filtered set of deployments so that GFac doesn't have to deal w= ith all the deployments of an application. Saves db query time, network tur= naround time and memory usage.
=C2=A0
=
  • Functions validating permissions to use the resources (eg: disabled= resources, inactive job managers etc)
Do you mind elaborating the = line above? Disabled resources on what level? from the gateway level or the= Airavata level?
This can be= one of the functions provided by App Catalog CPI. But gateways might be in= terested in managing these information themselves and during experiment cre= ation would imply such permission information. Disabled resources can be dr= illed down to application, deployment, host. Like I mentioned above it coul= d be at the gateway side if gateway wants to manage it. In which case AppCa= talog do not have to worry about it.




On Wed, May 21, 2014 at 1:15 AM, Sachith Withana <swsa= chith@gmail.com> wrote:
Hi = folks,

Here's[1] the initial Application Catal= og Design with the basic functions regarding the application, deployment, h= ost and gatewayProfile Objects.

In terms of the users of the App Catalog like the GFac, what kind of f= ine-grained CPI methods are required ?=C2=A0

ex: s= hould we pass the whole Application Object and expect the GFac to deal with= it or allow more fine-grained methods to get the inputs, hosts ...etc?=C2= =A0

Please note that I haven't included the search func= tionality in this phase of the CPI design.





--
Thanks,
Sachith Withana





--
Thanks,
=
Sachith Withana

--047d7bb04260e517dc04fa5e5647--