Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id DC176200B83 for ; Sat, 17 Sep 2016 23:07:05 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id CF867160ACD; Sat, 17 Sep 2016 21:07:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B8574160AB5 for ; Sat, 17 Sep 2016 23:07:04 +0200 (CEST) Received: (qmail 22871 invoked by uid 500); 17 Sep 2016 21:07:03 -0000 Mailing-List: contact user-help@predictionio.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@predictionio.incubator.apache.org Delivered-To: mailing list user@predictionio.incubator.apache.org Received: (qmail 22861 invoked by uid 99); 17 Sep 2016 21:07:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Sep 2016 21:07:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 74B4CC1A5D for ; Sat, 17 Sep 2016 21:07:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.299 X-Spam-Level: * X-Spam-Status: No, score=1.299 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=occamsmachete-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id raqeYFJerE9N for ; Sat, 17 Sep 2016 21:07:01 +0000 (UTC) Received: from mail-pf0-f181.google.com (mail-pf0-f181.google.com [209.85.192.181]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id E0A425F570 for ; Sat, 17 Sep 2016 21:07:00 +0000 (UTC) Received: by mail-pf0-f181.google.com with SMTP id p64so37913237pfb.1 for ; Sat, 17 Sep 2016 14:07:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=occamsmachete-com.20150623.gappssmtp.com; s=20150623; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=kntM9v1+2vzCIguO4hieU0NcUctNGrBmzX3WSBq2ZhE=; b=xt8DbpE3YfB/nKcrEzBqDvDsciyekxcWBUdiV0P6MZiPBZvha6jfZpICFR5eWeFsOe /mzoj5uOegcgq3SN6u6qEQaZji9v/11QHGV9j7J7/NzAWkN4/Uj+mCefOdzfu8KWeJwT lr85pBA+b4iOF2clbRvNuyl1HXijm/7j0XETTwKFiIjzxSDmHZWmguqm/CKWkHDPYWqI zgw6Rch8vk+4na8mt7mm+MFJv+MAKqNzvnOS8mrn4Sro2m/VfeSjbhfpKU8rsQli68Aw eQuhGYViqCTuVnatSjoR6WcPZQGb6YJYy01b7YvjeMChv14Zb5y3GxNOMBvDystPRE83 drpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=kntM9v1+2vzCIguO4hieU0NcUctNGrBmzX3WSBq2ZhE=; b=lw/CUGsk5k7xX4ITDNsIpJQ7+0JHoiBeV5NR4RHpeiF0jGzZWtLs62vXVF3cEr1o/9 HjgPjKukwzDUm4Tf5JLYPLzV5F04uux/sVXERgGAGr3Z1xOpV6z9ZJvFk64sMhyDUzmb jO80VIJVnhWwFEaVpme63ixczINb+Yjh281QYGD7QiGDzv3By9g7IdXsxO7G6j1Euoia hd5KkN/y2/ZzpstT3ZL/RO1Hl4cN1Q5h60RLocFfkceZAwCmwuvP+GW9JVS7BZ6eKgik cKgqft1fagHAH1ZNjukdm36g4C/ErdMS42/7i6KShNk1YJ8cRf+jAgmYFaYYb0W1hLpX +nzg== X-Gm-Message-State: AE9vXwMM+pP64gNChSMi96mxTwtMfUpEQABfGkrrNbcaMjQIkpJa/lrVvIvcUL8sB9zWPw== X-Received: by 10.98.74.77 with SMTP id x74mr34578802pfa.1.1474146414032; Sat, 17 Sep 2016 14:06:54 -0700 (PDT) Received: from [192.168.223.2] (ec2-52-59-250-132.eu-central-1.compute.amazonaws.com. [52.59.250.132]) by smtp.gmail.com with ESMTPSA id q8sm20938971pac.32.2016.09.17.14.06.50 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 17 Sep 2016 14:06:52 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_3EB2166C-9398-4474-83C3-CF0A1E05487B" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Remove engine registration From: Pat Ferrel In-Reply-To: Date: Sat, 17 Sep 2016 14:06:47 -0700 Cc: dev@predictionio.incubator.apache.org Message-Id: <9AF18596-169F-4D23-B2C3-CA72940E6A42@occamsmachete.com> References: To: user@predictionio.incubator.apache.org X-Mailer: Apple Mail (2.3124) archived-at: Sat, 17 Sep 2016 21:07:06 -0000 --Apple-Mail=_3EB2166C-9398-4474-83C3-CF0A1E05487B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Yes, a new thing though it might serve some of the same purposes. The = idea is to only use engine instance information from the metadata store = so the template commands will work from anywhere and mostly in any = order. On a cluster machine if the engine instance data is in the = metastore and the binary exists wherever it was on the machine = registered, then `pio deploy ` would work = without any other part of the workflow. Also `pio train = ` would work from any cluster machine with no = need of special folder layout or manifest.json Sorry to overload the term but though this new type of engine instance = would have much of the same info, it would also have to contain the path = to the binary and maybe other things. On Sep 16, 2016, at 7:51 PM, Kenneth Chan wrote: Pat, would you explain more about the 'instanceId' as in `pio register --variant path/to/some-engine.json --instanceId = some-REST-compatible-resource-id` ? Currently PIO also has a concept of engineInstanceId, which is output of = train. I think you are referring to different thing, right? Kenneth On Fri, Sep 16, 2016 at 12:58 PM, Pat Ferrel > wrote: This is a great discussion topic and a great idea. However the cons must also be addressed, we will need to do this before = multi-tenant deploys can happen and the benefits are just as large as = removing `pio build` It would be great to get rid of manifest.json and put all metadata in = the store with an externally visible id so all parts of the workflow on = all machines will get the right metadata and any template specific = commands will run from anywhere on any cluster machine and in any order. = All we need is a global engine-instance id. This will make = engine-instances behave more like datasets, which are given permanent = ids with `pio app new =E2=80=A6` This might be a new form of `pio = register` and it implies a new optional param to pio template specific = commands (the instance id) but removes a lot of misunderstandings people = have and easy mistakes in workflow. So workflow would be: 1) build with SBT/mvn 2) register any time engine.json changes so make the json file an = optional param to `pio register --variant path/to/some-engine.json = --instanceId some-REST-compatible-resource-id` the instance could also = be auto-generated and output or optionally in the engine.json. `pio = engine list` lists registered instances with instanceId. The path to the = binary would be put in the instanceId and would be expected to be the = same on all cluster machines that need it. 3) `pio train --instanceId` optional if it=E2=80=99s in engine.json 4) `pio deploy --instanceId` optional if it=E2=80=99s in engine.json 5) with easily recognized exceptions all the above can happen in any = order on any cluster machine and from any directory. This takes one big step to multi-tenancy since the instance data has an = externally visible id=E2=80=94call it a REST resource id=E2=80=A6 I bring this up not to confuse the issue but because if we change the = workflow commands we should avoid doing it often because of the = disruption it brings. On Sep 16, 2016, at 10:42 AM, Donald Szeto > wrote: Hi all, I want to start the discussion of removing engine registration. How many = people actually take advantage of being able to run pio commands = everywhere outside of an engine template directory? This will be a = nontrivial change on the operational side so I want to gauge the = potential impact to existing users. Pros: - Stateless build. This would work well with many PaaS. - Eliminate the "pio build" command once and for all. - Ability to use your own build system, i.e. Maven, Ant, Gradle, etc. - Potentially better experience with IDE since engine templates no = longer depends on an SBT plugin. Cons: - Inability to run pio engine training and deployment commands outside = of engine template directory. - No automatic version matching of PIO binary distribution and artifacts = version used in the engine template. - A less unified user experience: from pio-build-train-deploy to build, = then pio-train-deploy. Regards, Donald --Apple-Mail=_3EB2166C-9398-4474-83C3-CF0A1E05487B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Yes, a new thing though it might serve some of the same = purposes. The idea is to only use engine instance information from the = metadata store so the template commands will work from anywhere and = mostly in any order. On a cluster machine if the engine instance data is = in the metastore and the binary exists wherever it was on the machine = registered, then `pio deploy <new-kind-of-instance-id>` would work = without any other part of the workflow. Also `pio = train <new-kind-of-instance-id>` would work from any cluster = machine with no need of special folder layout or manifest.json

Sorry to overload the = term but though this new type of engine instance would have much of the = same info, it would also have to contain the path to the binary and = maybe other things.


On = Sep 16, 2016, at 7:51 PM, Kenneth Chan <kenneth@apache.org> = wrote:

Pat, would you explain more about the = 'instanceId' as in
`pio register --variant path/to/some-engine.json --instanceId = some-REST-compatible-resource-id` =  ?

Currently PIO = also has a concept of engineInstanceId, which is output of train. I = think you are referring to different thing, right?

Kenneth


On Fri, Sep 16, 2016 at 12:58 PM, = Pat Ferrel <pat@occamsmachete.com> wrote:
This is a great = discussion topic and a great idea.

However the cons must also be addressed, we will need to do this before = multi-tenant deploys can happen and the benefits are just as large as = removing `pio build`

It would be great to get rid of manifest.json and put all metadata in = the store with an externally visible id so all parts of the workflow on = all machines will get the right metadata and any template specific = commands will run from anywhere on any cluster machine and in any order. = All we need is a global engine-instance id. This will make = engine-instances behave more like datasets, which are given permanent = ids with `pio app new =E2=80=A6` This might be a new form of `pio = register` and it implies a new optional param to pio template specific = commands (the instance id) but removes a lot of misunderstandings people = have and easy mistakes in workflow.

So workflow would be:
1) build with SBT/mvn
2) register any time engine.json changes so make the json file an = optional param to `pio register --variant path/to/some-engine.json = --instanceId some-REST-compatible-resource-id` the = instance could also be auto-generated and output or optionally in the = engine.json. `pio engine list` lists registered instances with = instanceId. The path to the binary would be put in the instanceId and = would be expected to be the same on all cluster machines that need = it.
3) `pio train --instanceId` optional if it=E2=80=99s in engine.json
4) `pio deploy --instanceId` optional if it=E2=80=99s in engine.json
5) with easily recognized exceptions all the above can happen in any = order on any cluster machine and from any directory.

This takes one big step to multi-tenancy since the instance data has an = externally visible id=E2=80=94call it a REST resource id=E2=80=A6

I bring this up not to confuse the issue but because if we change the = workflow commands we should avoid doing it often because of the = disruption it brings.


On Sep 16, 2016, at 10:42 AM, Donald Szeto <donald@apache.org> = wrote:

Hi all,

I want to start the discussion of removing engine registration. How many = people actually take advantage of being able to run pio commands = everywhere outside of an engine template directory? This will be a = nontrivial change on the operational side so I want to gauge the = potential impact to existing users.

Pros:
- Stateless build. This would work well with many PaaS.
- Eliminate the "pio build" command once and for all.
- Ability to use your own build system, i.e. Maven, Ant, Gradle, etc.
- Potentially better experience with IDE since engine templates no = longer depends on an SBT plugin.

Cons:
- Inability to run pio engine training and deployment commands outside = of engine template directory.
- No automatic version matching of PIO binary distribution and artifacts = version used in the engine template.
- A less unified user experience: from pio-build-train-deploy to build, = then pio-train-deploy.

Regards,
Donald



= --Apple-Mail=_3EB2166C-9398-4474-83C3-CF0A1E05487B--