predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <>
Subject Re: Remove engine registration
Date Sat, 17 Sep 2016 21:06:47 GMT
Yes, a new thing though it might serve some of the same purposes. The idea is to only use engine
instance information from the metadata store so the template commands will work from anywhere
and mostly in any order. On a cluster machine if the engine instance data is in the metastore
and the binary exists wherever it was on the machine registered, then `pio deploy <new-kind-of-instance-id>`
would work without any other part of the workflow. Also `pio train <new-kind-of-instance-id>`
would work from any cluster machine with no need of special folder layout or manifest.json

Sorry to overload the term but though this new type of engine instance would have much of
the same info, it would also have to contain the path to the binary and maybe other things.

On Sep 16, 2016, at 7:51 PM, Kenneth Chan <> wrote:

Pat, would you explain more about the 'instanceId' as in
`pio register --variant path/to/some-engine.json --instanceId some-REST-compatible-resource-id`

Currently PIO also has a concept of engineInstanceId, which is output of train. I think you
are referring to different thing, right?


On Fri, Sep 16, 2016 at 12:58 PM, Pat Ferrel < <>>
This is a great discussion topic and a great idea.

However the cons must also be addressed, we will need to do this before multi-tenant deploys
can happen and the benefits are just as large as removing `pio build`

It would be great to get rid of manifest.json and put all metadata in the store with an externally
visible id so all parts of the workflow on all machines will get the right metadata and any
template specific commands will run from anywhere on any cluster machine and in any order.
All we need is a global engine-instance id. This will make engine-instances behave more like
datasets, which are given permanent ids with `pio app new …` This might be a new form of
`pio register` and it implies a new optional param to pio template specific commands (the
instance id) but removes a lot of misunderstandings people have and easy mistakes in workflow.

So workflow would be:
1) build with SBT/mvn
2) register any time engine.json changes so make the json file an optional param to `pio register
--variant path/to/some-engine.json --instanceId some-REST-compatible-resource-id` the instance
could also be auto-generated and output or optionally in the engine.json. `pio engine list`
lists registered instances with instanceId. The path to the binary would be put in the instanceId
and would be expected to be the same on all cluster machines that need it.
3) `pio train --instanceId` optional if it’s in engine.json
4) `pio deploy --instanceId` optional if it’s in engine.json
5) with easily recognized exceptions all the above can happen in any order on any cluster
machine and from any directory.

This takes one big step to multi-tenancy since the instance data has an externally visible
id—call it a REST resource id…

I bring this up not to confuse the issue but because if we change the workflow commands we
should avoid doing it often because of the disruption it brings.

On Sep 16, 2016, at 10:42 AM, Donald Szeto < <>>

Hi all,

I want to start the discussion of removing engine registration. How many people actually take
advantage of being able to run pio commands everywhere outside of an engine template directory?
This will be a nontrivial change on the operational side so I want to gauge the potential
impact to existing users.

- Stateless build. This would work well with many PaaS.
- Eliminate the "pio build" command once and for all.
- Ability to use your own build system, i.e. Maven, Ant, Gradle, etc.
- Potentially better experience with IDE since engine templates no longer depends on an SBT

- Inability to run pio engine training and deployment commands outside of engine template
- No automatic version matching of PIO binary distribution and artifacts version used in the
engine template.
- A less unified user experience: from pio-build-train-deploy to build, then pio-train-deploy.


View raw message