predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <...@occamsmachete.com>
Subject Templates First
Date Fri, 20 Oct 2017 17:00:59 GMT
PredictionIO is completely useless without a Template yet we seem as a group too focused on
releasing PIO without regard for Templates. This IMO must change. 90% of users will never
touch the code of a template and only 1% will actually create a template. These guesses come
from list questions. If this is true we need to switch our mindset to "templates first”
not “pio first”. Before any upgrade vote, every committer should make sure their favorite
template works with the new build. I will be doing so from now on.

We have one fairly significant problem that I see from a template supporter's side. PIO has
several new build directives that change dependencies like Spark version and tools like Scala
version. These are unknown to templates and there is no PIO supported way to communicate these
to the template's build.sbt. This leaves us with templates that will not work with most combinations
of PIO builds. If we are lucky they may be updated to work with the *default* pio config.
But this did not happen when PIO-0.12.0 was released, only shortly afterwards. This must change,
the Apache templates at least must have some support for PIO before release and here is one
idea that might help...

How do we solve this?

1) .make-distribution modifies or creates a script that can be imported by the templates build.sbt.
This might be pio-env if we use `pio build` to build templates because it is available to
the template’s build.sbt, or something else when we move to using sbt to build templates
directly. This script defines values used to build PIO.
2) update some or all of the Apache templates to use this mechanism to build with the right
scala version, etc. taken from the PIO build.

I had a user do this for the UR to support many different pio build directives, and some that
are new. The result was a build.sbt that includes such things as 

    val pioVersion = sys.env.getOrElse("PIO_VERSION","0.12.0-incubating”)
    val scalaVersion = sys.env.getOrElse(“PIO_SCALA_VERSION”, “2.10.0”)
    val elasticsearch1Version = sys.env.getOrElse("PIO_ELASTIC_VERSION","1.7.5")
    val sparkVersion = sys.env.getOrElse("PIO_SPARK_VERSION","1.4.0”)

these are then used in the lib dependencies lists to pull in the right versions of artifacts.

This in some form would allow templates to move along in lock step with changes in the way
pio is built on any given machine. Without something like this, users even less expert at
sbt than myself (hard to imagine) will have a significant problem dumped on them.

Since this is only partially baked it may not be ready for a Jira and so warrants discussion.
 
Mime
View raw message