Mailing-List: contact user-help@predictionio.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@predictionio.apache.org
From: Pat Ferrel <pat@occamsmachete.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Templates First
Message-Id: <8B95832A-A8FA-4534-892D-A6523B23D1D3@occamsmachete.com>
Date: Fri, 20 Oct 2017 10:00:59 -0700
To: user@predictionio.incubator.apache.org,
 dev@predictionio.incubator.apache.org
archived-at: Fri, 20 Oct 2017 17:01:09 -0000

PredictionIO is completely useless without a Template yet we seem as a =
group too focused on releasing PIO without regard for Templates. This =
IMO must change. 90% of users will never touch the code of a template =
and only 1% will actually create a template. These guesses come from =
list questions. If this is true we need to switch our mindset to =
"templates first=E2=80=9D not =E2=80=9Cpio first=E2=80=9D. Before any =
upgrade vote, every committer should make sure their favorite template =
works with the new build. I will be doing so from now on.

We have one fairly significant problem that I see from a template =
supporter's side. PIO has several new build directives that change =
dependencies like Spark version and tools like Scala version. These are =
unknown to templates and there is no PIO supported way to communicate =
these to the template's build.sbt. This leaves us with templates that =
will not work with most combinations of PIO builds. If we are lucky they =
may be updated to work with the *default* pio config. But this did not =
happen when PIO-0.12.0 was released, only shortly afterwards. This must =
change, the Apache templates at least must have some support for PIO =
before release and here is one idea that might help...

How do we solve this?

1) .make-distribution modifies or creates a script that can be imported =
by the templates build.sbt. This might be pio-env if we use `pio build` =
to build templates because it is available to the template=E2=80=99s =
build.sbt, or something else when we move to using sbt to build =
templates directly. This script defines values used to build PIO.
2) update some or all of the Apache templates to use this mechanism to =
build with the right scala version, etc. taken from the PIO build.

I had a user do this for the UR to support many different pio build =
directives, and some that are new. The result was a build.sbt that =
includes such things as=20

    val pioVersion =3D =
sys.env.getOrElse("PIO_VERSION","0.12.0-incubating=E2=80=9D)
    val scalaVersion =3D sys.env.getOrElse(=E2=80=9CPIO_SCALA_VERSION=E2=80=
=9D, =E2=80=9C2.10.0=E2=80=9D)
    val elasticsearch1Version =3D =
sys.env.getOrElse("PIO_ELASTIC_VERSION","1.7.5")
    val sparkVersion =3D =
sys.env.getOrElse("PIO_SPARK_VERSION","1.4.0=E2=80=9D)

these are then used in the lib dependencies lists to pull in the right =
versions of artifacts.

This in some form would allow templates to move along in lock step with =
changes in the way pio is built on any given machine. Without something =
like this, users even less expert at sbt than myself (hard to imagine) =
will have a significant problem dumped on them.

Since this is only partially baked it may not be ready for a Jira and so =
warrants discussion. =20=