Mailing-List: contact dev-help@brooklyn.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@brooklyn.apache.org
MIME-Version: 1.0
References: <CA+gmfaj1R1PhwT23wpVHkqAz=suj0sin7WbMPqx0C6iVP5SSBA@mail.gmail.com>
 <FCC19423-45B3-4B6F-B74F-76DBB063B101@cloudsoftcorp.com> <CAE7C0PdDXvOc=JTerDefm5fH8KfQJAsDqaaJjzZZ1X24XedoEQ@mail.gmail.com>
 <267df34a-adbf-45ef-89dc-0b9fbb100f16@gmail.com> <91F11F00-0691-4103-9BA6-5D4A91F758AD@cloudsoftcorp.com>
 <0d1cc758-1b03-1d79-8101-a4ae31ed17dc@gmail.com> <CAE7C0PcFNBHHfo7D9xCdKWB-JhP7fK5vE0KEEs_7d6XBzYjB-w@mail.gmail.com>
 <f6a36124-7029-0b71-29e9-f5c01953c4dd@gmail.com> <CAE7C0PeGwZ8Eykt7475v6pznN_pYWEStpRsb4Tk-V7C3cLJpEA@mail.gmail.com>
In-Reply-To: <CAE7C0PeGwZ8Eykt7475v6pznN_pYWEStpRsb4Tk-V7C3cLJpEA@mail.gmail.com>
From: Geoff Macartney <geoff.macartney@cloudsoftcorp.com>
Date: Fri, 03 Mar 2017 09:59:34 +0000
Message-ID: <CAE7C0PftLCkaJoO+LHtKRE0R2E5KrMJMfG1JTGM+4rizS_f01w@mail.gmail.com>
Subject: Re: Uploading ZIPs for a better dev workflow
To: dev@brooklyn.apache.org
Content-Type: multipart/alternative; boundary=001a114028d2f4a2750549d09a63
archived-at: Fri, 03 Mar 2017 09:59:55 -0000

--001a114028d2f4a2750549d09a63
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

It also means clients ('br' at least) can easily validate the upload even
before execution by checking that all the manifest details are present.

On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
geoff.macartney@cloudsoftcorp.com> wrote:

> Just on the last point of "We could also support the example below", I'd
> say let's not even be that flexible - my feeling is that the more flexibl=
e
> we are, the more confusing it is, and the harder to get right.  If we're
> going to add this capability let's just have one way to do it; if you are
> going to do a ZIP upload, you MUST have a catalog.bom, which MUST contain=
 a
> manifest section, which MUST have both a symbolic name and a version.  Th=
at
> way everything's explicit and very clear, every time.
>
>
>
> On Thu, 2 Mar 2017 at 11:45 Aled Sage <aled.sage@gmail.com> wrote:
>
> Geoff, all,
>
> I was imagining the manifest version (in the catalog.bom) to be separate
> from the item version. The reason is that we support multiple items in
> the bom that can be independently versioned.
>
> Somone perverted could write:
>
>     brooklyn.catalog:
>        version: 1.0
>        manifest:
>          symbolic_name: com.example.Foo
>          version: 2.0
>        items:
>        - id: foo
>          version: 3.0
>          itemType: entity
>          item:
>            type: blah
>        - id: bar
>          version: 4.0
>          itemType: entity
>          item:
>         type: blah.blah
>
> Here, the "1.0" version is not used by anything; the auto-generated OSGi
> bundle would be version 2.0; catalog item foo would be 3.0; and catalog
> item bar would be 4.0.
>
> If we were starting from scratch, I'd be tempted to lock things down a
> lot more for what we actually support. For now, it feels like asking for
> an explicit version is a reasonable thing to do.
>
> ---
> We could also support the example below, where "1.0" is used by both the
> manifest and the item:
>
>     brooklyn.catalog:
>        version: 1.0
>        manifest:
>          symbolic_name: com.example.Foo
>        items:
>        - id: foo
>          itemType: entity
>          item:
>            type: blah
>
> The general rule would be: if there is a version in the manifest
> section, that will be used; if not then a top-level version number will
> be used. If that is also missing, then fail.
>
> i.e. we would not accept:
>
>     # Fails - manifest has no version.
>     brooklyn.catalog:
>        manifest:
>          symbolic_name: com.example.Foo
>        items:
>        - id: foo
>          version: 1.0
>          itemType: entity
>          item:
>            type: blah
>
> Thoughts?
>
> Aled
>
>
> On 02/03/2017 10:02, Geoff Macartney wrote:
> > I take Alex's point about the manifest being Java specific, and I agree
> > therefore we shouldn't insist on it.
> >
> > +1 also to preferring explicit name/version in the catalog.bom rather
> than
> > as API params, I agree we do
> > need to keep the version in source control.
> >
> > Question on your straw man, does the 'version' below
> >
> >      brooklyn.catalog:
> >         manifest:
> >           symbolic_name: com.example.Foo
> >           version: 1.0.0-SNAPSHOT
> >         items:
> >
> > _replace_ the 'version' in existing catalogs
> >
> > brooklyn.catalog:
> >      version: "0.1.0-SNAPSHOT"
> >
> > or is it an independently variable value, i.e. the version of the bundl=
e,
> > separate from the version of catalog items that it happens to contain?
> > (Sounds a bit disquieting, but I guess it's possible).
> >
> > e.g. could you have
> >
> > brooklyn.catalog:
> >    version: 0.11.0-SNAPSHOT
> >    manifest:
> >      symbolic_name: com.example.Foo
> >      version: 1.0.0-SNAPSHOT
> >    items:
> >    - id: foo
> >      type: xyz
> >
> > which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens to
> > contain item foo:0.11.0-SNAPSHOT?
> >
> > If so, what would happen if you left out the second line above? What
> > version of 'foo' would the catalog contain?
> >
> > Or does the version within the manifest _mean_ that it is not only the
> > bundle version you are specifying, but the catalog item versions too? (=
I
> > guess unless the item explicitly supplies its own version.)
> >
> > Geoff
> >
> >
> >
> >
> > On Wed, 1 Mar 2017 at 17:26 Aled Sage <aled.sage@gmail.com> wrote:
> >
> > Hi all,
> >
> > Discussion of the REST api kicked off again in
> >
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366=
:
> >
> >      Alex wrote:
> >
> >      Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
> >      like to appeal to people who write YAML blueprints with co-bundled
> >      scripts and images. They'd wonder why they can't simply make and
> >      upload a ZIP. They could probably be persuaded to supply a name an=
d
> >      optional version on a CLI or UI, and understand why it is needed
> >      (hence supporting those args) but it would not be idiomatic to
> >      anyone but a java programmer to create a META-INF/ dir with a
> >      MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
> >      convert the ZIP to a JAR on the server. Feels like uncontroversial
> >      good UX.
> >
> >      OTOH for a java programmer a MANIFEST.MF is natural, and they'd wa=
nt
> >      to drop the name/version args if they are in that file, and I see =
no
> >      reason to forbid that pattern.
> >
> > I agree with Alex, that we should not require a META-INF/MANIFEST.MF. A=
s
> > for Geoff's suggestion that we could auto-generate a manifest in the
> > `br` CLI, I'd prefer a more general solution that works for users of th=
e
> > REST api as well (i.e. doing it server-side).
> >
> > ---
> > Svet suggested that the catalog.bom could give the symbolic name +
> > version, via additional metadata in that file.
> >
> > What I really like about that is it's in version control, becuase it's
> > in the file/repo. If alternatively the name/version are just passed as
> > REST api parameters, then it's not in version control (and is more
> > susceptible to typos etc).
> >
> > I'm not sure what we'd want this to look like. As a straw man:
> >
> >      brooklyn.catalog:
> >         manifest:
> >           symbolic_name: com.example.Foo
> >           version: 1.0.0-SNAPSHOT
> >         items:
> >         - ...
> >
> > If the user built their own real OSGi bundle, then they wouldn't need t=
o
> > include this "manifest" section. If they did include that and it
> > contradicted the OSGi bundle's manifest, then we'd fail-fast.
> >
> > (Note this reminds me of the (unrelated) metadata described in
> > https://github.com/brooklyncentral/blueprint-repository- there is a
> > "publish" block that can be added to the catalog.bom.)
> >
> > ---
> > With Svet's suggestion, if there was no manifest section/file, then
> > should we auto-generate something from the item(s) in the catalog.bom?
> >
> > I can see how that could easily work for a .bom file that has a single
> > item (e.g. we use the catalog item's id + version, possibly with a
> > special prefix in the symbolic name to avoid conflicts).
> >
> > However, if there are multiple items then it would get trickier.
> >
> > I'm inclined to say that for a minimal viable product (MVP) we can
> > insist on the "manifest" section in the catalog.bom.
> >
> > Aled
> >
> >
> > On 20/12/2016 16:34, Svetoslav Neykov wrote:
> >>> Svet, if instead we tried to infer it from the catalog.bom, would we
> > require some additional metadata within the .bom file? Or would we use
> the
> > catalog item's id + version? I'm not convinced by the latter - it would
> > mean some .bom files would work and others wouldn't (e.g. if the .bom h=
ad
> > multiple items with different versions). Better to support the explicit
> > approach IMO.
> >> I imagine it would be additional metadata. On the other hand I don't s=
ee
> > a technical reason why we need an explicit symbolicName and version -
> they
> > can be auto-generated.
> >> Svet.
> >>
> >>
> >>> On 20.12.2016 =D0=B3., at 17:50, Aled Sage <aled.sage@gmail.com> wrot=
e:
> >>>
> >>> Hi all,
> >>>
> >>> +1
> >>>
> >>> (D) sounds good. What version are you imagining the bundle would be, =
if
> > one runs `br catalog add ~/my/project/ --name com.example.myproject`?
> >>> ---
> >>> I like the idea of uploading a plain zip (rather than only supporting
> > OSGi bundles) - that makes it simpler for non-java folk. The use of OSG=
i
> > becomes a (hidden) implementation detail to many users.
> >>> ---
> >>> If auto-generating the manifest, I think we need the user to be
> explicit
> > about symbolic name and version. Having these supplied in the REST api
> call
> > (as Alex suggests) would achieve that.
> >>> Svet, if instead we tried to infer it from the catalog.bom, would we
> > require some additional metadata within the .bom file? Or would we use
> the
> > catalog item's id + version? I'm not convinced by the latter - it would
> > mean some .bom files would work and others wouldn't (e.g. if the .bom h=
ad
> > multiple items with different versions). Better to support the explicit
> > approach IMO.
> >>> ---
> >>> For E ("have a mechanism whereby deployed entities based on an affect=
ed
> > blueprint are optionally migrated to the new code"), that feels like a
> > separate discussion. It could equally apply to a pure YAML .bom file th=
at
> > has been added to the catalog.
> >>> I suggest we discuss that in a separate email thread.
> >>>
> >>> ---
> >>> For (G), it's an interesting suggestion from Svet to make use of Kara=
f
> > Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
> Brooklyn
> > node whose VM has died, then it adds big additional requirements for wh=
at
> > constitutes the "persisted state"). On the other hand, it's good to use
> > well-established technologies rather than re-inventing things!
> >>> An alternative ("pure brooklyn") approach could be to write the bundl=
e
> > to persisted state; on rebind, we'd install + activate those bundles.
> >>> ---
> >>> For "catalogGroupId", I agree with Svet that in the initial use-case
> > this can be an implementation detail.
> >>> It could be set as the bundle's symbolic name + version: everything
> from
> > the bundle should be deleted at once, along with the bundle.
> >>> Longer term, I can see how exposing "catalogGroupId" to the user coul=
d
> > support more use-cases (e.g. for several catalog items from different
> > bundles to work together). I don't think we should try to support that
> yet.
> >>> Aled
> >>>
> >>>
> >>> On 19/12/2016 17:19, Geoff Macartney wrote:
> >>>> hi Alex,
> >>>>
> >>>> this looks like a good feature to have, I shall look at the PR as so=
on
> > as I
> >>>> can.
> >>>>
> >>>> The catalog.bom scanner feature was initially enabled by default, bu=
t
> we
> >>>> had to
> >>>> disable it because it turned out not to work properly with rebind. I
> > don't
> >>>> think
> >>>> it should be a lot of work to fix that but it hasn't been something
> > we've
> >>>> got round
> >>>> to yet.  This would be a great opportunity to look back at that.
> >>>>
> >>>> Some random thoughts:
> >>>>
> >>>> re (C), if we are going to treat the zips as bundles, my gut feel is
> > that
> >>>> we
> >>>> should insist on a manifest and get the metadata from it.  It doesn'=
t
> > feel
> >>>> to me
> >>>> like it makes much sense to allow a zip file without a MANIFEST.MF b=
ut
> >>>> convey
> >>>> the intended bundle metadata to Brooklyn via HTTP headers.  And rath=
er
> > than
> >>>> infer bundle metadata I think it's better to ask users to be explici=
t
> > about
> >>>> what
> >>>> their intentions are.  To make users lives easier, we could
> >>>> add a command to br to generate the manifest (locally) with correct
> > syntax,
> >>>> so that the manifest is in the right place, rather than have br add
> the
> > data
> >>>> to the "upload" request headers.
> >>>>
> >>>> re. (D) will be glad to have a look at it
> >>>>
> >>>> re. (E) it would certainly need to be optional - maybe keep it as an
> >>>> explicit
> >>>> separate command ('upgrade'?)
> >>>>
> >>>> (F) it does seem like a lot of work but might be nice for users who
> are
> > not
> >>>> keen on command lines.
> >>>>
> >>>> G - I:  we'll definitely need to pay close attention to persistence
> and
> >>>> rebind;
> >>>> I wonder also about HA operation, are there any additional
> implications?
> >>>>
> >>>> (J) I think it would be good to treat all the files from a jar, sorr=
y
> >>>> bundle,
> >>>> as an atomic group - cleaner that way perhaps than allowing
> > delete/update
> >>>> of
> >>>> individual entries from a bundle on a piecemeal basis.  Rest support
> on
> >>>> delete
> >>>> catalog could warn about related catalog entries being deleted and a=
sk
> > for
> >>>> a "--force" param to confirm.
> >>>>
> >>>> Geoff
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
> >>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
> >>>>
> >>>>> +1
> >>>>>
> >>>>> Some thoughts:
> >>>>>     * (A) add a utility class BundleMaker
> >>>>>       Sounds very similar to
> >>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
> >>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
> >>>>>       Looking at the code it's much more focused on zip files so I
> guess
> >>>>> there's no much overlap, but worth keeping in mind
> >>>>>     * (C) accept bundle symbolic name and version
> >>>>>       Why require them at all? Could infer them from the catalog.bo=
m
> in
> > some
> >>>>> way - maybe require those properties to be in there. If not present
> are
> >>>>> they really needed?
> >>>>>     * (G) Bundles installed via this mechanism are not persisted
> > currently &
> >>>>> (I) We persist the individual catalog items as YAML, so we end up
> with
> > two
> >>>>> records
> >>>>>       Suggest marking the catalog items coming from bundles as
> >>>>> non-persistable. Then try to share the bundles between HA nodes.
> (Karaf
> >>>>> Cellar?)
> >>>>>     * (J) Introduce a catalogGroupId field on catalog items;
> >>>>>       Agree this could be useful and I like the idea of deleting th=
e
> > bundle
> >>>>> altogether with the catalog items. From user's perspective I don't
> see
> > the
> >>>>> need for an extra field (i.e. it's an implementation detail).
> >>>>>
> >>>>> Svet.
> >>>>>
> >>>>>
> >>>>>> On 16.12.2016 =D0=B3., at 12:50, Alex Heneveld <
> >>>>> alex.heneveld@cloudsoftcorp.com> wrote:
> >>>>>> Hi Brooklyners-
> >>>>>>
> >>>>>> In the code we currently have two routes for users to install new
> >>>>>> blueprints:
> >>>>>>
> >>>>>> (1) upload a catalog YAML file to /v1/catalog
> >>>>>>
> >>>>>> (2) install a bundle with catalog.bom in the root
> >>>>>>
> >>>>>> The feature (2) is disabled by default, but I'd like to move towar=
ds
> >>>>>> enabling it.  This will make it easier to create nicely structured
> BOM
> >>>>>> files because scripts etc can be taken out of the BOM, stored as
> > files in
> >>>>>> the same bundle.  (Because URLs of the form
> >>>>>> `classpath://scripts/install.sh`  use the bundle's classpath to
> > resolve.)
> >>>>>> As a first step in #485 [1] I do a few things:
> >>>>>>
> >>>>>> (A) add a utility class BundleMaker that lets us create and modify
> >>>>>> bundles/zips, to make it easier to do things we might want to with
> >>>>> bundles,
> >>>>>> especially for testing
> >>>>>>
> >>>>>> (B) add an endpoint to the REST API which allows uploading a bundl=
e
> > ZIP
> >>>>>> (C) accept bundle symbolic name and version in that REST API to
> >>>>> facilitate
> >>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is
> automatically
> >>>>>> generated
> >>>>>>
> >>>>>> With this PR, if you have a directory on your local file system wi=
th
> >>>>>> scripts and config files, and a BOM which refers to them, you can
> just
> >>>>> ZIP
> >>>>>> that up an upload it, specifying the bundle name so that a YAML
> > blueprint
> >>>>>> author never needs to touch any java-isms.
> >>>>>>
> >>>>>> Where I see this going is a development workflow where a user can
> edit
> >>>>>> files locally and upload the ZIP to have that installed, and if th=
ey
> > make
> >>>>>> changes locally they can POST it again to have catalog items updat=
ed
> >>>>>> (because default version is a SNAPSHOT).  We could also:
> >>>>>>
> >>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a
> ZIP
> >>>>> and
> >>>>>> POST it, with bundle name metadata, so essentially the user's
> process
> > is
> >>>>>> just to run that whenever they make a change
> >>>>>>
> >>>>>> (E) have a mechanism whereby deployed entities based on an affecte=
d
> >>>>>> blueprint are optionally migrated to the new code, so if you've
> > changed
> >>>>> an
> >>>>>> enricher the changes are picked up, or if say a launch.sh script h=
as
> >>>>>> changed, a restart will run the new code
> >>>>>>
> >>>>>> The above are fairly straightforward programmatically (although go=
od
> > user
> >>>>>> interaction with (E) needs some thought).  So I think we can prett=
y
> >>>>> quickly
> >>>>>> get to a much smoother dev workflow.
> >>>>>>
> >>>>>>
> >>>>>> That's the highlight of this message.  You can jump to the end,
> unless
> >>>>>> you're interested in some important but low level details...
> >>>>>>
> >>>>>>
> >>>>>> I'm also tempted by:
> >>>>>>
> >>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and
> writing
> >>>>>> straight from GitHub -- but this seems like a lot of work and I'm
> not
> >>>>>> convinced it's much better than (D) workflow-wise
> >>>>>>
> >>>>>> Before we can change (2) to be the default, or start widely using
> the
> >>>>> POST
> >>>>>> a ZIP feature, we need to sort out some issues to do with
> persistence
> > and
> >>>>>> reloading:
> >>>>>>
> >>>>>> (G) Bundles installed via this mechanism are not persisted
> currently,
> > so
> >>>>> if
> >>>>>> you move to a different Brooklyn using the same backing store,
> you'll
> >>>>> lose
> >>>>>> those bundles
> >>>>>>
> >>>>>> (H) On rebind, bundles aren't always activated when needed, meanin=
g
> > items
> >>>>>> can't be loaded
> >>>>>>
> >>>>>> (I) We persist the individual catalog items as YAML, so we end up
> with
> >>>>> two
> >>>>>> records =E2=80=94 the YAML from the catalog.bom in the bundle, and=
 the YAML
> >>>>>> persisted for the item.  This isn't a problem per se, but somethin=
g
> to
> >>>>>> think about, and some sometimes surprising behaviour.  In particul=
ar
> > if
> >>>>> you
> >>>>>> delete the persisted YAML, the bundle is still there so the item i=
s
> no
> >>>>>> longer deleted after a full rebind.
> >>>>>>
> >>>>>> One idea which might be useful is:
> >>>>>>
> >>>>>> (J) Introduce a catalogGroupId field on catalog items; this will d=
o
> > two
> >>>>>> things:  if you try to delete an item with such a record, you'll b=
e
> >>>>>> encouraged to delete all such items (maybe disallowed to delete an
> >>>>>> individual one), with the effect of deleting the bundle if it come=
s
> > from
> >>>>> a
> >>>>>> bundle; and when resolving types we search first for items with th=
e
> > same
> >>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and
> > MyNode:1.0 in
> >>>>>> the same group, the former can refer simply to "MyNode" but if I
> > install
> >>>>> a
> >>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 nod=
e
> --
> >>>>> this
> >>>>>> has bitten people i the past)
> >>>>>>
> >>>>>> There is a related Brooklyn upgrade problem worth mentioning, whic=
h
> > the
> >>>>>> above might help with, where:
> >>>>>>
> >>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no
> > longer
> >>>>>> have certain entities that were at v10, since we don't include
> those;
> > an
> >>>>>> upgrade could include rules that certain groupIds need to be
> updated,
> > or
> >>>>> it
> >>>>>> can search and attempt to automatically apply the updates
> >>>>>>
> >>>>>>
> >>>>>> Quite a lot here and we don't need to solve it but I wanted to:
> >>>>>>
> >>>>>> * Share the current thinking
> >>>>>>
> >>>>>> * Get opinions on the general dev workflow suggested by (D)
> >>>>>>
> >>>>>>
> >>>>>> Thanks for feedback -- and if we like it help with (D) would be
> >>>>> appreciated!
> >>>>>> Best
> >>>>>> Alex
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
>
>

--001a114028d2f4a2750549d09a63--