Mailing-List: contact dev-help@brooklyn.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@brooklyn.apache.org
From: Svetoslav Neykov <svetoslav.neykov@cloudsoftcorp.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.1 \(3251\))
Subject: Re: Uploading ZIPs for a better dev workflow
Date: Tue, 20 Dec 2016 18:34:04 +0200
References: <CA+gmfaj1R1PhwT23wpVHkqAz=suj0sin7WbMPqx0C6iVP5SSBA@mail.gmail.com>
 <FCC19423-45B3-4B6F-B74F-76DBB063B101@cloudsoftcorp.com>
 <CAE7C0PdDXvOc=JTerDefm5fH8KfQJAsDqaaJjzZZ1X24XedoEQ@mail.gmail.com>
 <267df34a-adbf-45ef-89dc-0b9fbb100f16@gmail.com>
To: dev@brooklyn.apache.org
In-Reply-To: <267df34a-adbf-45ef-89dc-0b9fbb100f16@gmail.com>
Message-Id: <91F11F00-0691-4103-9BA6-5D4A91F758AD@cloudsoftcorp.com>
archived-at: Tue, 20 Dec 2016 16:35:13 -0000

> Svet, if instead we tried to infer it from the catalog.bom, would we =
require some additional metadata within the .bom file? Or would we use =
the catalog item's id + version? I'm not convinced by the latter - it =
would mean some .bom files would work and others wouldn't (e.g. if the =
.bom had multiple items with different versions). Better to support the =
explicit approach IMO.

I imagine it would be additional metadata. On the other hand I don't see =
a technical reason why we need an explicit symbolicName and version - =
they can be auto-generated.=20

Svet.


> On 20.12.2016 =D0=B3., at 17:50, Aled Sage <aled.sage@gmail.com> =
wrote:
>=20
> Hi all,
>=20
> +1
>=20
> (D) sounds good. What version are you imagining the bundle would be, =
if one runs `br catalog add ~/my/project/ --name com.example.myproject`?
>=20
> ---
> I like the idea of uploading a plain zip (rather than only supporting =
OSGi bundles) - that makes it simpler for non-java folk. The use of OSGi =
becomes a (hidden) implementation detail to many users.
>=20
> ---
> If auto-generating the manifest, I think we need the user to be =
explicit about symbolic name and version. Having these supplied in the =
REST api call (as Alex suggests) would achieve that.
>=20
> Svet, if instead we tried to infer it from the catalog.bom, would we =
require some additional metadata within the .bom file? Or would we use =
the catalog item's id + version? I'm not convinced by the latter - it =
would mean some .bom files would work and others wouldn't (e.g. if the =
.bom had multiple items with different versions). Better to support the =
explicit approach IMO.
>=20
> ---
> For E ("have a mechanism whereby deployed entities based on an =
affected blueprint are optionally migrated to the new code"), that feels =
like a separate discussion. It could equally apply to a pure YAML .bom =
file that has been added to the catalog.
>=20
> I suggest we discuss that in a separate email thread.
>=20
> ---
> For (G), it's an interesting suggestion from Svet to make use of Karaf =
Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone =
Brooklyn node whose VM has died, then it adds big additional =
requirements for what constitutes the "persisted state"). On the other =
hand, it's good to use well-established technologies rather than =
re-inventing things!
>=20
> An alternative ("pure brooklyn") approach could be to write the bundle =
to persisted state; on rebind, we'd install + activate those bundles.
>=20
> ---
> For "catalogGroupId", I agree with Svet that in the initial use-case =
this can be an implementation detail.
>=20
> It could be set as the bundle's symbolic name + version: everything =
from the bundle should be deleted at once, along with the bundle.
>=20
> Longer term, I can see how exposing "catalogGroupId" to the user could =
support more use-cases (e.g. for several catalog items from different =
bundles to work together). I don't think we should try to support that =
yet.
>=20
> Aled
>=20
>=20
> On 19/12/2016 17:19, Geoff Macartney wrote:
>> hi Alex,
>>=20
>> this looks like a good feature to have, I shall look at the PR as =
soon as I
>> can.
>>=20
>> The catalog.bom scanner feature was initially enabled by default, but =
we
>> had to
>> disable it because it turned out not to work properly with rebind. I =
don't
>> think
>> it should be a lot of work to fix that but it hasn't been something =
we've
>> got round
>> to yet.  This would be a great opportunity to look back at that.
>>=20
>> Some random thoughts:
>>=20
>> re (C), if we are going to treat the zips as bundles, my gut feel is =
that
>> we
>> should insist on a manifest and get the metadata from it.  It doesn't =
feel
>> to me
>> like it makes much sense to allow a zip file without a MANIFEST.MF =
but
>> convey
>> the intended bundle metadata to Brooklyn via HTTP headers.  And =
rather than
>> infer bundle metadata I think it's better to ask users to be explicit =
about
>> what
>> their intentions are.  To make users lives easier, we could
>> add a command to br to generate the manifest (locally) with correct =
syntax,
>> so that the manifest is in the right place, rather than have br add =
the data
>> to the "upload" request headers.
>>=20
>> re. (D) will be glad to have a look at it
>>=20
>> re. (E) it would certainly need to be optional - maybe keep it as an
>> explicit
>> separate command ('upgrade'?)
>>=20
>> (F) it does seem like a lot of work but might be nice for users who =
are not
>> keen on command lines.
>>=20
>> G - I:  we'll definitely need to pay close attention to persistence =
and
>> rebind;
>> I wonder also about HA operation, are there any additional =
implications?
>>=20
>> (J) I think it would be good to treat all the files from a jar, sorry
>> bundle,
>> as an atomic group - cleaner that way perhaps than allowing =
delete/update
>> of
>> individual entries from a bundle on a piecemeal basis.  Rest support =
on
>> delete
>> catalog could warn about related catalog entries being deleted and =
ask for
>> a "--force" param to confirm.
>>=20
>> Geoff
>>=20
>>=20
>>=20
>>=20
>>=20
>>=20
>>=20
>>=20
>>=20
>>=20
>>=20
>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>>=20
>>> +1
>>>=20
>>> Some thoughts:
>>>   * (A) add a utility class BundleMaker
>>>     Sounds very similar to
>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles <
>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
>>>     Looking at the code it's much more focused on zip files so I =
guess
>>> there's no much overlap, but worth keeping in mind
>>>   * (C) accept bundle symbolic name and version
>>>     Why require them at all? Could infer them from the catalog.bom =
in some
>>> way - maybe require those properties to be in there. If not present =
are
>>> they really needed?
>>>   * (G) Bundles installed via this mechanism are not persisted =
currently &
>>> (I) We persist the individual catalog items as YAML, so we end up =
with two
>>> records
>>>     Suggest marking the catalog items coming from bundles as
>>> non-persistable. Then try to share the bundles between HA nodes. =
(Karaf
>>> Cellar?)
>>>   * (J) Introduce a catalogGroupId field on catalog items;
>>>     Agree this could be useful and I like the idea of deleting the =
bundle
>>> altogether with the catalog items. =46rom user's perspective I don't =
see the
>>> need for an extra field (i.e. it's an implementation detail).
>>>=20
>>> Svet.
>>>=20
>>>=20
>>>> On 16.12.2016 =D0=B3., at 12:50, Alex Heneveld <
>>> alex.heneveld@cloudsoftcorp.com> wrote:
>>>> Hi Brooklyners-
>>>>=20
>>>> In the code we currently have two routes for users to install new
>>>> blueprints:
>>>>=20
>>>> (1) upload a catalog YAML file to /v1/catalog
>>>>=20
>>>> (2) install a bundle with catalog.bom in the root
>>>>=20
>>>> The feature (2) is disabled by default, but I'd like to move =
towards
>>>> enabling it.  This will make it easier to create nicely structured =
BOM
>>>> files because scripts etc can be taken out of the BOM, stored as =
files in
>>>> the same bundle.  (Because URLs of the form
>>>> `classpath://scripts/install.sh`  use the bundle's classpath to =
resolve.)
>>>>=20
>>>> As a first step in #485 [1] I do a few things:
>>>>=20
>>>> (A) add a utility class BundleMaker that lets us create and modify
>>>> bundles/zips, to make it easier to do things we might want to with
>>> bundles,
>>>> especially for testing
>>>>=20
>>>> (B) add an endpoint to the REST API which allows uploading a bundle =
ZIP
>>>>=20
>>>> (C) accept bundle symbolic name and version in that REST API to
>>> facilitate
>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is =
automatically
>>>> generated
>>>>=20
>>>> With this PR, if you have a directory on your local file system =
with
>>>> scripts and config files, and a BOM which refers to them, you can =
just
>>> ZIP
>>>> that up an upload it, specifying the bundle name so that a YAML =
blueprint
>>>> author never needs to touch any java-isms.
>>>>=20
>>>> Where I see this going is a development workflow where a user can =
edit
>>>> files locally and upload the ZIP to have that installed, and if =
they make
>>>> changes locally they can POST it again to have catalog items =
updated
>>>> (because default version is a SNAPSHOT).  We could also:
>>>>=20
>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a =
ZIP
>>> and
>>>> POST it, with bundle name metadata, so essentially the user's =
process is
>>>> just to run that whenever they make a change
>>>>=20
>>>> (E) have a mechanism whereby deployed entities based on an affected
>>>> blueprint are optionally migrated to the new code, so if you've =
changed
>>> an
>>>> enricher the changes are picked up, or if say a launch.sh script =
has
>>>> changed, a restart will run the new code
>>>>=20
>>>> The above are fairly straightforward programmatically (although =
good user
>>>> interaction with (E) needs some thought).  So I think we can pretty
>>> quickly
>>>> get to a much smoother dev workflow.
>>>>=20
>>>>=20
>>>> That's the highlight of this message.  You can jump to the end, =
unless
>>>> you're interested in some important but low level details...
>>>>=20
>>>>=20
>>>> I'm also tempted by:
>>>>=20
>>>> (F) Integration with web-based IDE and/or Brooklyn reading and =
writing
>>>> straight from GitHub -- but this seems like a lot of work and I'm =
not
>>>> convinced it's much better than (D) workflow-wise
>>>>=20
>>>> Before we can change (2) to be the default, or start widely using =
the
>>> POST
>>>> a ZIP feature, we need to sort out some issues to do with =
persistence and
>>>> reloading:
>>>>=20
>>>> (G) Bundles installed via this mechanism are not persisted =
currently, so
>>> if
>>>> you move to a different Brooklyn using the same backing store, =
you'll
>>> lose
>>>> those bundles
>>>>=20
>>>> (H) On rebind, bundles aren't always activated when needed, meaning =
items
>>>> can't be loaded
>>>>=20
>>>> (I) We persist the individual catalog items as YAML, so we end up =
with
>>> two
>>>> records =E2=80=94 the YAML from the catalog.bom in the bundle, and =
the YAML
>>>> persisted for the item.  This isn't a problem per se, but something =
to
>>>> think about, and some sometimes surprising behaviour.  In =
particular if
>>> you
>>>> delete the persisted YAML, the bundle is still there so the item is =
no
>>>> longer deleted after a full rebind.
>>>>=20
>>>> One idea which might be useful is:
>>>>=20
>>>> (J) Introduce a catalogGroupId field on catalog items; this will do =
two
>>>> things:  if you try to delete an item with such a record, you'll be
>>>> encouraged to delete all such items (maybe disallowed to delete an
>>>> individual one), with the effect of deleting the bundle if it comes =
from
>>> a
>>>> bundle; and when resolving types we search first for items with the =
same
>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and =
MyNode:1.0 in
>>>> the same group, the former can refer simply to "MyNode" but if I =
install
>>> a
>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 node =
--
>>> this
>>>> has bitten people i the past)
>>>>=20
>>>> There is a related Brooklyn upgrade problem worth mentioning, which =
the
>>>> above might help with, where:
>>>>=20
>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no =
longer
>>>> have certain entities that were at v10, since we don't include =
those; an
>>>> upgrade could include rules that certain groupIds need to be =
updated, or
>>> it
>>>> can search and attempt to automatically apply the updates
>>>>=20
>>>>=20
>>>> Quite a lot here and we don't need to solve it but I wanted to:
>>>>=20
>>>> * Share the current thinking
>>>>=20
>>>> * Get opinions on the general dev workflow suggested by (D)
>>>>=20
>>>>=20
>>>> Thanks for feedback -- and if we like it help with (D) would be
>>> appreciated!
>>>> Best
>>>> Alex
>>>>=20
>>>>=20
>>>>=20
>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
>>>=20
>=20