brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aled Sage <>
Subject Re: Uploading ZIPs for a better dev workflow
Date Wed, 01 Mar 2017 17:26:48 GMT
Hi all,

Discussion of the REST api kicked off again in

    Alex wrote:

    Requiring a MANIFEST.MF makes the input strongly java-centric; I'd
    like to appeal to people who write YAML blueprints with co-bundled
    scripts and images. They'd wonder why they can't simply make and
    upload a ZIP. They could probably be persuaded to supply a name and
    optional version on a CLI or UI, and understand why it is needed
    (hence supporting those args) but it would not be idiomatic to
    anyone but a java programmer to create a META-INF/ dir with a
    MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
    convert the ZIP to a JAR on the server. Feels like uncontroversial
    good UX.

    OTOH for a java programmer a MANIFEST.MF is natural, and they'd want
    to drop the name/version args if they are in that file, and I see no
    reason to forbid that pattern.

I agree with Alex, that we should not require a META-INF/MANIFEST.MF. As 
for Geoff's suggestion that we could auto-generate a manifest in the 
`br` CLI, I'd prefer a more general solution that works for users of the 
REST api as well (i.e. doing it server-side).

Svet suggested that the could give the symbolic name + 
version, via additional metadata in that file.

What I really like about that is it's in version control, becuase it's 
in the file/repo. If alternatively the name/version are just passed as 
REST api parameters, then it's not in version control (and is more 
susceptible to typos etc).

I'm not sure what we'd want this to look like. As a straw man:

         symbolic_name: com.example.Foo
         version: 1.0.0-SNAPSHOT
       - ...

If the user built their own real OSGi bundle, then they wouldn't need to 
include this "manifest" section. If they did include that and it 
contradicted the OSGi bundle's manifest, then we'd fail-fast.

(Note this reminds me of the (unrelated) metadata described in there is a 
"publish" block that can be added to the

With Svet's suggestion, if there was no manifest section/file, then 
should we auto-generate something from the item(s) in the

I can see how that could easily work for a .bom file that has a single 
item (e.g. we use the catalog item's id + version, possibly with a 
special prefix in the symbolic name to avoid conflicts).

However, if there are multiple items then it would get trickier.

I'm inclined to say that for a minimal viable product (MVP) we can 
insist on the "manifest" section in the


On 20/12/2016 16:34, Svetoslav Neykov wrote:
>> Svet, if instead we tried to infer it from the, would we require some
additional metadata within the .bom file? Or would we use the catalog item's id + version?
I'm not convinced by the latter - it would mean some .bom files would work and others wouldn't
(e.g. if the .bom had multiple items with different versions). Better to support the explicit
approach IMO.
> I imagine it would be additional metadata. On the other hand I don't see a technical
reason why we need an explicit symbolicName and version - they can be auto-generated.
> Svet.
>> On 20.12.2016 г., at 17:50, Aled Sage <> wrote:
>> Hi all,
>> +1
>> (D) sounds good. What version are you imagining the bundle would be, if one runs
`br catalog add ~/my/project/ --name com.example.myproject`?
>> ---
>> I like the idea of uploading a plain zip (rather than only supporting OSGi bundles)
- that makes it simpler for non-java folk. The use of OSGi becomes a (hidden) implementation
detail to many users.
>> ---
>> If auto-generating the manifest, I think we need the user to be explicit about symbolic
name and version. Having these supplied in the REST api call (as Alex suggests) would achieve
>> Svet, if instead we tried to infer it from the, would we require some
additional metadata within the .bom file? Or would we use the catalog item's id + version?
I'm not convinced by the latter - it would mean some .bom files would work and others wouldn't
(e.g. if the .bom had multiple items with different versions). Better to support the explicit
approach IMO.
>> ---
>> For E ("have a mechanism whereby deployed entities based on an affected blueprint
are optionally migrated to the new code"), that feels like a separate discussion. It could
equally apply to a pure YAML .bom file that has been added to the catalog.
>> I suggest we discuss that in a separate email thread.
>> ---
>> For (G), it's an interesting suggestion from Svet to make use of Karaf Cellar for
HA nodes. I'm hesitant (e.g. if restarting a standalone Brooklyn node whose VM has died, then
it adds big additional requirements for what constitutes the "persisted state"). On the other
hand, it's good to use well-established technologies rather than re-inventing things!
>> An alternative ("pure brooklyn") approach could be to write the bundle to persisted
state; on rebind, we'd install + activate those bundles.
>> ---
>> For "catalogGroupId", I agree with Svet that in the initial use-case this can be
an implementation detail.
>> It could be set as the bundle's symbolic name + version: everything from the bundle
should be deleted at once, along with the bundle.
>> Longer term, I can see how exposing "catalogGroupId" to the user could support more
use-cases (e.g. for several catalog items from different bundles to work together). I don't
think we should try to support that yet.
>> Aled
>> On 19/12/2016 17:19, Geoff Macartney wrote:
>>> hi Alex,
>>> this looks like a good feature to have, I shall look at the PR as soon as I
>>> can.
>>> The scanner feature was initially enabled by default, but we
>>> had to
>>> disable it because it turned out not to work properly with rebind. I don't
>>> think
>>> it should be a lot of work to fix that but it hasn't been something we've
>>> got round
>>> to yet.  This would be a great opportunity to look back at that.
>>> Some random thoughts:
>>> re (C), if we are going to treat the zips as bundles, my gut feel is that
>>> we
>>> should insist on a manifest and get the metadata from it.  It doesn't feel
>>> to me
>>> like it makes much sense to allow a zip file without a MANIFEST.MF but
>>> convey
>>> the intended bundle metadata to Brooklyn via HTTP headers.  And rather than
>>> infer bundle metadata I think it's better to ask users to be explicit about
>>> what
>>> their intentions are.  To make users lives easier, we could
>>> add a command to br to generate the manifest (locally) with correct syntax,
>>> so that the manifest is in the right place, rather than have br add the data
>>> to the "upload" request headers.
>>> re. (D) will be glad to have a look at it
>>> re. (E) it would certainly need to be optional - maybe keep it as an
>>> explicit
>>> separate command ('upgrade'?)
>>> (F) it does seem like a lot of work but might be nice for users who are not
>>> keen on command lines.
>>> G - I:  we'll definitely need to pay close attention to persistence and
>>> rebind;
>>> I wonder also about HA operation, are there any additional implications?
>>> (J) I think it would be good to treat all the files from a jar, sorry
>>> bundle,
>>> as an atomic group - cleaner that way perhaps than allowing delete/update
>>> of
>>> individual entries from a bundle on a piecemeal basis.  Rest support on
>>> delete
>>> catalog could warn about related catalog entries being deleted and ask for
>>> a "--force" param to confirm.
>>> Geoff
>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
>>>> wrote:
>>>> +1
>>>> Some thoughts:
>>>>    * (A) add a utility class BundleMaker
>>>>      Sounds very similar to
>>>> <
>>>>      Looking at the code it's much more focused on zip files so I guess
>>>> there's no much overlap, but worth keeping in mind
>>>>    * (C) accept bundle symbolic name and version
>>>>      Why require them at all? Could infer them from the in some
>>>> way - maybe require those properties to be in there. If not present are
>>>> they really needed?
>>>>    * (G) Bundles installed via this mechanism are not persisted currently
>>>> (I) We persist the individual catalog items as YAML, so we end up with two
>>>> records
>>>>      Suggest marking the catalog items coming from bundles as
>>>> non-persistable. Then try to share the bundles between HA nodes. (Karaf
>>>> Cellar?)
>>>>    * (J) Introduce a catalogGroupId field on catalog items;
>>>>      Agree this could be useful and I like the idea of deleting the bundle
>>>> altogether with the catalog items. From user's perspective I don't see the
>>>> need for an extra field (i.e. it's an implementation detail).
>>>> Svet.
>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
>>>>> wrote:
>>>>> Hi Brooklyners-
>>>>> In the code we currently have two routes for users to install new
>>>>> blueprints:
>>>>> (1) upload a catalog YAML file to /v1/catalog
>>>>> (2) install a bundle with in the root
>>>>> The feature (2) is disabled by default, but I'd like to move towards
>>>>> enabling it.  This will make it easier to create nicely structured BOM
>>>>> files because scripts etc can be taken out of the BOM, stored as files
>>>>> the same bundle.  (Because URLs of the form
>>>>> `classpath://scripts/`  use the bundle's classpath to resolve.)
>>>>> As a first step in #485 [1] I do a few things:
>>>>> (A) add a utility class BundleMaker that lets us create and modify
>>>>> bundles/zips, to make it easier to do things we might want to with
>>>> bundles,
>>>>> especially for testing
>>>>> (B) add an endpoint to the REST API which allows uploading a bundle ZIP
>>>>> (C) accept bundle symbolic name and version in that REST API to
>>>> facilitate
>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF is automatically
>>>>> generated
>>>>> With this PR, if you have a directory on your local file system with
>>>>> scripts and config files, and a BOM which refers to them, you can just
>>>> ZIP
>>>>> that up an upload it, specifying the bundle name so that a YAML blueprint
>>>>> author never needs to touch any java-isms.
>>>>> Where I see this going is a development workflow where a user can edit
>>>>> files locally and upload the ZIP to have that installed, and if they
>>>>> changes locally they can POST it again to have catalog items updated
>>>>> (because default version is a SNAPSHOT).  We could also:
>>>>> (D) have `br catalog add ~/my/project/ --name my.project` create a ZIP
>>>> and
>>>>> POST it, with bundle name metadata, so essentially the user's process
>>>>> just to run that whenever they make a change
>>>>> (E) have a mechanism whereby deployed entities based on an affected
>>>>> blueprint are optionally migrated to the new code, so if you've changed
>>>> an
>>>>> enricher the changes are picked up, or if say a script has
>>>>> changed, a restart will run the new code
>>>>> The above are fairly straightforward programmatically (although good
>>>>> interaction with (E) needs some thought).  So I think we can pretty
>>>> quickly
>>>>> get to a much smoother dev workflow.
>>>>> That's the highlight of this message.  You can jump to the end, unless
>>>>> you're interested in some important but low level details...
>>>>> I'm also tempted by:
>>>>> (F) Integration with web-based IDE and/or Brooklyn reading and writing
>>>>> straight from GitHub -- but this seems like a lot of work and I'm not
>>>>> convinced it's much better than (D) workflow-wise
>>>>> Before we can change (2) to be the default, or start widely using the
>>>> POST
>>>>> a ZIP feature, we need to sort out some issues to do with persistence
>>>>> reloading:
>>>>> (G) Bundles installed via this mechanism are not persisted currently,
>>>> if
>>>>> you move to a different Brooklyn using the same backing store, you'll
>>>> lose
>>>>> those bundles
>>>>> (H) On rebind, bundles aren't always activated when needed, meaning items
>>>>> can't be loaded
>>>>> (I) We persist the individual catalog items as YAML, so we end up with
>>>> two
>>>>> records — the YAML from the in the bundle, and the YAML
>>>>> persisted for the item.  This isn't a problem per se, but something to
>>>>> think about, and some sometimes surprising behaviour.  In particular
>>>> you
>>>>> delete the persisted YAML, the bundle is still there so the item is no
>>>>> longer deleted after a full rebind.
>>>>> One idea which might be useful is:
>>>>> (J) Introduce a catalogGroupId field on catalog items; this will do two
>>>>> things:  if you try to delete an item with such a record, you'll be
>>>>> encouraged to delete all such items (maybe disallowed to delete an
>>>>> individual one), with the effect of deleting the bundle if it comes from
>>>> a
>>>>> bundle; and when resolving types we search first for items with the same
>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0 and MyNode:1.0
>>>>> the same group, the former can refer simply to "MyNode" but if I install
>>>> a
>>>>> 2.0 version of that group, the 1.0 cluster still loads the 1.0 node --
>>>> this
>>>>> has bitten people i the past)
>>>>> There is a related Brooklyn upgrade problem worth mentioning, which the
>>>>> above might help with, where:
>>>>> (K) If I migrate from Brooklyn 10 to 11 when it comes out, I'll no longer
>>>>> have certain entities that were at v10, since we don't include those;
>>>>> upgrade could include rules that certain groupIds need to be updated,
>>>> it
>>>>> can search and attempt to automatically apply the updates
>>>>> Quite a lot here and we don't need to solve it but I wanted to:
>>>>> * Share the current thinking
>>>>> * Get opinions on the general dev workflow suggested by (D)
>>>>> Thanks for feedback -- and if we like it help with (D) would be
>>>> appreciated!
>>>>> Best
>>>>> Alex
>>>>> [1] .

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message