brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aled Sage <>
Subject Re: Bundles in Brooklyn
Date Thu, 22 Jun 2017 17:51:13 GMT
+1 to this approach, but I suspect there's a lot more to discuss/plan.

_*Leaking OSGi*_
I agree with Geoff's worry of us "leaking OSGi". However, we should just 
"leak" the semantics of bundle dependencies and versioning, which need 
to be explained to the user. I think we can avoid exposing the OSGi 
syntax anywhere (except for power-users who want it).

_*`brooklyn.libraries` versus `package-imports analogy*_
OSGi recommends the use of package-imports, rather than explicitly 
depending on specific bundles.

The analogy for us would be to depend on other catalog item 
name:versionRange (e.g. in a `brooklyn.imports` section, or whatever). 
We'd then figure out which bundle to get those from using the same rules 
as OSGi package-imports.

Is it worth exploring further that option, rather than going down the 
`brooklyn.libraries` route?

_*References Between Bundles*_
I don't think long-term the `brooklyn.libraries` should have URLs of 
other bundles. It could give the name:versionRange of other bundles 
(which assumes they have somehow already been added to Brooklyn).

(Reasoning is that other smart folk don't embed the URLs, presumably 
because they want people to chose which repo things will come from - 
e.g. enterprise private repo, versus maven central, etc).

_*Deploying Multiple Bundles*_
I lean towards something analogous to the kar file, but maybe lighter 
weight rather than introducing anything like "features".

Really we just need a list of bundle URLs. If someone then deletes one 
of those bundles, which breaks the other bundles deployed along with it, 
then tough!

_*Blueprint Validation*_
Longer term, it would be nice if we further improved our blueprint 
validation to ensure all references in a catalog item being added are 
actually resolvable.

Adding to Brooklyn the ability to add a list-of-bundles will no doubt 
make that harder!

_*Incremental Steps?*_
This might well prove to be a big change under-the-covers. Things that 
scare me include:
* do we need to re-implement for catalog-items the same 
dependency/version rules as OSGi uses for class-loading? (+)
* how much will backwards compatibility with historic persisted state 
make things hard for us.

Are there sensible ways to scope this and de-risk it? Any incremental 
milestones, which give us incremental value or de-risk it?

(+) We could potentially hack something where we synthesise a 
package/resource per catalog item in our auto-generated bundles, and 
then ask OSGi where it would load that from! (But that feels like a bad 
idea, because power-users might add their own bundles directly to karaf.)

_*Business Case (aka value to end-users)*_
It would be good to be clearer on the user-value we deliver from this 
(but please shout if others think that is already clear enough).

We're making "upgrades" easier - focusing specifically on deploying a 
second version of a catalog item, which can then be used (ignoring 
upgrade of existing app instances for now).

We're solving something like the DLL-hell problem (currently, if you 
don't declare explicit versions within your blueprint, then your 
blueprint might break when someone else adds a newer version of your 
dependent blueprint for their own usage). Our current workaround for DLL 
hell is impractical: hardcode explicit version numbers of your 
dependencies in the `type: name:version` (no support for version ranges 
etc, and it would be burried deep within the yaml files).

We're making it easier to deploy different versions of a "group of 
catalog items" (e.g. a big app that is split into several catalog 
items). Currently (if you didn't hardcode versions in every `type: ...` 
reference), deploying a 2.0.0 version would likely break a subsequent 
deployment of the 1.0.0 version, because it would default to the latest 
for each catalog reference. That is, you'd only get the 1.0.0 version of 
the main item, and then the 2.0.0 version of each item it referenced.

We'd move away from `brooklyn.libraries` hard-coding URLs to other 
bundles, which most likely goes against the security policies of many 

Is that a fair summary? Anything to add/change?


On 21/06/2017 09:41, Geoff Macartney wrote:
> hi Alex,
> that sounds fair enough.  We'll want to add good documentation around what
> it means if *everything* is now a bundle (and how we wrap plain catalogs
> into bundles on users' behalf).  We can use the text of your email as a
> starting point for the docs; it would be good to explain some of the
> _motivation_ around the choice to make everything a bundle, rather than
> just say what we do.
> I do like the idea of version ranges in brooklyn.libraries, but agree it's
> best to proceed in small steps so that would be a later development.
> Some more questions on implementation details
> - is it part of this plan to move bundle storage into the persistence
> location so that HA Brooklyn nodes can find all bundles?
> - if a (symbolic name xyz) includes brooklyn.libraries and
> specifies a remote URL (which gets downloaded), what happens if we delete
> the bundle xyz - do the downloaded bundles get "cascade deleted"?
> Geoff
> On Tue, 20 Jun 2017 at 17:22 Alex Heneveld <>
> wrote:
>> Hi Geoff
>> Thanks.  Replies to specific points:
>>   > whether we are "leaking" an implementation detail (the fact that we
>> use an OSGI runtime)
>> My motivation was to try to minimize this leak.  The fact that we use
>> OSGi internally for
>> dependency isolation forces us to use OSGi versions with bundles. But
>> most people
>> most places use the "-qualifier" suffix not ".qualifier", so to
>> accommodate that the
>> proposal suggests a recommended syntax in line with what most people do
>> ("-qualifer")
>> which we can internally safely convert to an OSGi version when needed.
>> There is no need for authors ever to see OSGi and the docs PR doesn't
>> mention OSGi
>> until an "advanced" section.
>>   > proposal would rely on Brooklyn moving exclusively to Karaf
>> No.  The bundle version format is already a problem with the use of
>> embedded Felix
>> (it solves the problem noted in #672 about needing to use ".qualifer" in
>> your
>>   > libraries / required-bundle / version ranges
>> Where users want to indicate dependent bundles they currently have the
>> choice of
>> "brooklyn.libraries" (non-osgi) or "require-bundle" (osgi).  Both work.
>> We don't need to disable the
>> former.  I'm in two minds about whether we'd want to, at some point, but
>> not right now of course.
>> An argument for disabling is that it sets up a single canonical way to
>> do things.  Yes it
>> means people have to learn OSGi but this is a fairly advanced use case
>> and OSGi is the right tool for the job.
>> If we didn't, it almost certainly means adding version range support in
>> `brooklyn.libraries` which
>> is yet more duplication of what OSGi lets us do.  However it would be
>> easier for non-OSGi users
>> (and a lot of OSGi users I bet!) if we did invest in a nice range syntax
>> in `brooklyn.libraries`.
>> Note `brooklyn.libraries` is the only place where you can specify a URL
>> for a dependent bundle;
>> the OSGi manifest doesn't let you do this (though if using Karaf you can
>> configure it to look in a set of repos,
>> which is possibly).
>> It boils down to how we expect people to manage the installation of
>> multiple catalog bundles
>> (URLs v configured repos v something else ... maybe KAR archives which
>> could easily be supported
>> but they're wasteful if the same dependency is included in many KARs).
>> Not sure where we'll want to go with this but it's not something we need
>> to solve for this proposal.
>> HTH
>> Best
>> Alex
>> On 20/06/2017 17:00, Geoff Macartney wrote:
>>> hi Alex,
>>> In general I think this is a good direction to go in, some questions and
>>> thoughts below.
>>> One concern is whether we are "leaking" an implementation detail (the
>> fact
>>> that we
>>> use an OSGI runtime) into the specification of what Brooklyn is.  I think
>>> it is good to
>>> aim for the improved modularity of having a bundle of catalog definitions
>>> and related
>>> resources, all at a given version and added or removed as a unit. I'm
>> less
>>> sure we
>>> want to advertise that all Brooklyn definitions must be contained in OSGI
>>> bundles,
>>> even if we will wrap them for you.  This seems to impose an additional
>>> requirement
>>> on blueprint authors (learn OSGI) that we don't really want to impose?
>>> As an aside, I'm assuming that your proposal would rely on Brooklyn
>> moving
>>> exclusively to the Karaf launcher, is that the case?
>>> Specific notes inline below.
>>> Geoff
>>> On Mon, 19 Jun 2017 at 17:39 Alex Heneveld <
>>> wrote:
>>>> Hi All-
>>>> With bundles now being neatly supported for add and remove, I think it
>>>> makes sense to use make the "bundle" a first-class concept in managing a
>>>> brooklyn catalog (type registry).
>>>> The basic idea is that users will work with ZIPs (or directories if
>>>> using the br CLI), where a "" is in the root; this is a
>>>> bundle, and operations like add/remove/update apply to the bundle.
>>>> Increasingly people are doing this for additions, so it seems pretty
>>>> obvious that delete should follow suit, and something like a "/bundles"
>>>> REST endpoint is appropriate.
>>> I do like this; as part of this we should re-instate the automatic
>> scanning
>>> of bundles for (feature FEATURE_LOAD_BUNDLE_CATALOG_BOM).
>>> Now that catalog items can tell you their containing bundle
>>> (getContainingBundle())
>>> it should be possible to make this work with rebind.
>>>> You can still upload just a file; in this proposal it will
>>>> be wrapped in a bundle (internally, hidden from user).  A bundle name
>>>> and version will have to be inferred/guessed for backwards
>>>> compatibility, and probably we should deprecate supplying a BOM which
>>>> doesn't define a bundle name and version.
>>> +1
>>>> If a ZIP/folder is supplied, it doesn't have to contain any OSGi bundle
>>>> definition; that call all be inferred.  If a MANIFEST.MF is supplied,
>>>> any bundle name/version given there must of course match that in the
>>>> file.
>>>> If dependent bundles are declared, they are obeyed at
>>>> runtime and can be used to find things (pretty much identically to how
>>>> we treat "libraries" in the
>>> Not so sure about that last paragraph - again this is related to leaking
>>> the use
>>> of OSGI.  This would allow someone to write a that would ONLY
>>> work when supplied along with a related manifest (to specify certain
>>> dependencies
>>> and so set up the search paths).  Do we want to allow that, or should all
>>> definitions
>>> required for entities in a catalog be supplied in the (or its
>>> included
>>> catalog definitions).
>>>> * Search Paths - Bundles fix an issue where types have implicit version
>>>> dependencies.  Consider an "acme-cluster" of "acme-node" entities,
>>>> defined in a bundle.  If I release a 1.0 version of this, and then a 2.0
>>>> version, I'd probably expect the 1.0 acme-cluster to load 1.0 acme-node
>>>> entities, and same for 2.0.  But currently any reference to "acme-node"
>>>> takes the latest version, even if there is a different version in the
>>>> same bundle.  Shifting to bundles, I think we should change this, so
>>>> that versions of items in the same bundle (or on the OSGi or library
>>>> search path for that bundle) are preferred over more recent versions in
>>>> the repository.
>>> +1 this sounds like a better match to what blueprint authors intend
>>>> (And since we will follow OSGi loading, we can also, if
>>>> we need to, use OSGi config to say that "acme" bundle requires a given
>>>> version _range_ for some other bundle, to have more control over
>>>> dependency versions when required.)
>>> again, how "use OSGI config"  - there may be value in specifying ranges
>> but
>>> we wouldn't want users to have to use OSGI declarations to achieve it.
>>>> For most users and blueprints this will cause no changes.  However there
>>>> are some changes to note:
>>>> * If people have been sloppy with versions or metadata, in most cases
>>>> they will get a warning
>>>> * In some cases if uploading a ZIP, the may need to be
>>>> edited (the upload will give a clear error, and it should not expect
>>>> rebind)
>>>> * When a type is referenced with no version, the loading semantics may
>>>> change if that type is in the same bundle (this is probably what the
>>>> user expects)
>>>> Does anyone oppose any of the above, or have further questions?
>>>> Best
>>>> Alex
>>>> [1]

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message