brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Bouron <thomas.bou...@cloudsoftcorp.com>
Subject Re: Uploading ZIPs for a better dev workflow
Date Mon, 20 Mar 2017 15:16:32 GMT
Hi Brooklyners.

Uploading ZIPs is really something I'm excited to see coming to life so I
would like to help to push this forward.

I've re-read this email thread and it sounds like we have a consensus on
the "one true way" approach, i.e. always requiring the `symbolicName` and
`version` when uploading ZIP/JAR (that includes Svet in his previous email).

One thing not explicitly settled is the format of bom for those 2 fields:
Alex prefers to put these at the root, Aled in a `manifest` section. I
don't have any strong opinion on this: I was leaning toward Aled's
suggestion but the argument that says this applies to the whole bundle make
sense so I now think putting this at the root is probably better. I can see
that Alex already updated his PR[1] to this structure. What the people
think about this?

I'm going to review [1] but are we happy with this consensus? One thing
that will need to be addressed is the persistence/rebind issues but this
can be tackled in following PRs.

Best.

[1] https://github.com/apache/brooklyn-server/pull/485

On Tue, 7 Mar 2017 at 16:18 Svetoslav Neykov <
svetoslav.neykov@cloudsoftcorp.com> wrote:

> Having quickly scanned the thread I still don't see a reason why the
> requirement to have a consistent naming for the bundle itself which is not
> user visible? Could instead have behaviour similar to the machinNamer here.
>
> Slight preference for optional symbolicName when it's a jar file. Though I
> really like Geoff's idea of "one true way". The workflow to create a jar
> file is very different from creating a zip file (either from a folder or
> user supplied) so don't think they conflict.
>
> Svet.
>
>
> > On 7.03.2017 г., at 17:47, Alex Heneveld <
> alex.heneveld@cloudsoftcorp.com> wrote:
> >
> >
> > Considering the points made I'm pretty sure it's the case that requiring
> a symbolic name and version in the BOM presents little if any unnecessary
> burden, in which case I'm persuaded of the "one true way" philosophy.  I
> really like it.  :)
> >
> > Can anyone think of a case where it would be a burden?  My thinking is
> that:
> >
> > * If you're uploading a ZIP of YAML and scripts and icons, you have a
> catalog.bom and that's a useful place to require the symname and version
> > * If your ZIP/JAR also has a MANIFEST.MF the symname and version will be
> present in it also; the user is duplicating that info (this is the "little"
> unnecssary burden) but it's tiny and the benefit of one true way dominates;
> we do require that they match
> > * If your ZIP/JAR doesn't have a BOM there's no point in uploading it to
> the catalog endpoint; either it's a JAR but this isn't meant as an "install
> arbitrary bundles" endpoint, or a non-JAR in which case what are we
> supposed to do with it once you upload it?
> >
> > It keeps the contract simple, and the CLI very simple.   `br push
> /path/to/blueprint/x/`
> > (But Geoff as a side point I think the CLI should _not_ attempt
> validation of the input/version -- the server will do that anyway and no
> benefit in doing it twice, but more code to maintain.)
> >
> > I'm further persuaded given it is in a file, we should require a
> version.  I was concerned about needing to supply the version every time on
> the CLI but this has gone away.
> >
> > I see no reason for a separate "manifest" section.  Removing this
> eliminates the prospect of Aled's pathological/"perverted" example.  I
> favour:
> >
> >    brooklyn.catalog:
> >       symbolicName: com.example.Foo
> >       version: 1.0
> >       items:
> >       - id: foo
> >         version: 3.0   # optional version override
> >         itemType: entity
> >         item:
> >           type: blah
> >           ...
> >
> > That is, the fields symbolic-name and version are _required_ under the
> `brooklyn.catalog` when POSTing a ZIP to the catalog endpoint, and are used
> to define the bundle.  The version (and icon, etc) are then inherited by
> default by items defined therein.  If the ZIP is a JAR which contains a
> MANIFEST.MF, the bundle information (symbolic name and version) must match
> exactly.
> >
> > Sound right?
> >
> > Best
> > Alex
> >
> >
> > On 03/03/2017 09:59, Geoff Macartney wrote:
> >> It also means clients ('br' at least) can easily validate the upload
> even
> >> before execution by checking that all the manifest details are present.
> >>
> >> On Fri, 3 Mar 2017 at 09:57 Geoff Macartney <
> >> geoff.macartney@cloudsoftcorp.com> wrote:
> >>
> >>> Just on the last point of "We could also support the example below",
> I'd
> >>> say let's not even be that flexible - my feeling is that the more
> flexible
> >>> we are, the more confusing it is, and the harder to get right.  If
> we're
> >>> going to add this capability let's just have one way to do it; if you
> are
> >>> going to do a ZIP upload, you MUST have a catalog.bom, which MUST
> contain a
> >>> manifest section, which MUST have both a symbolic name and a version.
> That
> >>> way everything's explicit and very clear, every time.
> >>>
> >>>
> >>>
> >>> On Thu, 2 Mar 2017 at 11:45 Aled Sage <aled.sage@gmail.com> wrote:
> >>>
> >>> Geoff, all,
> >>>
> >>> I was imagining the manifest version (in the catalog.bom) to be
> separate
> >>> from the item version. The reason is that we support multiple items in
> >>> the bom that can be independently versioned.
> >>>
> >>> Somone perverted could write:
> >>>
> >>>     brooklyn.catalog:
> >>>        version: 1.0
> >>>        manifest:
> >>>          symbolic_name: com.example.Foo
> >>>          version: 2.0
> >>>        items:
> >>>        - id: foo
> >>>          version: 3.0
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>        - id: bar
> >>>          version: 4.0
> >>>          itemType: entity
> >>>          item:
> >>>         type: blah.blah
> >>>
> >>> Here, the "1.0" version is not used by anything; the auto-generated
> OSGi
> >>> bundle would be version 2.0; catalog item foo would be 3.0; and catalog
> >>> item bar would be 4.0.
> >>>
> >>> If we were starting from scratch, I'd be tempted to lock things down a
> >>> lot more for what we actually support. For now, it feels like asking
> for
> >>> an explicit version is a reasonable thing to do.
> >>>
> >>> ---
> >>> We could also support the example below, where "1.0" is used by both
> the
> >>> manifest and the item:
> >>>
> >>>     brooklyn.catalog:
> >>>        version: 1.0
> >>>        manifest:
> >>>          symbolic_name: com.example.Foo
> >>>        items:
> >>>        - id: foo
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>
> >>> The general rule would be: if there is a version in the manifest
> >>> section, that will be used; if not then a top-level version number will
> >>> be used. If that is also missing, then fail.
> >>>
> >>> i.e. we would not accept:
> >>>
> >>>     # Fails - manifest has no version.
> >>>     brooklyn.catalog:
> >>>        manifest:
> >>>          symbolic_name: com.example.Foo
> >>>        items:
> >>>        - id: foo
> >>>          version: 1.0
> >>>          itemType: entity
> >>>          item:
> >>>            type: blah
> >>>
> >>> Thoughts?
> >>>
> >>> Aled
> >>>
> >>>
> >>> On 02/03/2017 10:02, Geoff Macartney wrote:
> >>>> I take Alex's point about the manifest being Java specific, and I
> agree
> >>>> therefore we shouldn't insist on it.
> >>>>
> >>>> +1 also to preferring explicit name/version in the catalog.bom rather
> >>> than
> >>>> as API params, I agree we do
> >>>> need to keep the version in source control.
> >>>>
> >>>> Question on your straw man, does the 'version' below
> >>>>
> >>>>      brooklyn.catalog:
> >>>>         manifest:
> >>>>           symbolic_name: com.example.Foo
> >>>>           version: 1.0.0-SNAPSHOT
> >>>>         items:
> >>>>
> >>>> _replace_ the 'version' in existing catalogs
> >>>>
> >>>> brooklyn.catalog:
> >>>>      version: "0.1.0-SNAPSHOT"
> >>>>
> >>>> or is it an independently variable value, i.e. the version of the
> bundle,
> >>>> separate from the version of catalog items that it happens to contain?
> >>>> (Sounds a bit disquieting, but I guess it's possible).
> >>>>
> >>>> e.g. could you have
> >>>>
> >>>> brooklyn.catalog:
> >>>>    version: 0.11.0-SNAPSHOT
> >>>>    manifest:
> >>>>      symbolic_name: com.example.Foo
> >>>>      version: 1.0.0-SNAPSHOT
> >>>>    items:
> >>>>    - id: foo
> >>>>      type: xyz
> >>>>
> >>>> which would be bundle com.example.Foo:1.0.0-SNAPSHOT, which happens
to
> >>>> contain item foo:0.11.0-SNAPSHOT?
> >>>>
> >>>> If so, what would happen if you left out the second line above? What
> >>>> version of 'foo' would the catalog contain?
> >>>>
> >>>> Or does the version within the manifest _mean_ that it is not only the
> >>>> bundle version you are specifying, but the catalog item versions too?
> (I
> >>>> guess unless the item explicitly supplies its own version.)
> >>>>
> >>>> Geoff
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Wed, 1 Mar 2017 at 17:26 Aled Sage <aled.sage@gmail.com> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> Discussion of the REST api kicked off again in
> >>>>
> >>>
> https://github.com/apache/brooklyn-server/pull/485#issuecomment-283280366:
> >>>>      Alex wrote:
> >>>>
> >>>>      Requiring a MANIFEST.MF makes the input strongly java-centric;
> I'd
> >>>>      like to appeal to people who write YAML blueprints with
> co-bundled
> >>>>      scripts and images. They'd wonder why they can't simply make and
> >>>>      upload a ZIP. They could probably be persuaded to supply a name
> and
> >>>>      optional version on a CLI or UI, and understand why it is needed
> >>>>      (hence supporting those args) but it would not be idiomatic to
> >>>>      anyone but a java programmer to create a META-INF/ dir with a
> >>>>      MANIFEST.MF and its syntax. Meanwhile it is very easy for us to
> >>>>      convert the ZIP to a JAR on the server. Feels like
> uncontroversial
> >>>>      good UX.
> >>>>
> >>>>      OTOH for a java programmer a MANIFEST.MF is natural, and they'd
> want
> >>>>      to drop the name/version args if they are in that file, and I
> see no
> >>>>      reason to forbid that pattern.
> >>>>
> >>>> I agree with Alex, that we should not require a META-INF/MANIFEST.MF.
> As
> >>>> for Geoff's suggestion that we could auto-generate a manifest in the
> >>>> `br` CLI, I'd prefer a more general solution that works for users of
> the
> >>>> REST api as well (i.e. doing it server-side).
> >>>>
> >>>> ---
> >>>> Svet suggested that the catalog.bom could give the symbolic name +
> >>>> version, via additional metadata in that file.
> >>>>
> >>>> What I really like about that is it's in version control, becuase it's
> >>>> in the file/repo. If alternatively the name/version are just passed
as
> >>>> REST api parameters, then it's not in version control (and is more
> >>>> susceptible to typos etc).
> >>>>
> >>>> I'm not sure what we'd want this to look like. As a straw man:
> >>>>
> >>>>      brooklyn.catalog:
> >>>>         manifest:
> >>>>           symbolic_name: com.example.Foo
> >>>>           version: 1.0.0-SNAPSHOT
> >>>>         items:
> >>>>         - ...
> >>>>
> >>>> If the user built their own real OSGi bundle, then they wouldn't need
> to
> >>>> include this "manifest" section. If they did include that and it
> >>>> contradicted the OSGi bundle's manifest, then we'd fail-fast.
> >>>>
> >>>> (Note this reminds me of the (unrelated) metadata described in
> >>>> https://github.com/brooklyncentral/blueprint-repository- there is a
> >>>> "publish" block that can be added to the catalog.bom.)
> >>>>
> >>>> ---
> >>>> With Svet's suggestion, if there was no manifest section/file, then
> >>>> should we auto-generate something from the item(s) in the catalog.bom?
> >>>>
> >>>> I can see how that could easily work for a .bom file that has a single
> >>>> item (e.g. we use the catalog item's id + version, possibly with a
> >>>> special prefix in the symbolic name to avoid conflicts).
> >>>>
> >>>> However, if there are multiple items then it would get trickier.
> >>>>
> >>>> I'm inclined to say that for a minimal viable product (MVP) we can
> >>>> insist on the "manifest" section in the catalog.bom.
> >>>>
> >>>> Aled
> >>>>
> >>>>
> >>>> On 20/12/2016 16:34, Svetoslav Neykov wrote:
> >>>>>> Svet, if instead we tried to infer it from the catalog.bom,
would we
> >>>> require some additional metadata within the .bom file? Or would we use
> >>> the
> >>>> catalog item's id + version? I'm not convinced by the latter - it
> would
> >>>> mean some .bom files would work and others wouldn't (e.g. if the .bom
> had
> >>>> multiple items with different versions). Better to support the
> explicit
> >>>> approach IMO.
> >>>>> I imagine it would be additional metadata. On the other hand I don't
> see
> >>>> a technical reason why we need an explicit symbolicName and version
-
> >>> they
> >>>> can be auto-generated.
> >>>>> Svet.
> >>>>>
> >>>>>
> >>>>>> On 20.12.2016 г., at 17:50, Aled Sage <aled.sage@gmail.com>
wrote:
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> (D) sounds good. What version are you imagining the bundle would
> be, if
> >>>> one runs `br catalog add ~/my/project/ --name com.example.myproject`?
> >>>>>> ---
> >>>>>> I like the idea of uploading a plain zip (rather than only
> supporting
> >>>> OSGi bundles) - that makes it simpler for non-java folk. The use of
> OSGi
> >>>> becomes a (hidden) implementation detail to many users.
> >>>>>> ---
> >>>>>> If auto-generating the manifest, I think we need the user to
be
> >>> explicit
> >>>> about symbolic name and version. Having these supplied in the REST api
> >>> call
> >>>> (as Alex suggests) would achieve that.
> >>>>>> Svet, if instead we tried to infer it from the catalog.bom,
would we
> >>>> require some additional metadata within the .bom file? Or would we use
> >>> the
> >>>> catalog item's id + version? I'm not convinced by the latter - it
> would
> >>>> mean some .bom files would work and others wouldn't (e.g. if the .bom
> had
> >>>> multiple items with different versions). Better to support the
> explicit
> >>>> approach IMO.
> >>>>>> ---
> >>>>>> For E ("have a mechanism whereby deployed entities based on
an
> affected
> >>>> blueprint are optionally migrated to the new code"), that feels like
a
> >>>> separate discussion. It could equally apply to a pure YAML .bom file
> that
> >>>> has been added to the catalog.
> >>>>>> I suggest we discuss that in a separate email thread.
> >>>>>>
> >>>>>> ---
> >>>>>> For (G), it's an interesting suggestion from Svet to make use
of
> Karaf
> >>>> Cellar for HA nodes. I'm hesitant (e.g. if restarting a standalone
> >>> Brooklyn
> >>>> node whose VM has died, then it adds big additional requirements for
> what
> >>>> constitutes the "persisted state"). On the other hand, it's good to
> use
> >>>> well-established technologies rather than re-inventing things!
> >>>>>> An alternative ("pure brooklyn") approach could be to write
the
> bundle
> >>>> to persisted state; on rebind, we'd install + activate those bundles.
> >>>>>> ---
> >>>>>> For "catalogGroupId", I agree with Svet that in the initial
use-case
> >>>> this can be an implementation detail.
> >>>>>> It could be set as the bundle's symbolic name + version: everything
> >>> from
> >>>> the bundle should be deleted at once, along with the bundle.
> >>>>>> Longer term, I can see how exposing "catalogGroupId" to the
user
> could
> >>>> support more use-cases (e.g. for several catalog items from different
> >>>> bundles to work together). I don't think we should try to support that
> >>> yet.
> >>>>>> Aled
> >>>>>>
> >>>>>>
> >>>>>> On 19/12/2016 17:19, Geoff Macartney wrote:
> >>>>>>> hi Alex,
> >>>>>>>
> >>>>>>> this looks like a good feature to have, I shall look at
the PR as
> soon
> >>>> as I
> >>>>>>> can.
> >>>>>>>
> >>>>>>> The catalog.bom scanner feature was initially enabled by
default,
> but
> >>> we
> >>>>>>> had to
> >>>>>>> disable it because it turned out not to work properly with
rebind.
> I
> >>>> don't
> >>>>>>> think
> >>>>>>> it should be a lot of work to fix that but it hasn't been
something
> >>>> we've
> >>>>>>> got round
> >>>>>>> to yet.  This would be a great opportunity to look back
at that.
> >>>>>>>
> >>>>>>> Some random thoughts:
> >>>>>>>
> >>>>>>> re (C), if we are going to treat the zips as bundles, my
gut feel
> is
> >>>> that
> >>>>>>> we
> >>>>>>> should insist on a manifest and get the metadata from it.
 It
> doesn't
> >>>> feel
> >>>>>>> to me
> >>>>>>> like it makes much sense to allow a zip file without a MANIFEST.MF
> but
> >>>>>>> convey
> >>>>>>> the intended bundle metadata to Brooklyn via HTTP headers.
 And
> rather
> >>>> than
> >>>>>>> infer bundle metadata I think it's better to ask users to
be
> explicit
> >>>> about
> >>>>>>> what
> >>>>>>> their intentions are.  To make users lives easier, we could
> >>>>>>> add a command to br to generate the manifest (locally) with
correct
> >>>> syntax,
> >>>>>>> so that the manifest is in the right place, rather than
have br add
> >>> the
> >>>> data
> >>>>>>> to the "upload" request headers.
> >>>>>>>
> >>>>>>> re. (D) will be glad to have a look at it
> >>>>>>>
> >>>>>>> re. (E) it would certainly need to be optional - maybe keep
it as
> an
> >>>>>>> explicit
> >>>>>>> separate command ('upgrade'?)
> >>>>>>>
> >>>>>>> (F) it does seem like a lot of work but might be nice for
users who
> >>> are
> >>>> not
> >>>>>>> keen on command lines.
> >>>>>>>
> >>>>>>> G - I:  we'll definitely need to pay close attention to
persistence
> >>> and
> >>>>>>> rebind;
> >>>>>>> I wonder also about HA operation, are there any additional
> >>> implications?
> >>>>>>> (J) I think it would be good to treat all the files from
a jar,
> sorry
> >>>>>>> bundle,
> >>>>>>> as an atomic group - cleaner that way perhaps than allowing
> >>>> delete/update
> >>>>>>> of
> >>>>>>> individual entries from a bundle on a piecemeal basis. 
Rest
> support
> >>> on
> >>>>>>> delete
> >>>>>>> catalog could warn about related catalog entries being deleted
and
> ask
> >>>> for
> >>>>>>> a "--force" param to confirm.
> >>>>>>>
> >>>>>>> Geoff
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, 16 Dec 2016 at 15:24 Svetoslav Neykov <
> >>>>>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
> >>>>>>>
> >>>>>>>> +1
> >>>>>>>>
> >>>>>>>> Some thoughts:
> >>>>>>>>     * (A) add a utility class BundleMaker
> >>>>>>>>       Sounds very similar to
> >>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles
<
> >>>>>>>> https://ops4j1.jira.com/wiki/display/ops4j/Tinybundles>
> >>>>>>>>       Looking at the code it's much more focused on
zip files so I
> >>> guess
> >>>>>>>> there's no much overlap, but worth keeping in mind
> >>>>>>>>     * (C) accept bundle symbolic name and version
> >>>>>>>>       Why require them at all? Could infer them from
the
> catalog.bom
> >>> in
> >>>> some
> >>>>>>>> way - maybe require those properties to be in there.
If not
> present
> >>> are
> >>>>>>>> they really needed?
> >>>>>>>>     * (G) Bundles installed via this mechanism are not
persisted
> >>>> currently &
> >>>>>>>> (I) We persist the individual catalog items as YAML,
so we end up
> >>> with
> >>>> two
> >>>>>>>> records
> >>>>>>>>       Suggest marking the catalog items coming from
bundles as
> >>>>>>>> non-persistable. Then try to share the bundles between
HA nodes.
> >>> (Karaf
> >>>>>>>> Cellar?)
> >>>>>>>>     * (J) Introduce a catalogGroupId field on catalog
items;
> >>>>>>>>       Agree this could be useful and I like the idea
of deleting
> the
> >>>> bundle
> >>>>>>>> altogether with the catalog items. From user's perspective
I don't
> >>> see
> >>>> the
> >>>>>>>> need for an extra field (i.e. it's an implementation
detail).
> >>>>>>>>
> >>>>>>>> Svet.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 16.12.2016 г., at 12:50, Alex Heneveld <
> >>>>>>>> alex.heneveld@cloudsoftcorp.com> wrote:
> >>>>>>>>> Hi Brooklyners-
> >>>>>>>>>
> >>>>>>>>> In the code we currently have two routes for users
to install new
> >>>>>>>>> blueprints:
> >>>>>>>>>
> >>>>>>>>> (1) upload a catalog YAML file to /v1/catalog
> >>>>>>>>>
> >>>>>>>>> (2) install a bundle with catalog.bom in the root
> >>>>>>>>>
> >>>>>>>>> The feature (2) is disabled by default, but I'd
like to move
> towards
> >>>>>>>>> enabling it.  This will make it easier to create
nicely
> structured
> >>> BOM
> >>>>>>>>> files because scripts etc can be taken out of the
BOM, stored as
> >>>> files in
> >>>>>>>>> the same bundle.  (Because URLs of the form
> >>>>>>>>> `classpath://scripts/install.sh`  use the bundle's
classpath to
> >>>> resolve.)
> >>>>>>>>> As a first step in #485 [1] I do a few things:
> >>>>>>>>>
> >>>>>>>>> (A) add a utility class BundleMaker that lets us
create and
> modify
> >>>>>>>>> bundles/zips, to make it easier to do things we
might want to
> with
> >>>>>>>> bundles,
> >>>>>>>>> especially for testing
> >>>>>>>>>
> >>>>>>>>> (B) add an endpoint to the REST API which allows
uploading a
> bundle
> >>>> ZIP
> >>>>>>>>> (C) accept bundle symbolic name and version in that
REST API to
> >>>>>>>> facilitate
> >>>>>>>>> uploading non-bundle ZIPs where the OSGi MANIFEST.MF
is
> >>> automatically
> >>>>>>>>> generated
> >>>>>>>>>
> >>>>>>>>> With this PR, if you have a directory on your local
file system
> with
> >>>>>>>>> scripts and config files, and a BOM which refers
to them, you can
> >>> just
> >>>>>>>> ZIP
> >>>>>>>>> that up an upload it, specifying the bundle name
so that a YAML
> >>>> blueprint
> >>>>>>>>> author never needs to touch any java-isms.
> >>>>>>>>>
> >>>>>>>>> Where I see this going is a development workflow
where a user can
> >>> edit
> >>>>>>>>> files locally and upload the ZIP to have that installed,
and if
> they
> >>>> make
> >>>>>>>>> changes locally they can POST it again to have catalog
items
> updated
> >>>>>>>>> (because default version is a SNAPSHOT).  We could
also:
> >>>>>>>>>
> >>>>>>>>> (D) have `br catalog add ~/my/project/ --name my.project`
create
> a
> >>> ZIP
> >>>>>>>> and
> >>>>>>>>> POST it, with bundle name metadata, so essentially
the user's
> >>> process
> >>>> is
> >>>>>>>>> just to run that whenever they make a change
> >>>>>>>>>
> >>>>>>>>> (E) have a mechanism whereby deployed entities based
on an
> affected
> >>>>>>>>> blueprint are optionally migrated to the new code,
so if you've
> >>>> changed
> >>>>>>>> an
> >>>>>>>>> enricher the changes are picked up, or if say a
launch.sh script
> has
> >>>>>>>>> changed, a restart will run the new code
> >>>>>>>>>
> >>>>>>>>> The above are fairly straightforward programmatically
(although
> good
> >>>> user
> >>>>>>>>> interaction with (E) needs some thought).  So I
think we can
> pretty
> >>>>>>>> quickly
> >>>>>>>>> get to a much smoother dev workflow.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That's the highlight of this message.  You can jump
to the end,
> >>> unless
> >>>>>>>>> you're interested in some important but low level
details...
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I'm also tempted by:
> >>>>>>>>>
> >>>>>>>>> (F) Integration with web-based IDE and/or Brooklyn
reading and
> >>> writing
> >>>>>>>>> straight from GitHub -- but this seems like a lot
of work and I'm
> >>> not
> >>>>>>>>> convinced it's much better than (D) workflow-wise
> >>>>>>>>>
> >>>>>>>>> Before we can change (2) to be the default, or start
widely using
> >>> the
> >>>>>>>> POST
> >>>>>>>>> a ZIP feature, we need to sort out some issues to
do with
> >>> persistence
> >>>> and
> >>>>>>>>> reloading:
> >>>>>>>>>
> >>>>>>>>> (G) Bundles installed via this mechanism are not
persisted
> >>> currently,
> >>>> so
> >>>>>>>> if
> >>>>>>>>> you move to a different Brooklyn using the same
backing store,
> >>> you'll
> >>>>>>>> lose
> >>>>>>>>> those bundles
> >>>>>>>>>
> >>>>>>>>> (H) On rebind, bundles aren't always activated when
needed,
> meaning
> >>>> items
> >>>>>>>>> can't be loaded
> >>>>>>>>>
> >>>>>>>>> (I) We persist the individual catalog items as YAML,
so we end up
> >>> with
> >>>>>>>> two
> >>>>>>>>> records — the YAML from the catalog.bom in the
bundle, and the
> YAML
> >>>>>>>>> persisted for the item.  This isn't a problem per
se, but
> something
> >>> to
> >>>>>>>>> think about, and some sometimes surprising behaviour.
 In
> particular
> >>>> if
> >>>>>>>> you
> >>>>>>>>> delete the persisted YAML, the bundle is still there
so the item
> is
> >>> no
> >>>>>>>>> longer deleted after a full rebind.
> >>>>>>>>>
> >>>>>>>>> One idea which might be useful is:
> >>>>>>>>>
> >>>>>>>>> (J) Introduce a catalogGroupId field on catalog
items; this will
> do
> >>>> two
> >>>>>>>>> things:  if you try to delete an item with such
a record, you'll
> be
> >>>>>>>>> encouraged to delete all such items (maybe disallowed
to delete
> an
> >>>>>>>>> individual one), with the effect of deleting the
bundle if it
> comes
> >>>> from
> >>>>>>>> a
> >>>>>>>>> bundle; and when resolving types we search first
for items with
> the
> >>>> same
> >>>>>>>>> catalogGroupId (so that e.g. if I install MyCluster:1.0
and
> >>>> MyNode:1.0 in
> >>>>>>>>> the same group, the former can refer simply to "MyNode"
but if I
> >>>> install
> >>>>>>>> a
> >>>>>>>>> 2.0 version of that group, the 1.0 cluster still
loads the 1.0
> node
> >>> --
> >>>>>>>> this
> >>>>>>>>> has bitten people i the past)
> >>>>>>>>>
> >>>>>>>>> There is a related Brooklyn upgrade problem worth
mentioning,
> which
> >>>> the
> >>>>>>>>> above might help with, where:
> >>>>>>>>>
> >>>>>>>>> (K) If I migrate from Brooklyn 10 to 11 when it
comes out, I'll
> no
> >>>> longer
> >>>>>>>>> have certain entities that were at v10, since we
don't include
> >>> those;
> >>>> an
> >>>>>>>>> upgrade could include rules that certain groupIds
need to be
> >>> updated,
> >>>> or
> >>>>>>>> it
> >>>>>>>>> can search and attempt to automatically apply the
updates
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Quite a lot here and we don't need to solve it but
I wanted to:
> >>>>>>>>>
> >>>>>>>>> * Share the current thinking
> >>>>>>>>>
> >>>>>>>>> * Get opinions on the general dev workflow suggested
by (D)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks for feedback -- and if we like it help with
(D) would be
> >>>>>>>> appreciated!
> >>>>>>>>> Best
> >>>>>>>>> Alex
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [1] . https://github.com/apache/brooklyn-server/pull/485
> >>>
> >
>
> --

Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
http://www.cloudsoftcorp.com/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message