Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1D104200CAD for ; Wed, 28 Jun 2017 13:31:36 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 1B962160BF7; Wed, 28 Jun 2017 11:31:36 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3A01C160BE8 for ; Wed, 28 Jun 2017 13:31:35 +0200 (CEST) Received: (qmail 11819 invoked by uid 500); 28 Jun 2017 11:31:34 -0000 Mailing-List: contact dev-help@brooklyn.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@brooklyn.apache.org Delivered-To: mailing list dev@brooklyn.apache.org Received: (qmail 11808 invoked by uid 99); 28 Jun 2017 11:31:34 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jun 2017 11:31:34 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 1780EDFB0D; Wed, 28 Jun 2017 11:31:34 +0000 (UTC) From: ahgittin To: dev@brooklyn.apache.org Reply-To: dev@brooklyn.apache.org Message-ID: Subject: [GitHub] brooklyn-server pull request #746: [WIP] Auto wrap YAML BOMs as catalog bund... Content-Type: text/plain Date: Wed, 28 Jun 2017 11:31:34 +0000 (UTC) archived-at: Wed, 28 Jun 2017 11:31:36 -0000 GitHub user ahgittin opened a pull request: https://github.com/apache/brooklyn-server/pull/746 [WIP] Auto wrap YAML BOMs as catalog bundles Wraps pure-YAML BOMs uploaded to catalog in a bundle, creating a name and version if needed. This allows us to see all items added as a unit. This will allow us to prefer loading co-defined items (though that's not fully done, still relies on preferring search path when doing item lookup), and to edit or remove that item as a set (crucially we now keep the BOM, previously we discarded it!). Docs changed to reflect this and explain differences in another PR. The basic wrapping was pretty simple but a surprising number of subtleties emerged which this fixes: * java scanning doesn't work in OSGi; docs and errors make this clear * impacts replacement semantics (now it does a hash on the bundle to make most things the same, though a re-post of a non-SNAPSHOT BOM that doesn't declare a bundle will now fail because the random bundle name is stored on the item as containingBundle and is different on a re-post; and needed extra work to remove leaked anonymous bundles) * had to change the way we set up the libraries / search path, to exclude this "bundle" (mainly so tests are unchanged, but also improves efficiency for simple BOMs; also the code is cleaner now) And there is one thing that emerged that I *haven't* fixed: on rebind it became vital to handle the situation where dependent bundles might not be loaded before the depender bundle; see failing test`testLongReferenceSequence`. Not entirely sure how to do this /cc @neykov @geomacy @aledsage. Depends on #743. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ahgittin/brooklyn-server osgi-auto-wrap-yaml Alternatively you can review and apply these changes as the patch at: https://github.com/apache/brooklyn-server/pull/746.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #746 ---- commit 36ffdeeca0ac5290727b3bf5d2a7267664e9638a Author: Alex Heneveld Date: 2017-06-16T08:49:44Z remove REST reference to removed method commit 3860c3498d63a9a9d880480363c7947599d40a7c Author: Alex Heneveld Date: 2017-06-16T10:53:33Z new utilities for registered type (catalog item id) naming syntax commit 607fc853aa395915e15d527405b7a7a8a6e97b48 Author: Alex Heneveld Date: 2017-06-16T12:24:28Z move many things to use RegisteredTypeNaming rather than hokey `looksLikeVersionedId` commit eb08f8ec0d65b59bb2c4201a98632aa070ddac97 Author: Alex Heneveld Date: 2017-06-16T12:38:45Z notes on related methods for looking up catalog items flexibly commit c9d03c93e35eb5417ca6168207f4b0ddc3de9e7c Author: Alex Heneveld Date: 2017-06-16T12:46:20Z remove deprecated CAMP ServiceTypeResolver changed a while ago to EntitySpecResolver; removes references to catalog items and hokey lookup commit db77f4e14d07fe9e23fb464fb23302afc1fba2f0 Author: Alex Heneveld Date: 2017-06-20T08:37:29Z fix tests that relied on removed deprecated behaviour commit dc4942a345dc2fbd5125a34131addb632e634fe8 Author: Alex Heneveld Date: 2017-06-20T08:51:42Z Merge branch 'master' into version-id commit 5b2cc48685ee606207257e74fd8a61f13f876a00 Author: Alex Heneveld Date: 2017-06-20T15:48:25Z prefer versions which comply with recommendations, heuristics elsewhere expand tests. some obscure ordering items are different now. commit cc703928e1a2bd9c97795acab9a747a3733eb4a2 Author: Alex Heneveld Date: 2017-06-20T15:49:57Z tidy version comparator, removing old code commit 45cb10a57af9e4aba0e7d624f1af6bfb25e5cee2 Author: Alex Heneveld Date: 2017-06-21T16:04:04Z add lots of text explaining BrooklynVersionSyntax, update comparator, tests, and usages commit ed98e27e934ddf54d6c1c80b4a61afaab1225e47 Author: Alex Heneveld Date: 2017-06-21T17:26:43Z add routines to convert brooklyn/osgi versions and bundle finder supports either, and does the conversion for you commit b46188e60e62247f8ffbe09c11acf37aaee69fad Author: Alex Heneveld Date: 2017-06-21T17:33:06Z This closes #737 commit 5a7b64a751abdfbd3d75f20232a2c2a4dbb28feb Author: Alex Heneveld Date: 2017-06-21T17:33:14Z Merge branch 'version-new-rules' into version-new-rules-prep commit c5c5afc385341620f199737ddae3d2efaeb3a0a4 Author: Alex Heneveld Date: 2017-06-22T13:11:22Z This closes #737 commit d8c03c82a14f2d82c4280067322604bc0982a976 Author: Alex Heneveld Date: 2017-06-22T13:11:38Z This closes #740 commit 89fb13955eef518f53e3c3330a20191c89a3cf2e Author: Alex Heneveld Date: 2017-06-22T12:17:32Z record underlying supplied version in VersionedName support for persistence, and test commit 701a1d7fb70203fd6a0503c369b16674972344ac Author: Alex Heneveld Date: 2017-06-22T12:18:28Z update OsgiBundleWithUrl to act like versioned name, and uses of VersionedName OSGi versions are taken at the very last moment, allowing us to preserve brooklyn-recommended syntax throughout nearly all our code; and lookups/matches in OSGi space compare the OSGi-target commit 97b50e4c304f0886f96a7cd503f1d634d1fbd9c4 Author: Alex Heneveld Date: 2017-06-26T11:49:38Z add routine to create new temp bundle commit d7975d1414269f0f14617d3987170eba729c9c7d Author: Alex Heneveld Date: 2017-06-23T09:53:51Z implicitly wrap yaml in bundles when uploading to catalog creates a new bundle when given yaml in catalog; bundle name and version now recommended in the BOM. scan-java option tweaked but in a way that is consistent with the past and sensible in the new world. auto-wrapped bundles are identified with a header so we can simplify their handling in some cases (scanning, search paths). some things clumsy and need fixed: * uploading a different BOM (or bundle) at the same version says "ignoring because identical"; osgi identicality check should look at contents * auto-wrapped bundles are added to the library search path (no need for this as the bundles are empty) * failed installations keep the bundle installed, blocking subsequent installs; probably don't install unless forced? * warn if different version declared in BOM some things clumsy but we could live with: * containing bundles are added as libraries by munging the yaml; now we have a record to that so can do a bit better than munge yaml * if bundle has no name, a random one is chosen (probably deprecated this behaviour); if you re-submit we don't know it's the same bundle so we don't detect it's a bundle update; this means: * uploading the exact same non-snapshot BOM twice will fail the second time saying the items are different (because containing bundle is different) * uploading an updated item in a snapshot BOM will correctly replace, but the old bundle will still be around (just masked; though we can fix that) commit 2165778a354fa4221eec0e8ba03a96ef5bef92a3 Author: Alex Heneveld Date: 2017-06-27T13:23:34Z fix bundles set on scanning, and aborted work to scan osgi bundles scanning for OSGi can be made to work in some cases if we have the jar, but it won't work after rebind, or even in some cases if persisting (race deleting the JAR), plus there are likely classloading issues; note OSGi deliberately doesn't give us the JAR. java scanning is discouraged by OSGi; it wants us to explicitly list things e.g. in the catalog.bom, which is fine. commit e7dc875af19138261f95994c5da08a2187ee64c7 Author: Alex Heneveld Date: 2017-06-27T14:38:15Z Add checksum to managed bundle so we can tell if it changes and fail accordingly This permits us to allow the same BOM YAML to be added multiple times (whether or not bundles are declared), except where it's a non-snapshot BOM that doesn't declare a bundle, because the containing bundle will be different there. This also fixes a bug where containingBundle wasn't persisted; not a big problem as we re-read bundle items, but cleaner to have it working. commit f16bda9310469ec35f4de8f3a084746abfe08a44 Author: Alex Heneveld Date: 2017-06-27T15:28:33Z support force when wrapping a BOM new batch of tests around catalog versioning with OSGi commit 8385f133273116ec5df628325a7d71ad29d4dcb8 Author: Alex Heneveld Date: 2017-06-28T08:12:19Z repair tests where semantics changed, new tests including explicit test for new load order problems: * during rebind osgi bundles can have BOM loaded in any order, but they expect their deps to be available two remaining problems: * wrapper library is added (changing assertions about numbers) * added containing library changes YAML (changing assertions about comments) commit e32d74d15359152c9b1900db2ee0a7958b15c29b Author: Alex Heneveld Date: 2017-06-28T08:28:13Z comments on BOM scanning commit dc43f03fe2fa269e49a7865eafb491430f4e7e32 Author: Alex Heneveld Date: 2017-06-28T08:46:13Z add library bundles directly, without munging yaml, and only if not wrapper commit 480a0c76ae434a7d05455ec0de63e33b5c05aebc Author: Alex Heneveld Date: 2017-06-28T09:43:17Z Delete wrapper bundles who have no types installed. There is no point to them (they aren't OSGi). Fixes bug where uploading the same BOM multiple times, could cause bundle leakage. And tidy logging, more tests. commit 21e93d27bed7efd3d8f115c8fc2234b39360ee3a Author: Alex Heneveld Date: 2017-06-28T10:28:24Z restore test bundle counts, and resolve library URLs on install commit 0236dbcd5809827c0f2548561c1ddfa5cc26d7a3 Author: Alex Heneveld Date: 2017-06-28T10:45:28Z address comments on PR in #737 commit 14aa7486ffbc07f57acf67836e5466205cb3b524 Author: Alex Heneveld Date: 2017-06-28T10:52:35Z Merge branch 'version-id' into version-new-rules-prep commit ee853f5806f3d2d89f2c76521ec236695f61f80d Author: Alex Heneveld Date: 2017-06-28T11:24:36Z tidy esp javadoc and regexes, as per PR comments ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastructure@apache.org or file a JIRA ticket with INFRA. ---