Return-Path: X-Original-To: apmail-ace-users-archive@minotaur.apache.org Delivered-To: apmail-ace-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AB69C1071D for ; Wed, 7 Aug 2013 14:49:17 +0000 (UTC) Received: (qmail 14014 invoked by uid 500); 7 Aug 2013 14:49:17 -0000 Delivered-To: apmail-ace-users-archive@ace.apache.org Received: (qmail 13956 invoked by uid 500); 7 Aug 2013 14:49:16 -0000 Mailing-List: contact users-help@ace.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@ace.apache.org Delivered-To: mailing list users@ace.apache.org Received: (qmail 13942 invoked by uid 99); 7 Aug 2013 14:49:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Aug 2013 14:49:15 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of bdekruijff@gmail.com designates 209.85.216.51 as permitted sender) Received: from [209.85.216.51] (HELO mail-qa0-f51.google.com) (209.85.216.51) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Aug 2013 14:49:09 +0000 Received: by mail-qa0-f51.google.com with SMTP id f11so1091132qae.17 for ; Wed, 07 Aug 2013 07:48:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=y+2myvfdlUJk+ifS79LnHDgW7+AgFNdibT0ZeYsKY2o=; b=VaGuzizPLdbUhb1wahoSqEegUXG4mbAo6wn7t4zGqYgkuL8HOiu5v6fHDm2VkfwfQ6 3QM9edEjlHpZUDpikfkkWvkfNZg84qK059ForgGndH8Ld0smowZ8wiaYqS9M26uoFPGB KE9LBHT+zzisH5unwXEBHdZfHlHgJJCyTNBQaSP56fDPazDhQmwPyomorfrmU2VJWY+1 uz9BAVlLKiQJjuIwUgxnD9JCw+C5le7PGEulR8xq05KZE3tvJvXdWiP4X9TX0OyXP87f qaQKxfq/9t9SbsesdHmcYMBrKZDn8DxqIyFh0ksUy+MEB9zUwyUu3dmZubf7wrdIcW8j s4HQ== MIME-Version: 1.0 X-Received: by 10.224.69.2 with SMTP id x2mr987127qai.77.1375886928958; Wed, 07 Aug 2013 07:48:48 -0700 (PDT) Received: by 10.49.87.41 with HTTP; Wed, 7 Aug 2013 07:48:48 -0700 (PDT) In-Reply-To: References: <2E54C939-BFD6-435D-BBEB-E3B6A92459B0@luminis.nl> <39532688-6CD3-45B4-A356-64EBEDC9932B@luminis.nl> Date: Wed, 7 Aug 2013 16:48:48 +0200 Message-ID: Subject: Re: Where does Ace server store everything? From: Bram de Kruijff To: users@ace.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org On Wed, Aug 7, 2013 at 3:30 PM, Matthew Bishop wrote: > We have two problems here: definition of artifact / feature / distributions > (AFD?), and ACE Server loadout of AFDs. > > *AFD definitions:* it makes a lot of sense to me to use the Maven project > file structure to declare which bundles that define the AFD components. > Maven is a well-known and capable at this task and has considerable tooling > for transitive dependency declaration and discovery. I can imagine the > following design: > > *Artifact* - maven jar artifact > *Feature* - POM that lists on artifacts. Builds out a pom > artifact. > *Distribution* - POM that lists on features. Builds out a > pom artifact. > > I know you might be thinking that maven isn't good enough for OSGi because > it doesn't understand the OSGi manifest headers, or OBR, or capabilities, > and you are right. However, at the end of the day, what server boffins want > to do is define every single bundle artifact in a way that can be > understood and managed in a familiar technology. Maven fits this bill very > well. > Well I did not see that one coming, but I can see the 'familiar technology' argument ;) However, IMHO we should not require the maven schema in ACE for this purpose. It is way too heavy (you would only be able to use a small subset of the schema) and limited in flexibility (associations are 1-1 based on the dependency graph). But, then again mapping pom to the ACE model should be pretty simple. That is, once you figured out the mapping. Simple example mapping; An ACE bundle artifact has a name, description, bsn, version, url (could even point to maven central), and an arbitrary number of tags. So a rough mapping (excuse my rusty maven vocabulary); name = ${pom.name} description = ${pom.description} bsn = ${pom.groupId}.${pom.artifactId} version = ${pom.version} url = ${some.repourl} tags = ${pom.properties} //?? An ACE feature basically only has a name/identitfier So simply; name = ${pom.name} The ACE artifact2FeatureAssociations derived from the feature pom dependencies would be something like for each dependency; leftEndpoint = &((Bundle-SymbolicName=${dependency.groupId}.${dependency.artifactId})(version=${dependency.version})) rightEndpoint = (name=${pom.name}) And the same pattern applies for distribution and feature2Distribution. > Most of the tech in OSGi that you know and love at dev time, like BND > Tools, is still applicable. All I am suggesting is that the AFD definitions > use the maven POM files, not necessarily the 'mvn' runtimes to generate the > actual bundle jars. Using maven like this will populate the maven > repositories with AFD definitions that will be useful for loadout. > > > *Server Loadout:* The ACE Server (not the targets) would be able to consume > the various Feature and Distribution POMs. It would be configured to look > to maven repositories (like maven central) to load in the bundles as > artifacts. The versioning of the AFDs would be managed by the maven version > numbering itself. Again, very familiar, trusted tech. Easy to roll back if > all hell breaks loose. Easy to write scripts to automate. > Again I would not want to make the maven shema the default/requirement for ACE and moving from some poms to a more neutral/generic representation or even directly to ACE would probably best be handled by maven itself through a plugin are custom packaging. Specifically when you are talking about transitive dependencies this will be the easiest by far. That small piece of maven tooling wil then be able to talk to the ACE importer of even REST/client apis directly. So given you multi-module project for example; $ mvn ace:export-model > ace-deployment-descriptor.xml $ import curl ace-deployment-descriptor.xml (using a future generic ACE importer) OR $ mvn ace:deploy-model -D serverEndpoint=http://localhost:8080/ace // talk to the REST api directly Actually I wrote the latter using that Amdatu client for a former employer to provide continuous deployment from a build, but that only covered artifacts. > The mountain I am really trying to climb here is to provide a distribution > package for OSGi applications, a package that can be understood and > examined in existing and well-integrated deployment technology. > Well if a maven model serves you purpose I think the above approach could work well. As long as you are aware that you will still need OSGi aware tooling to answer question as "will this distribution package actually resolve in an OSGi runtime". Anyway, I would be happy to collaborate on coming up with a generic ACE model/importer that would serve your purpose. Regards, Bram > > On Tue, Aug 6, 2013 at 11:25 PM, Bram de Kruijff wrote: > >> Hi Matthew, >> >> On Wed, Aug 7, 2013 at 1:06 AM, Matthew Bishop >> wrote: >> > Hmm, this looks like a lot like nimble's scripting. >> > >> > Can ACE generate a deployment script that loads up a set of >> > artifacts/features/etc? It would go a long ways towards the >> checkin-ability >> > of what is manipulated in the UI / API. >> > >> >> Generating scripts does not feel like the way to go for me. In this >> example script the artifacts come from a declarative model (the R5 >> repository representation), but obviously the association, feature and >> distribution do not. As you already state this information should be >> easy enough to capture in a declarative manner. You would need to do >> this anyway if you would want to generate a script so why not support >> that from ACE directly? I am curious to hear what your idea of the >> 'input format' to a generic import would ideally be. >> >> As a reference, some time ago in wrote an Amdatu ACE Client [0] >> (source repo not online) that talks to the REST API and is also >> capable of doing basic import/export based on a simple JSON >> representation of the custom java model that is in there. However, >> that client does not talk to the OBR, so no uploads. In fact the ACE >> repository uses a similar XML representation for disk storage just >> listing the different entities. Also something we could leverage. >> >> Another idea that I am toying with is to leverage the OSGi Resource >> model of the R5 Repository which would allow us to store all this >> information in a R5 repository. Features and distributions could be >> modeled as specific resource types, providing meta-data >> (attributes/tags) through capabilities and modelling association >> through requirements. This approach would allow capturing all >> information in one single source and standard representation. >> >> So I think there are options :) Again I would like to hear what your >> preferred approach would be an maybe we can collaborate on coming up >> with a generic solution. >> >> Best Regards, >> Bram >> >> [0] http://repository.amdatu.org/release-util/org.amdatu.ace.client/ >> >> > >> > On Tue, Aug 6, 2013 at 2:34 PM, Marcel Offermans < >> > marcel.offermans@luminis.nl> wrote: >> > >> >> Hello Matthew, >> >> >> >> On Aug 6, 2013, at 23:10 PM, Matthew Bishop >> wrote: >> >> >> >> > On Tue, Aug 6, 2013 at 1:58 PM, Marcel Offermans < >> >> > marcel.offermans@luminis.nl> wrote: >> >> > >> >> >> To give you another example, we recently automated our continuous >> build >> >> >> system so for each succesful build it takes the artifacts that come >> out >> >> of >> >> >> that build and (through a GoGo Shell script) upload them to a >> repository >> >> >> and then create all the metadata in ACE. The end result is that we >> can >> >> >> automatically deploy such code directly to our QA and demo >> environments, >> >> >> completely automated. >> >> > >> >> > Is this gogo script checked into ACE somewhere? It is definitely an >> >> > interesting part of the story. I am working my backwards from the end >> >> > deployment story to the developer, as we did it developer-first and >> ended >> >> > up with pax and maven on the production server. Not cool. >> >> >> >> It is not, because it is specific to our environment. The GoGo commands >> >> are something we only very recently added, so they're not documented >> well >> >> yet. >> >> >> >> However, I have adapted the script we use a bit (taking out the specific >> >> bits). It probably requires some more context to be fully understood, >> and >> >> probably Bram can answer some more detailed questions about it (he wrote >> >> the script). Basically it takes the artifacts from a build, checks if >> they >> >> are really different from the ones we already deployed the previous >> time, >> >> uploads all changed artifacts and configures ACE. We run this script >> with a >> >> client jar from the ACE distribution, and configure it as the startup >> >> script. As soon as the script is done, we shutdown this client again. >> >> >> >> >> >> # >> >> # Continuous deployment gogo script >> >> # >> >> >> >> # environment variables we import here >> >> targetrepourl = "$DEPLOYMENT_TARGETOBR_URL" >> >> releaserepourl = "$DEPLOYMENT_RELEASEOBR_URL" >> >> autoconfurl = "$DEPLOYMENT_AUTOCONF_URL" >> >> >> >> sourceindex = (repo:index /tmp/ace-cdeploy) >> >> sourcerepo = (repo:repo R5 $sourceindex) >> >> targetrepo = (repo:repo OBR $targetrepourl) >> >> releaserepo = (repo:repo OBR $releaserepourl) >> >> >> >> echo "Deploying release resources" >> >> deployed = repo:cd $releaserepo $sourcerepo $targetrepo >> >> >> >> echo "Ensuring autoconf resource" >> >> autoconf = (coll:first ($targetrepo find >> >> "(osgi.identity=org.apache.felix.deployment.rp.autoconf)")) >> >> if { $autoconf } { echo " -> exists" } \ >> >> { >> >> repo:d $targetrepo $autoconfurl >> >> autoconf = (coll:first ($targetrepo find >> >> "(osgi.identity=org.apache.felix.deployment.rp.autoconf)")) >> >> echo " -> deployed" >> >> } >> >> >> >> echo "Opening client workspace" >> >> workspace = (ace:cw) >> >> >> >> echo "Ensuring cdtarget exists" >> >> target = (coll:first ($workspace lt "(id=cdtarget)")) >> >> if { $target } { echo " -> exists" } \ >> >> { >> >> $workspace ct "cdtarget"; >> >> target = (coll:first ($workspace lt "(id=cdtarget)")) >> >> echo " -> created" >> >> } >> >> >> >> echo "Ensuring cdtarget is registered" >> >> if { $target isRegistered } { } { $target register } >> >> >> >> echo "Ensuring cdtarget is !autoapprove" >> >> $target setAutoApprove false >> >> >> >> # here we set some specific properties used by our configurations >> >> (anonymized) >> >> $target addTag "domain" "foo.com" >> >> $target addTag "ip-address" "127.0.0.1" >> >> $target addTag "timezone" "Europe/Amsterdam" >> >> >> >> echo "Ensuring cdfeature exists" >> >> if { (coll:first ($workspace lf "(name=cdfeature)")) } { echo " -> >> exists" >> >> } { $workspace cf "cdfeature"; echo " -> created" } >> >> >> >> echo "Ensuring cddistribution exists" >> >> if { (coll:first ($workspace ld "(name=cddistribution)")) } { echo " -> >> >> exists" } { $workspace cd "cddistribution"; echo " -> created" } >> >> >> >> echo "Ensuring f2d association" >> >> if { (coll:first ($workspace lf2d "(leftEndpoint=*name=cdfeature*)")) } >> { >> >> echo " -> exists" } { $workspace cf2d "(name=cdfeature)" >> >> "(name=cddistribution)" "1" "1"; echo " -> created" } >> >> >> >> echo "Ensuring a2f association" >> >> if { (coll:first ($workspace la2f "(rightEndpoint=*name=cdfeature*)")) >> } { >> >> echo " -> exists" } { $workspace ca2f "(cdartifact=true)" >> >> "(name=cdfeature)" "10000" "1"; echo " -> created" } >> >> >> >> echo "Ensuring d2t association" >> >> if { (coll:first ($workspace ld2t >> "(leftEndpoint=*name=cddistribution*)")) >> >> } { echo " -> exists" } \ >> >> { $workspace cd2t "(name=cddistribution)" "(id=*)" "1" "1000"; echo " >> -> >> >> created" } >> >> >> >> echo "Ensuring autoconf artifact" >> >> if { (coll:first ($workspace lrp)) } { echo " -> exists" } \ >> >> { >> >> identity = $autoconf getIdentity >> >> version = $autoconf getVersion >> >> name = "$identity - $version" >> >> url = $autoconf getUrl >> >> mimetype = $autoconf getMimetype >> >> $workspace ca [ artifactName="$name" url="$url" mimetype="$mimetype" >> >> Bundle-SymbolicName="$identity" Bundle-Version="$version" >> >> Deployment-ProvidesResourceProcessor="org.osgi.deployment.rp.autoconf" ] >> >> echo " -> created" >> >> } >> >> >> >> echo "Purging old artifacts" >> >> artifacts = ($workspace la) >> >> each $artifacts { >> >> $workspace da $it >> >> } >> >> >> >> echo "Creating new artifacts" >> >> each $deployed { >> >> identity = $it getIdentity >> >> version = $it getVersion >> >> name = "$identity - $version" >> >> url = $it getUrl >> >> mimetype = $it getMimetype >> >> if { $mimetype equals "application/xml:osgi-autoconf" } \ >> >> { $workspace ca [ artifactName="$name" url="$url" mimetype="$mimetype" >> >> filename="$name" processorPid="org.osgi.deployment.rp.autoconf" ] [ >> >> cdartifact="true" ] } \ >> >> { $workspace ca [ artifactName="$name" url="$url" mimetype="$mimetype" >> >> Bundle-SymbolicName="$identity" Bundle-Version="$version" ] [ >> >> cdartifact="true"] } >> >> } >> >> >> >> echo "Ensuring approval on target" >> >> if { $target needsApprove } { echo " approving"; $target approve } { >> echo >> >> " no changes" } >> >> >> >> echo "Committing workspace" >> >> $workspace commit >> >> >> >> # Let events settle >> >> misc:sleep 2000 >> >> ace:rw $workspace >> >> >> >> # Let events settle >> >> misc:shutdown 2000 >> >> >> >> >> >> Greetings, Marcel >> >> >> >> >>