ace-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sibla Wilfried <Wilfried.Si...@bosch-si.com>
Subject AW: Where does Ace server store everything?
Date Wed, 07 Aug 2013 16:28:26 GMT
Hi to all

In the last days I implemented a utility to derive a ACE feature from a maven pom project.
I build a simple pom project with all directly required artifacts in the dependency list.
But I had to enrich the dependencies with a list of exclusions, because some included projects
were only repackaging non-OSGi artifacts as OSGi bundles.
But e.g. if those feature references quite high number of artifacts, and all artifacts are
already available within a maven repo, this works quite good.

@Bram: I could possibly provide this solution in the next months...


Greetings
Wilfried


-----Urspr√ľngliche Nachricht-----
Von: Bram de Kruijff [mailto:bdekruijff@gmail.com] 
Gesendet: Mittwoch, 7. August 2013 16:49
An: users@ace.apache.org
Betreff: Re: Where does Ace server store everything?

On Wed, Aug 7, 2013 at 3:30 PM, Matthew Bishop <matt@thebishops.org> wrote:
> We have two problems here: definition of artifact / feature / 
> distributions (AFD?), and ACE Server loadout of AFDs.
>
> *AFD definitions:* it makes a lot of sense to me to use the Maven 
> project file structure to declare which bundles that define the AFD components.
> Maven is a well-known and capable at this task and has considerable 
> tooling for transitive dependency declaration and discovery. I can 
> imagine the following design:
>
> *Artifact* - maven jar artifact
> *Feature* - POM that lists <dependencies> on artifacts. Builds out a 
> pom artifact.
> *Distribution* - POM that lists <dependencies> on features. Builds out 
> a pom artifact.
>
> I know you might be thinking that maven isn't good enough for OSGi 
> because it doesn't understand the OSGi manifest headers, or OBR, or 
> capabilities, and you are right. However, at the end of the day, what 
> server boffins want to do is define every single bundle artifact in a 
> way that can be understood and managed in a familiar technology. Maven 
> fits this bill very well.
>

Well I did not see that one coming, but I can see the 'familiar technology' argument ;) However,
IMHO we should not require the maven schema in ACE for this purpose. It is way too heavy (you
would only be able to use a small subset of the schema) and limited in flexibility (associations
are 1-1 based on the dependency graph). But, then again mapping pom to the ACE model should
be pretty simple. That is, once you figured out the mapping.

Simple example mapping;

An ACE bundle artifact has a name, description, bsn, version, url (could even point to maven
central), and an arbitrary number of tags.
So a rough mapping (excuse my rusty maven vocabulary);

name = ${pom.name}
description = ${pom.description}
bsn = ${pom.groupId}.${pom.artifactId}
version = ${pom.version}
url = ${some.repourl}
tags = ${pom.properties} //??

An ACE feature basically only has a name/identitfier So simply;

name = ${pom.name}

The ACE artifact2FeatureAssociations derived from the feature pom dependencies would be something
like for each dependency;

leftEndpoint = &((Bundle-SymbolicName=${dependency.groupId}.${dependency.artifactId})(version=${dependency.version}))
rightEndpoint = (name=${pom.name})

And the same pattern applies for distribution and feature2Distribution.


> Most of the tech in OSGi that you know and love at dev time, like BND 
> Tools, is still applicable. All I am suggesting is that the AFD 
> definitions use the maven POM files, not necessarily the 'mvn' 
> runtimes to generate the actual bundle jars. Using maven like this 
> will populate the maven repositories with AFD definitions that will be useful for loadout.
>
>
> *Server Loadout:* The ACE Server (not the targets) would be able to 
> consume the various Feature and Distribution POMs. It would be 
> configured to look to maven repositories (like maven central) to load 
> in the bundles as artifacts. The versioning of the AFDs would be 
> managed by the maven version numbering itself. Again, very familiar, 
> trusted tech. Easy to roll back if all hell breaks loose. Easy to write scripts to automate.
>

Again I would not want to make the maven shema the default/requirement for ACE and moving
from some poms to a more neutral/generic representation or even directly to ACE would probably
best be handled by maven itself through a plugin are custom packaging. Specifically when you
are talking about transitive dependencies this will be the easiest by far.

That small piece of maven tooling wil then be able to talk to the ACE importer of even REST/client
apis directly. So given you multi-module project for example;

$ mvn ace:export-model > ace-deployment-descriptor.xml $ import curl ace-deployment-descriptor.xml
 (using a future generic ACE importer)

  OR

$ mvn ace:deploy-model -D serverEndpoint=http://localhost:8080/ace // talk to the REST api
directly

Actually I wrote the latter using that Amdatu client for a former employer to provide continuous
deployment from a build, but that only covered artifacts.


> The mountain I am really trying to climb here is to provide a 
> distribution package for OSGi applications, a package that can be 
> understood and examined in existing and well-integrated deployment technology.
>

Well if a maven model serves you purpose I think the above approach could work well. As long
as you are aware that you will still need OSGi aware tooling to answer question as "will this
distribution package actually resolve in an OSGi runtime".

Anyway, I would be happy to collaborate on coming up with a generic ACE model/importer that
would serve your purpose.

Regards,
Bram

>
> On Tue, Aug 6, 2013 at 11:25 PM, Bram de Kruijff <bdekruijff@gmail.com>wrote:
>
>> Hi Matthew,
>>
>> On Wed, Aug 7, 2013 at 1:06 AM, Matthew Bishop <matt@thebishops.org>
>> wrote:
>> > Hmm, this looks like a lot like nimble's scripting.
>> >
>> > Can ACE generate a deployment script that loads up a set of 
>> > artifacts/features/etc? It would go a long ways towards the
>> checkin-ability
>> > of what is manipulated in the UI / API.
>> >
>>
>> Generating scripts does not feel like the way to go for me. In this 
>> example script the artifacts come from a declarative model (the R5 
>> repository representation), but obviously the association, feature 
>> and distribution do not. As you already state this information should 
>> be easy enough to capture in a declarative manner. You would need to 
>> do this anyway if you would want to generate a script so why not 
>> support that from ACE directly? I am curious to hear what your idea 
>> of the 'input format' to a generic import would ideally be.
>>
>> As a reference, some time ago in wrote an Amdatu ACE Client [0] 
>> (source repo not online) that talks to the REST API and is also 
>> capable of doing basic import/export based on a simple JSON 
>> representation of the custom java model that is in there. However, 
>> that client does not talk to the OBR, so no uploads. In fact the ACE 
>> repository uses a similar XML representation for disk storage just 
>> listing the different entities. Also something we could leverage.
>>
>> Another idea that I am toying with is to leverage the OSGi Resource 
>> model of the R5 Repository which would allow us to store all this 
>> information in a R5 repository. Features and distributions could be 
>> modeled as specific resource types, providing meta-data
>> (attributes/tags) through capabilities and modelling association 
>> through requirements. This approach would allow capturing all 
>> information in one single source and standard representation.
>>
>> So I think there are options :) Again I would like to hear what your 
>> preferred approach would be an maybe we can collaborate on coming up 
>> with a generic solution.
>>
>> Best Regards,
>> Bram
>>
>> [0] http://repository.amdatu.org/release-util/org.amdatu.ace.client/
>>
>> >
>> > On Tue, Aug 6, 2013 at 2:34 PM, Marcel Offermans < 
>> > marcel.offermans@luminis.nl> wrote:
>> >
>> >> Hello Matthew,
>> >>
>> >> On Aug 6, 2013, at 23:10 PM, Matthew Bishop <matt@thebishops.org>
>> wrote:
>> >>
>> >> > On Tue, Aug 6, 2013 at 1:58 PM, Marcel Offermans < 
>> >> > marcel.offermans@luminis.nl> wrote:
>> >> >
>> >> >> To give you another example, we recently automated our 
>> >> >> continuous
>> build
>> >> >> system so for each succesful build it takes the artifacts that

>> >> >> come
>> out
>> >> of
>> >> >> that build and (through a GoGo Shell script) upload them to a
>> repository
>> >> >> and then create all the metadata in ACE. The end result is that

>> >> >> we
>> can
>> >> >> automatically deploy such code directly to our QA and demo
>> environments,
>> >> >> completely automated.
>> >> >
>> >> > Is this gogo script checked into ACE somewhere? It is definitely 
>> >> > an interesting part of the story. I am working my backwards from 
>> >> > the end deployment story to the developer, as we did it 
>> >> > developer-first and
>> ended
>> >> > up with pax and maven on the production server. Not cool.
>> >>
>> >> It is not, because it is specific to our environment. The GoGo 
>> >> commands are something we only very recently added, so they're not 
>> >> documented
>> well
>> >> yet.
>> >>
>> >> However, I have adapted the script we use a bit (taking out the 
>> >> specific bits). It probably requires some more context to be fully 
>> >> understood,
>> and
>> >> probably Bram can answer some more detailed questions about it (he 
>> >> wrote the script). Basically it takes the artifacts from a build, 
>> >> checks if
>> they
>> >> are really different from the ones we already deployed the 
>> >> previous
>> time,
>> >> uploads all changed artifacts and configures ACE. We run this 
>> >> script
>> with a
>> >> client jar from the ACE distribution, and configure it as the 
>> >> startup script. As soon as the script is done, we shutdown this client again.
>> >>
>> >>
>> >> #
>> >> # Continuous deployment gogo script #
>> >>
>> >> # environment variables we import here targetrepourl = 
>> >> "$DEPLOYMENT_TARGETOBR_URL"
>> >> releaserepourl = "$DEPLOYMENT_RELEASEOBR_URL"
>> >> autoconfurl = "$DEPLOYMENT_AUTOCONF_URL"
>> >>
>> >> sourceindex = (repo:index /tmp/ace-cdeploy) sourcerepo  = 
>> >> (repo:repo R5 $sourceindex) targetrepo = (repo:repo OBR 
>> >> $targetrepourl) releaserepo = (repo:repo OBR $releaserepourl)
>> >>
>> >> echo "Deploying release resources"
>> >> deployed = repo:cd $releaserepo $sourcerepo $targetrepo
>> >>
>> >> echo "Ensuring autoconf resource"
>> >> autoconf = (coll:first ($targetrepo find
>> >> "(osgi.identity=org.apache.felix.deployment.rp.autoconf)"))
>> >> if { $autoconf } { echo " -> exists" } \ {
>> >>   repo:d $targetrepo $autoconfurl
>> >>   autoconf = (coll:first ($targetrepo find
>> >> "(osgi.identity=org.apache.felix.deployment.rp.autoconf)"))
>> >>   echo " -> deployed"
>> >> }
>> >>
>> >> echo "Opening client workspace"
>> >> workspace = (ace:cw)
>> >>
>> >> echo "Ensuring cdtarget exists"
>> >> target = (coll:first ($workspace lt "(id=cdtarget)")) if { $target 
>> >> } { echo " -> exists" } \ {
>> >>   $workspace ct "cdtarget";
>> >>   target = (coll:first ($workspace lt "(id=cdtarget)"))
>> >>   echo " -> created"
>> >> }
>> >>
>> >> echo "Ensuring cdtarget is registered"
>> >> if { $target isRegistered } { } { $target register }
>> >>
>> >> echo "Ensuring cdtarget is !autoapprove"
>> >> $target setAutoApprove false
>> >>
>> >> # here we set some specific properties used by our configurations
>> >> (anonymized)
>> >> $target addTag "domain" "foo.com"
>> >> $target addTag "ip-address" "127.0.0.1"
>> >> $target addTag "timezone" "Europe/Amsterdam"
>> >>
>> >> echo "Ensuring cdfeature exists"
>> >> if { (coll:first ($workspace lf "(name=cdfeature)")) } { echo " ->
>> exists"
>> >> } { $workspace cf "cdfeature"; echo " -> created" }
>> >>
>> >> echo "Ensuring cddistribution exists"
>> >> if { (coll:first ($workspace ld "(name=cddistribution)")) } { echo 
>> >> " -> exists" } { $workspace cd "cddistribution"; echo " -> 
>> >> created" }
>> >>
>> >> echo "Ensuring f2d association"
>> >> if { (coll:first ($workspace lf2d 
>> >> "(leftEndpoint=*name=cdfeature*)")) }
>> {
>> >> echo " -> exists" } { $workspace cf2d "(name=cdfeature)"
>> >> "(name=cddistribution)" "1" "1"; echo " -> created" }
>> >>
>> >> echo "Ensuring a2f association"
>> >> if { (coll:first ($workspace la2f 
>> >> "(rightEndpoint=*name=cdfeature*)"))
>> } {
>> >> echo " -> exists" } { $workspace ca2f "(cdartifact=true)"
>> >> "(name=cdfeature)" "10000" "1"; echo " -> created" }
>> >>
>> >> echo "Ensuring d2t association"
>> >> if { (coll:first ($workspace ld2t
>> "(leftEndpoint=*name=cddistribution*)"))
>> >> } { echo " -> exists" } \
>> >>   { $workspace cd2t "(name=cddistribution)" "(id=*)" "1" "1000"; echo "
>> ->
>> >> created" }
>> >>
>> >> echo "Ensuring autoconf artifact"
>> >> if { (coll:first ($workspace lrp)) } { echo " -> exists" } \ {
>> >>   identity = $autoconf getIdentity
>> >>   version = $autoconf getVersion
>> >>   name = "$identity - $version"
>> >>   url = $autoconf getUrl
>> >>   mimetype = $autoconf getMimetype
>> >>   $workspace ca [ artifactName="$name" url="$url" mimetype="$mimetype"
>> >> Bundle-SymbolicName="$identity" Bundle-Version="$version"
>> >> Deployment-ProvidesResourceProcessor="org.osgi.deployment.rp.autoconf" ]
>> >>   echo " -> created"
>> >> }
>> >>
>> >> echo "Purging old artifacts"
>> >> artifacts = ($workspace la)
>> >> each $artifacts {
>> >>   $workspace da $it
>> >> }
>> >>
>> >> echo "Creating new artifacts"
>> >> each $deployed {
>> >>   identity = $it getIdentity
>> >>   version = $it getVersion
>> >>   name = "$identity - $version"
>> >>   url = $it getUrl
>> >>   mimetype = $it getMimetype
>> >>   if { $mimetype equals "application/xml:osgi-autoconf" } \
>> >>   { $workspace ca [ artifactName="$name" url="$url" mimetype="$mimetype"
>> >> filename="$name" processorPid="org.osgi.deployment.rp.autoconf" ] 
>> >> [ cdartifact="true" ] } \
>> >>   { $workspace ca [ artifactName="$name" url="$url" mimetype="$mimetype"
>> >> Bundle-SymbolicName="$identity" Bundle-Version="$version" ] [ 
>> >> cdartifact="true"] } }
>> >>
>> >> echo "Ensuring approval on target"
>> >> if { $target needsApprove } { echo " approving"; $target approve } 
>> >> {
>> echo
>> >> " no changes" }
>> >>
>> >> echo "Committing workspace"
>> >> $workspace commit
>> >>
>> >> # Let events settle
>> >> misc:sleep 2000
>> >> ace:rw $workspace
>> >>
>> >> # Let events settle
>> >> misc:shutdown 2000
>> >>
>> >>
>> >> Greetings, Marcel
>> >>
>> >>
>>

Mime
View raw message