karaf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré ...@nanthrax.net>
Subject Re: [heads up - long] "Features Processor" that replaces overrides and blacklisting (and more)
Date Thu, 07 Dec 2017 00:54:52 GMT
That's really a great improvement. Thanks for that !

Regards
JB

On 12/06/2017 08:12 PM, Grzegorz Grzybek wrote:
> Hello again
> 
> I'd like to summary recent additions related to my changes:
> 
>   - feature:install and feature:repo-add won't install/add blacklisted
> feature/repository
> 
>   - etc/org.apache.karaf.features.cfg may use property placeholders and
> these will be resolved from bundle context properties and from
> etc/versions.properties (configurable). The idea is to have e.g.,:
> 
>      <bundleReplacements>
>          <bundle
> replacement="mvn:commons-io/commons-io/${version.commons-io}" />
>      </bundleReplacements>
> 
> and just list all the versions we need in etc/versions.properties. So when
> fixed bundle comes out, it's a matter of changing property in single file
> and all features will use overriden version.
> 
>   - implemented overriding feature definitions (replace entire feature
> definition)
> 
>   - implemented overriding dependency="true|false" flag in bundles. If we
> know that some external features XML should use this flag and it doesn't,
> we can configure it externally without trying to have our PR merged in
> upstream project.
> 
>   - profiles can use "blacklisted.bundle.xxx", "blacklisted.feature.xxx" and
> "blacklisted.repository.xxx" properties (KARAF-5339)
> 
>   - features specified in karaf-maven-plugin:assembly configuration may use
> wildcards (KARAF-5273)
> 
> I still hope to get some comments/review ;) I know it's not quick fix. If
> you think I went in wrong direction, please let me know.
> 
> best regards
> Grzegorz Grzybek
> 
> 2017-12-02 21:27 GMT+01:00 Grzegorz Grzybek <gr.grzybek@gmail.com>:
> 
>> Hello
>>
>> I want to report about my progress with https://issues.apache.org/
>> jira/browse/KARAF-5376 (Processor mechanism for feature definitions
>> (a.k.a. "better overrides")).
>>
>> *Goal* - to replace (or provide more flexible alternative) of "overrides"
>> and "blacklisting" mechanisms
>>
>> *Idea* - Karaf features come (normally) from features XML files,
>> according to XML Schema specified by Karaf itself. My idea was that we can
>> "intercept" the process of loading the repositories and alter the
>> definitions. Effectively we could then affect the definitions which may be
>> out of our influence, or use some bundles we'd rather like to replace with
>> others. Some feature repositories include other repositories (e.g., Camel
>> 2.19.2 features referencing CXF 3.1.11 features) and we'd rather like to
>> avoid it.
>>
>> *Problems with blacklisting and overrides* - blacklisting is purely
>> static thing - applied at distro creation time. Overrides are a bit awkward
>> to use ("range" clause) and have some implicit behavior (requiring runtime
>> access to resource, to verify version and symbolic name).
>>
>> *What I did*
>>
>> In short words - I've created a mechanism steered by
>> etc/org.apache.karaf.features.xml (name matching
>> etc/org.apache.karaf.features.cfg PID) that looks roughly like this:
>>
>> <featuresProcessing xmlns="http://karaf.apache.
>> org/xmlns/features-processing/v1.0.0"
>>          xmlns:f="http://karaf.apache.org/xmlns/features/v1.5.0">
>>
>>      <blacklistedRepositories>
>>          <repository>mvn:org.hibernate/hibernate-validator-osgi-
>> karaf-features/[5,*)/xml/features</repository>
>>          <!-- ... -->
>>      </blacklistedRepositories>
>>      <blacklistedFeatures>
>>          <feature>*jetty*</feature>
>>          <feature version="[2,3)">*jclouds*</feature>
>>          <!-- ... -->
>>      </blacklistedFeatures>
>>      <blacklistedBundles>
>>          <bundle>mvn:commons-logging/*</bundle>
>>      </blacklistedBundles>
>>
>>      <overrideBundleDependency>
>>          <!-- Override "dependency" flag for all bundles of all features
>> for given repository URI(s) -->
>>          <repository uri="mvn:org.ops4j.pax.cdi/pax-cdi-features/*/xml/features"
>> dependency="true" />
>>          <repository uri="mvn:*/xml/features" dependency="true" />
>>          <!-- Override "dependency" flag for all bundles of given
>> feature(s) -->
>>          <feature name="jclouds*" version="[1,3)" dependency="true" />
>>          <!-- Override "dependency" flag for given bundle(s) -->
>>          <bundle uri="mvn:javax.annotation/javax.annotation-api/*"
>> dependency="true" />
>> ...
>>      </overrideBundleDependency>
>>
>>      <bundleReplacements>
>>          <bundle originalUri="mvn:commons-beanutils/commons-beanutils/[
>> 1.9,2)"
>>                  replacement="mvn:commons-beanutils/commons-beanutils/1.9.3"
>> />
>>          <bundle replacement="mvn:commons-collections/commons-collections/3.2.2"
>> />
>>          <bundle originalUri="mvn:org.eclipse.jetty.orbit/javax.servlet/[3,
>> 4)"
>>                  replacement="mvn:org.apache.geronimo.specs/geronimo-servlet_3.0_spec/1.0"
>> mode="maven" />
>>          <!-- ... -->
>>      </bundleReplacements>
>>
>>      <featureReplacements>
>>          <replacement mode="replace">
>>              <feature name="pax-jsf-resources-support"
>> description="Provide sharing of resources according to Servlet 3.0 for OSGi
>> bundles and JSF" version="6.0.7">
>>                  <f:feature version="[6.0,6.1)">pax-jsf-support</f:feature>
>>                  <f:bundle dependency="true">mvn:org.ops4j.pax.web/pax-web-
>> resources-extender/6.0.7</f:bundle>
>>                  <f:bundle>mvn:org.ops4j.pax.web/pax-web-resources-jsf/6.0.
>> 7</f:bundle>
>>              </feature>
>>          </replacement>
>>          <!-- ... -->
>>      </featureReplacements>
>>
>> </featuresProcessing>
>>
>> Description:
>>
>>   - blacklistedRepositories, blacklistedFeatures and blacklistedBundles
>> mean what they mean - mark (instead of removing) given item kind from JAXB
>> model of loaded features XML. This information is preserved at runtime, and
>> for example is displayed with "features:list" command
>>
>>   - overrideBundleDependency (not yet implemented) - is a method of
>> overriding "dependency="true|false"" flags in feature definitions
>>
>>   - bundleReplacements (implemented) - is an extension of "overrides"
>> mechanism. In "osgi" mode, it roughly works like before - requires access
>> to runtime ResourceImpl, to check headers and compare versions and symbolic
>> names. In "maven" mode it's just static change of given bundle URI - any
>> feature declaring <bundle>originalUri</bundle> will be seen as declaring
>> <bundle>replacement</bundle>. This allows for replacement of e.g., SMX
>> specs bundle with javax.* bundle (or vice versa).
>>
>>   - featureReplacements (not yet implemented) - is a way of changing (or
>> altering) any features by name (or pattern) and version simply by providing
>> different definition of given feature. This could be useful if we as Karaf
>> users know "better" than authors of original features, for whom OSGi and
>> Karaf were rather an afterthought.
>>
>> Both bundle/feature/repository blacklisting and bundle/feature override
>> use two new classes to "match" candidate to replace/override:
>>   - org.apache.karaf.features.LocationPattern - matches URI (of bundle or
>> repository) in two modes - for non mvn: URI, it just uses a string with
>> possible "*" glob. For mvn: URI, it splits the URI into well known
>> components (groupId, artifactId, version, type and classifier). For version
>> it can use version range (e.g, "[2.1,2.2)") and for other components it can
>> use "*" glob.
>>   - org.apache.karaf.features.FeaturePattern - matches feature by name
>> (possibly containing "*" glob) and version range (org.apache.felix.utils.
>> version.VersionRange)
>>
>> Universally applicable:
>>
>> FeaturesProcessor generally postprocesses org.apache.karaf.features.internal.model.Features
>> instance(s) and marks items as blacklisted if they match given pattern and
>> changes the model (according to overrides/replacements).
>>
>> The behavior after processing JAXB model of features is exactly the same
>> both when using FeaturesServiceImpl at runtime and when using profile
>> Builder during custom assembly creation. For example changed flow of
>> karaf-maven-plugin:asembly invocation looks like this:
>>
>> [INFO] --- karaf-maven-plugin:4.2.0-SNAPSHOT:assembly (default-assembly)
>> @ test-karaf-plugins ---
>> [INFO] Using repositories:
>> [INFO]    https://repo.maven.apache.org/maven2@id=central
>> ...
>> [INFO]    Custom startup KAR found: mvn:grgr.test.karaf.plugins/
>> my-kar/0.1.0.BUILD-SNAPSHOT/kar
>> ...
>> [INFO] Unzipping kars
>> [INFO]    processing KAR: mvn:grgr.test.karaf.plugins/
>> my-kar/0.1.0.BUILD-SNAPSHOT/kar
>> [INFO]       found repository: mvn:grgr.test.karaf.plugins/
>> my-kar/0.1.0.BUILD-SNAPSHOT/xml/features
>> [INFO] Found features processor configuration: ../classes/etc/org.apache.
>> karaf.features.xml
>> ...
>> [INFO] Loading profiles from:
>> [INFO]    file:///home/ggrzybek/sources/_testing/grgr-test-karaf-
>> plugins/src/main/profiles/1
>> [INFO]    jar:mvn:test/profiles/42!/profiles/2
>> [INFO]    Found profiles: p1a, p1b-another, p2b, p2c-two, p2c-one-two, p2a
>> ...
>> [INFO] Generating features processor configuration:
>> etc/org.apache.karaf.features.xml
>> [INFO] Startup stage
>> [INFO]    Loading startup repositories
>> [INFO]    Resolving startup features and bundles
>> [INFO]       Features: undertow, my-startup-feature
>> ...
>> [INFO]       skipping blacklisted maven bundle: mvn:org.ops4j.pax.tipi/org.
>> ops4j.pax.tipi.undertow.websocket-jsr/1.3.25.1
>> ...
>> [INFO] Boot stage
>> [INFO]    Loading boot repositories
>> [INFO]       adding feature repository: mvn:org.ops4j.pax.web/pax-web-
>> features/6.0.7/xml/features
>> [INFO]       adding feature repository: mvn:grgr.test.karaf.plugins/
>> my-kar/0.1.0.BUILD-SNAPSHOT/xml/features
>> [INFO]    Feature undertow/1.3.25.1 is defined as a boot feature
>> ...
>> [INFO]       adding overriden maven artifact: mvn:org.jboss.logging/jboss-logging/3.3.1.Final
>> (original location: mvn:org.jboss.logging/jboss-logging/3.3.0.Final)
>> ...
>> [INFO]       skipping blacklisted maven artifact:
>> mvn:org.ops4j.pax.tipi/org.ops4j.pax.tipi.undertow.websocket-jsr/1.3.25.1
>> [INFO]       adding overriden maven artifact: mvn:javax.servlet/javax.servlet-api/3.1-b09
>> (original location: mvn:javax.servlet/javax.servlet-api/3.1.0)
>> ...
>> [INFO] Install stage
>> [INFO]    Loading installed repositories
>>
>> overrides and blacklisting information may come from karaf-maven-plugin
>> configuration (in POM), from externally provided
>> org.apache.karaf.features.xml (e.g., in resource KAR or in POM:
>> <configuration>/<featuresProcessing>) or from profiles.
>>
>> There are still few things to polish, but I'll appreciate any feedback to
>> check if I misunderstood something or there were other plans related to
>> these mechanisms. The changes are in https://github.com/apache/
>> karaf/commits/KARAF-5376-overrides_v2 branch.
>>
>> best regards
>> Grzegorz Grzybek
>>
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Mime
View raw message