camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Quinn Stevenson <qu...@pronoia-solutions.com>
Subject Re: Usage of Lambdas in our Maven build (jdk8-lambdas branch)
Date Mon, 28 Mar 2016 15:03:26 GMT
I’d like to try the bnd-maven-plugin, but I can’t get the build to work on the branch.

When I run

mvn -DskipTests clean install

I get

[ERROR] Failed to execute goal org.codehaus.mojo:ianal-maven-plugin:1.0-alpha-1:verify-legal-files
(default) on project maven-plugins: Artifact does not contain any legal files: maven-plugins-2.18-SNAPSHOT.jar
-> [Help 1]

What am I doing wrong here?


> On Mar 27, 2016, at 10:24 AM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
> 
> Hi
> 
> I got the OSGi tests to work with karaf 4 and they are all passing now
> for the features tests. There is a few that has been @Ignored because
> of known issues. eg we need a new SMX bundle for them.
> 
> You can run these tests reliable using a bash script
> 
> cd tests/camel-itest-karaf
> ./run-tests.sh
> 
> You can also try running using mvn with
> 
> mvn clean install
> 
> But the latter tend to break on my computer after 20 or so tests. The
> scripts runs reliable and it also cleanup dangling karaf JVMs. But
> generally its been safer to run one mvn goal per test it seems.
> 
> I also fixed the mistakes in karaf features and the tests so they are
> working again.
> 
> Raul you should be able to use that on your branch to give the osgi tests a go.
> 
> 
> 
> 
> 
> 
> On Sat, Mar 26, 2016 at 5:47 PM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
>> I managed to get the CI jobs to complete now with setting higher memory.
>> 
>> The snapshots are now deployed and the CI jobs for osgi is triggered
>> 
>> Triggering a new build of Camel.trunk.itest.karaf
>> Triggering a new build of Camel.trunk.itest.osgi
>> Finished: SUCCESS
>> 
>> 
>> 
>> On Sat, Mar 26, 2016 at 12:36 PM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
>>> Oh I forgot the CI jobs are here
>>> https://builds.apache.org/view/A-D/view/Camel/
>>> 
>>> I got the link from Mueller. Beforehand it was hard/impossible to find
>>> Camel from
>>> https://builds.apache.org/
>>> 
>>> So I was for a longer time not able to see all these jobs, and just
>>> got the CI emails to look at.
>>> 
>>> But the OSGi jobs rely on the SNAPSHOT of Camel has been built prior
>>> to that, and that job is unreliable. I think we need to look at making
>>> the -notest also publish the SNAPSHOT to the snapshot mvn repo, so the
>>> CI jobs can pickup these new JARs.
>>> 
>>> If we get that more reliable then Camel end users can also easier use
>>> SNAPSHOT instead of having to build locally to be sure they get a
>>> clean build of it, otherwise you can end up with mixed builds or
>>> stale/old snapshots.
>>> 
>>> 
>>> 
>>> On Sat, Mar 26, 2016 at 12:33 PM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
>>>> On Sat, Mar 26, 2016 at 12:12 PM, Raul Kripalani <raulk@apache.org>
wrote:
>>>>> Agree with your points.
>>>>> 
>>>>> 1. If you use lambdas on 2.18, you can't backport the code.
>>>>> 
>>>>> 2. I'll spend some time this weekend installing the bundles on Karaf.
Karaf
>>>>> 4 is OK, or are we sticking with older versions for 2.x?
>>>>> 
>>>> 
>>>> Karaf 2.x is dropped. Use Karaf 4.x as the primary karaf version.
>>>> We may want to add a -Pkaraf3 profile to run against older Karaf 3.x containers.
>>>> 
>>>> 
>>>>> 3. I'll also run our OSGi itests, but it may be worth having Jenkins
do all
>>>>> this. Could you configure a job in JKS to build this branch? I don't
have
>>>>> job admin rights I believe.
>>>>> 
>>>> 
>>>> We already have some CI jobs to run 2 osgi based tests
>>>> 
>>>> - tests/camel-itest-karaf
>>>> - tests/camel-itest-osgi
>>>> 
>>>> The former tests that the Camel component can be installed, it does
>>>> this using pax-exam to bootup karaf and install the camel-xxx feature
>>>> and find the component and then tear down the test.
>>>> 
>>>> You can run these tests with the -Pkaraf4 profile.
>>>> But mind I had trouble with these tests may stop after 20 or so tests
>>>> and leave some karaf JVMs dangling. There is a kill-karaf.sh script to
>>>> kill em.
>>>> 
>>>> The camel-itest-osgi is likely in some state where some tests would
>>>> fail even for older branches. But these are unit tests that run Camel
>>>> apps in karaf and do some testing stuff like we do in regular
>>>> component tests. This has not yet been migrated to karaf 4, and run on
>>>> karaf 2.x. But you can give it a go and see how bad/good it is. It
>>>> used to be able to run almost all tests and pass. And we had also had
>>>> periods where everything was green.
>>>> 
>>>> However people dont write so many osgi tests so this module do not
>>>> have so much love. And there are plenty of components that are not
>>>> being tested.
>>>> 
>>>> 
>>>>> 4. It's necessary to configure the maven-jar-plugin to explicitly use
a
>>>>> manifest file, otherwise the plugin generates its own. Originally I tried
>>>>> adding the config to the build plugins in the parent POM, but it failed
for
>>>>> artifacts of type=maven-plugin. Hence I placed it in pluginManagement
and
>>>>> then incorporated it in those intermediate POMs (components, examples,
>>>>> etc.) whose children built normal JARs. However, I ended up tracing the
>>>>> issue to a conflict introduced by the config of the maven-jar-plugin
that
>>>>> maven-plugin injects, and solved it with a couple
>>>>> of combine.self="override". Hence, I think I might be able to push our
>>>>> maven-jar-plugin config back to the parent POM and to the build section.
>>>>> 
>>>> 
>>>> Ah okay. I am okay with we can do "anything" we need to build all the
>>>> Camel components and whatnot.
>>>> 
>>>> Its more what the impact is for end users. If we can make out examples
>>>> and maven archetypes simpler and easier that would be cool. But mind
>>>> that not all examples are osgi examples.
>>>> 
>>>> But it looks promising your work - thanks for starting this hard work.
>>>> 
>>>> 
>>>> 
>>>>> Cheers,
>>>>> 
>>>>> *Raúl Kripalani*
>>>>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
Data and
>>>>> Messaging Engineer
>>>>> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
>>>>> Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk>
>>>>> 
>>>>> On Sat, Mar 26, 2016 at 10:12 AM, Claus Ibsen <claus.ibsen@gmail.com>
wrote:
>>>>> 
>>>>>> Hi
>>>>>> 
>>>>>> Good to hear that we may drop the <bundle> extension, nice
work.
>>>>>> 
>>>>>> For camel-core I have thought of build the manifest.mf by hand so
we
>>>>>> can better control it. It seems the bundle plugin has its issues
with
>>>>>> different versions to build it, as you say.
>>>>>> 
>>>>>> Have you tried testing any of these JARs in OSGi? It would be good
if
>>>>>> you try that and install some of the examples in a karaf 4 container
>>>>>> and see how it runs.
>>>>>> 
>>>>>> For lambda code in the existing source code then we cannot do that
for
>>>>>> code that need to be backported to 2.17 and 2.16 branches. We can
only
>>>>>> use lambdas in new code that is not to be backported.
>>>>>> 
>>>>>> For Camel 3.0 we can then covert the code to use lambads all over
>>>>>> where applicable. There is likely some tools in IDEA or Eclipse etc
>>>>>> that can assist with that.
>>>>>> 
>>>>>> 
>>>>>> And Camel end users can use lamdas today with their own code, also
>>>>>> with older Camel versions on java 8.
>>>>>> 
>>>>>> It may be that those people are using a new bundle plugin, or more
>>>>>> likely not using OSGi at all.
>>>>>> 
>>>>>> 
>>>>>> I would like to see more OSGi testing on this branch before any merge.
>>>>>> OSGi is complex and pain to maintain and build modules. So we need
to
>>>>>> be sure we are on a path that we can do this. There is 250+ maven
>>>>>> modules that are build as OSGi so it better have to work for us.
>>>>>> 
>>>>>> But I really like that the bundle plugin is just generating a
>>>>>> MANIFEST.MF file that the JAR plugin just slurps in. Can it not be
>>>>>> configured somehow so the generated MANIFEST.MF is not replace so
end
>>>>>> users do not need to do anything with a JAR plugin?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Sat, Mar 26, 2016 at 1:09 AM, Raul Kripalani <raulk@apache.org>
wrote:
>>>>>>> There is good news, bad news, and good news again ;-)
>>>>>>> 
>>>>>>> *Good news:* Now that Camel 2.18 is officially on JDK8, we can
use
>>>>>> lambdas
>>>>>>> in our code. Woohoo!
>>>>>>> 
>>>>>>> *Bad news:* Our current Maven build breaks when doing so.
>>>>>>> 
>>>>>>> This is what happens:
>>>>>>> 
>>>>>>>   1. maven-bundle-plugin version 2.3.x cannot parse JDK8 lambdas.
[1]
>>>>>>> 
>>>>>>>   2. Upgrading it to the latest version (3.0.1) makes the plugin
fail
>>>>>>> miserably (at least on camel-core).
>>>>>>> 
>>>>>>>   3. Using the latest 2.x version of the maven-bundle-plugin
(2.5.4)
>>>>>> makes
>>>>>>> the maven-shade-plugin fail with this error in turn:
>>>>>>> 
>>>>>>> [ERROR] The project main artifact does not exist. This could
have the
>>>>>>> following
>>>>>>> [ERROR] reasons:
>>>>>>> [ERROR] - You have invoked the goal directly from the command
line. This
>>>>>> is
>>>>>>> not
>>>>>>> [ERROR]   supported. Please add the goal to the default lifecycle
via an
>>>>>>> [ERROR]   <execution> element in your POM and use "mvn
package" to have
>>>>>> it
>>>>>>> run.
>>>>>>> [ERROR] - You have bound the goal to a lifecycle phase before
"package".
>>>>>>> Please
>>>>>>> [ERROR]   remove this binding from your POM such that the goal
will be
>>>>>> run
>>>>>>> in
>>>>>>> [ERROR]   the proper phase.
>>>>>>> 
>>>>>>>   4. Upgrading the shade plugin doesn't help.
>>>>>>> 
>>>>>>>   5. The situation is dirty :P
>>>>>>> 
>>>>>>> The problem is the usage of the 'bundle' custom packaging type
enabled by
>>>>>>> extensions='true' in the maven-bundle-plugin.
>>>>>>> 
>>>>>>> *Good news again:*
>>>>>>> 
>>>>>>> We can avoid the problem by reverting back to the standard 'jar'
>>>>>> packaging
>>>>>>> type, which even puts us in a less brittle situation wrt. to
future Maven
>>>>>>> plugins we may want to add/upgrade. This implies:
>>>>>>> 
>>>>>>>   1. Calling the maven-bundle-plugin:manifest goal (instead of
the
>>>>>> bundle
>>>>>>> goal) to only generate the MANIFEST.MF.
>>>>>>> 
>>>>>>>   2. Adding it to the JAR via an option in the the maven-jar-plugin:
>>>>>>> 
>>>>>>>    <plugin>
>>>>>>>        <groupId>org.apache.maven.plugins</groupId>
>>>>>>>        <artifactId>maven-jar-plugin</artifactId>
>>>>>>>        <version>${maven-jar-plugin-version}</version>
>>>>>>>        <configuration>
>>>>>>>          <archive>
>>>>>>> 
>>>>>>> 
>>>>>> <manifestFile>${project.build.outputDirectory}/META-INF/MANIFEST.MF</manifestFile>
>>>>>>>          </archive>
>>>>>>>        </configuration>
>>>>>>>      </plugin>
>>>>>>> 
>>>>>>> I've made the change. But since it's a big one, I didn't push
to master
>>>>>> but
>>>>>>> to a branch for us to review first: jdk8-lambdas.
>>>>>>> 
>>>>>>> The build passes (yay!) but OSGi battle testing is a must to
ensure there
>>>>>>> are no regressions.
>>>>>>> 
>>>>>>> Could you have a quick look? If no feedback is received until
Monday EOD,
>>>>>>> I'll merge to master. If feedback is positive, we can merge earlier
to
>>>>>>> enable people to develop with lambdas finally.
>>>>>>> 
>>>>>>> ---
>>>>>>> 
>>>>>>> Just in case you're not aware, the OSGi legend Neil Bartlett
has just
>>>>>>> released a new bnd-maven-plugin [2] that claims to overcome such
>>>>>>> dysfunctions, as well as others, of the Felix plugin. He complained
about
>>>>>>> how the custom packaging type breaks integration with other plugins
>>>>>>> downstream (what we're experiencing).
>>>>>>> 
>>>>>>> I see no reason to migrate to his bnd-maven-plugin, but we should
keep it
>>>>>>> in mind for the future.
>>>>>>> 
>>>>>>> [1] https://issues.apache.org/jira/browse/FELIX-4005
>>>>>>> [2] http://njbartlett.name/2015/03/27/announcing-bnd-maven-plugin.html
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> 
>>>>>>> *Raúl Kripalani*
>>>>>>> PMC & Committer @ Apache Ignite, Apache Camel | Integration,
Big Data and
>>>>>>> Messaging Engineer
>>>>>>> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
>>>>>>> Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk>
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> -----------------
>>>>>> http://davsclaus.com @davsclaus
>>>>>> Camel in Action 2: https://www.manning.com/ibsen2
>>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> http://davsclaus.com @davsclaus
>>>> Camel in Action 2: https://www.manning.com/ibsen2
>>> 
>>> 
>>> 
>>> --
>>> Claus Ibsen
>>> -----------------
>>> http://davsclaus.com @davsclaus
>>> Camel in Action 2: https://www.manning.com/ibsen2
>> 
>> 
>> 
>> --
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2
> 
> 
> 
> -- 
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message