camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Usage of Lambdas in our Maven build (jdk8-lambdas branch)
Date Sat, 26 Mar 2016 16:47:00 GMT
I managed to get the CI jobs to complete now with setting higher memory.

The snapshots are now deployed and the CI jobs for osgi is triggered

Triggering a new build of Camel.trunk.itest.karaf
Triggering a new build of Camel.trunk.itest.osgi
Finished: SUCCESS



On Sat, Mar 26, 2016 at 12:36 PM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
> Oh I forgot the CI jobs are here
> https://builds.apache.org/view/A-D/view/Camel/
>
> I got the link from Mueller. Beforehand it was hard/impossible to find
> Camel from
> https://builds.apache.org/
>
> So I was for a longer time not able to see all these jobs, and just
> got the CI emails to look at.
>
> But the OSGi jobs rely on the SNAPSHOT of Camel has been built prior
> to that, and that job is unreliable. I think we need to look at making
> the -notest also publish the SNAPSHOT to the snapshot mvn repo, so the
> CI jobs can pickup these new JARs.
>
> If we get that more reliable then Camel end users can also easier use
> SNAPSHOT instead of having to build locally to be sure they get a
> clean build of it, otherwise you can end up with mixed builds or
> stale/old snapshots.
>
>
>
> On Sat, Mar 26, 2016 at 12:33 PM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
>> On Sat, Mar 26, 2016 at 12:12 PM, Raul Kripalani <raulk@apache.org> wrote:
>>> Agree with your points.
>>>
>>> 1. If you use lambdas on 2.18, you can't backport the code.
>>>
>>> 2. I'll spend some time this weekend installing the bundles on Karaf. Karaf
>>> 4 is OK, or are we sticking with older versions for 2.x?
>>>
>>
>> Karaf 2.x is dropped. Use Karaf 4.x as the primary karaf version.
>> We may want to add a -Pkaraf3 profile to run against older Karaf 3.x containers.
>>
>>
>>> 3. I'll also run our OSGi itests, but it may be worth having Jenkins do all
>>> this. Could you configure a job in JKS to build this branch? I don't have
>>> job admin rights I believe.
>>>
>>
>> We already have some CI jobs to run 2 osgi based tests
>>
>> - tests/camel-itest-karaf
>> - tests/camel-itest-osgi
>>
>> The former tests that the Camel component can be installed, it does
>> this using pax-exam to bootup karaf and install the camel-xxx feature
>> and find the component and then tear down the test.
>>
>> You can run these tests with the -Pkaraf4 profile.
>> But mind I had trouble with these tests may stop after 20 or so tests
>> and leave some karaf JVMs dangling. There is a kill-karaf.sh script to
>> kill em.
>>
>> The camel-itest-osgi is likely in some state where some tests would
>> fail even for older branches. But these are unit tests that run Camel
>> apps in karaf and do some testing stuff like we do in regular
>> component tests. This has not yet been migrated to karaf 4, and run on
>> karaf 2.x. But you can give it a go and see how bad/good it is. It
>> used to be able to run almost all tests and pass. And we had also had
>> periods where everything was green.
>>
>> However people dont write so many osgi tests so this module do not
>> have so much love. And there are plenty of components that are not
>> being tested.
>>
>>
>>> 4. It's necessary to configure the maven-jar-plugin to explicitly use a
>>> manifest file, otherwise the plugin generates its own. Originally I tried
>>> adding the config to the build plugins in the parent POM, but it failed for
>>> artifacts of type=maven-plugin. Hence I placed it in pluginManagement and
>>> then incorporated it in those intermediate POMs (components, examples,
>>> etc.) whose children built normal JARs. However, I ended up tracing the
>>> issue to a conflict introduced by the config of the maven-jar-plugin that
>>> maven-plugin injects, and solved it with a couple
>>> of combine.self="override". Hence, I think I might be able to push our
>>> maven-jar-plugin config back to the parent POM and to the build section.
>>>
>>
>> Ah okay. I am okay with we can do "anything" we need to build all the
>> Camel components and whatnot.
>>
>> Its more what the impact is for end users. If we can make out examples
>> and maven archetypes simpler and easier that would be cool. But mind
>> that not all examples are osgi examples.
>>
>> But it looks promising your work - thanks for starting this hard work.
>>
>>
>>
>>> Cheers,
>>>
>>> *Raúl Kripalani*
>>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
>>> Messaging Engineer
>>> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
>>> Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk>
>>>
>>> On Sat, Mar 26, 2016 at 10:12 AM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> Good to hear that we may drop the <bundle> extension, nice work.
>>>>
>>>> For camel-core I have thought of build the manifest.mf by hand so we
>>>> can better control it. It seems the bundle plugin has its issues with
>>>> different versions to build it, as you say.
>>>>
>>>> Have you tried testing any of these JARs in OSGi? It would be good if
>>>> you try that and install some of the examples in a karaf 4 container
>>>> and see how it runs.
>>>>
>>>> For lambda code in the existing source code then we cannot do that for
>>>> code that need to be backported to 2.17 and 2.16 branches. We can only
>>>> use lambdas in new code that is not to be backported.
>>>>
>>>> For Camel 3.0 we can then covert the code to use lambads all over
>>>> where applicable. There is likely some tools in IDEA or Eclipse etc
>>>> that can assist with that.
>>>>
>>>>
>>>> And Camel end users can use lamdas today with their own code, also
>>>> with older Camel versions on java 8.
>>>>
>>>> It may be that those people are using a new bundle plugin, or more
>>>> likely not using OSGi at all.
>>>>
>>>>
>>>> I would like to see more OSGi testing on this branch before any merge.
>>>> OSGi is complex and pain to maintain and build modules. So we need to
>>>> be sure we are on a path that we can do this. There is 250+ maven
>>>> modules that are build as OSGi so it better have to work for us.
>>>>
>>>> But I really like that the bundle plugin is just generating a
>>>> MANIFEST.MF file that the JAR plugin just slurps in. Can it not be
>>>> configured somehow so the generated MANIFEST.MF is not replace so end
>>>> users do not need to do anything with a JAR plugin?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Mar 26, 2016 at 1:09 AM, Raul Kripalani <raulk@apache.org>
wrote:
>>>> > There is good news, bad news, and good news again ;-)
>>>> >
>>>> > *Good news:* Now that Camel 2.18 is officially on JDK8, we can use
>>>> lambdas
>>>> > in our code. Woohoo!
>>>> >
>>>> > *Bad news:* Our current Maven build breaks when doing so.
>>>> >
>>>> > This is what happens:
>>>> >
>>>> >    1. maven-bundle-plugin version 2.3.x cannot parse JDK8 lambdas. [1]
>>>> >
>>>> >    2. Upgrading it to the latest version (3.0.1) makes the plugin fail
>>>> > miserably (at least on camel-core).
>>>> >
>>>> >    3. Using the latest 2.x version of the maven-bundle-plugin (2.5.4)
>>>> makes
>>>> > the maven-shade-plugin fail with this error in turn:
>>>> >
>>>> > [ERROR] The project main artifact does not exist. This could have the
>>>> > following
>>>> > [ERROR] reasons:
>>>> > [ERROR] - You have invoked the goal directly from the command line.
This
>>>> is
>>>> > not
>>>> > [ERROR]   supported. Please add the goal to the default lifecycle via
an
>>>> > [ERROR]   <execution> element in your POM and use "mvn package"
to have
>>>> it
>>>> > run.
>>>> > [ERROR] - You have bound the goal to a lifecycle phase before "package".
>>>> > Please
>>>> > [ERROR]   remove this binding from your POM such that the goal will
be
>>>> run
>>>> > in
>>>> > [ERROR]   the proper phase.
>>>> >
>>>> >    4. Upgrading the shade plugin doesn't help.
>>>> >
>>>> >    5. The situation is dirty :P
>>>> >
>>>> > The problem is the usage of the 'bundle' custom packaging type enabled
by
>>>> > extensions='true' in the maven-bundle-plugin.
>>>> >
>>>> > *Good news again:*
>>>> >
>>>> > We can avoid the problem by reverting back to the standard 'jar'
>>>> packaging
>>>> > type, which even puts us in a less brittle situation wrt. to future
Maven
>>>> > plugins we may want to add/upgrade. This implies:
>>>> >
>>>> >    1. Calling the maven-bundle-plugin:manifest goal (instead of the
>>>> bundle
>>>> > goal) to only generate the MANIFEST.MF.
>>>> >
>>>> >    2. Adding it to the JAR via an option in the the maven-jar-plugin:
>>>> >
>>>> >     <plugin>
>>>> >         <groupId>org.apache.maven.plugins</groupId>
>>>> >         <artifactId>maven-jar-plugin</artifactId>
>>>> >         <version>${maven-jar-plugin-version}</version>
>>>> >         <configuration>
>>>> >           <archive>
>>>> >
>>>> >
>>>> <manifestFile>${project.build.outputDirectory}/META-INF/MANIFEST.MF</manifestFile>
>>>> >           </archive>
>>>> >         </configuration>
>>>> >       </plugin>
>>>> >
>>>> > I've made the change. But since it's a big one, I didn't push to master
>>>> but
>>>> > to a branch for us to review first: jdk8-lambdas.
>>>> >
>>>> > The build passes (yay!) but OSGi battle testing is a must to ensure
there
>>>> > are no regressions.
>>>> >
>>>> > Could you have a quick look? If no feedback is received until Monday
EOD,
>>>> > I'll merge to master. If feedback is positive, we can merge earlier
to
>>>> > enable people to develop with lambdas finally.
>>>> >
>>>> > ---
>>>> >
>>>> > Just in case you're not aware, the OSGi legend Neil Bartlett has just
>>>> > released a new bnd-maven-plugin [2] that claims to overcome such
>>>> > dysfunctions, as well as others, of the Felix plugin. He complained
about
>>>> > how the custom packaging type breaks integration with other plugins
>>>> > downstream (what we're experiencing).
>>>> >
>>>> > I see no reason to migrate to his bnd-maven-plugin, but we should keep
it
>>>> > in mind for the future.
>>>> >
>>>> > [1] https://issues.apache.org/jira/browse/FELIX-4005
>>>> > [2] http://njbartlett.name/2015/03/27/announcing-bnd-maven-plugin.html
>>>> >
>>>> > Cheers,
>>>> >
>>>> > *Raúl Kripalani*
>>>> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
Data and
>>>> > Messaging Engineer
>>>> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
>>>> > Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> http://davsclaus.com @davsclaus
>>>> Camel in Action 2: https://www.manning.com/ibsen2
>>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Mime
View raw message