edgent-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John D. Ament" <johndam...@apache.org>
Subject Re: Understanding the snapshot and release process
Date Sun, 28 May 2017 15:26:14 GMT
On Sun, May 28, 2017 at 11:12 AM Christofer Dutz <christofer.dutz@c-ware.de>
wrote:

> The Maven build should work on the ASF Jenkins and hereby publish the
> snapshots. I think that in the flex project I added this to ensure its
> added to the generated "site".
>
> Which one is the hosted Jenkins you are referring to? The asf Jenkins?
>
>
Yes, the ASF jenkins.


> And what should be the problem with Travis? Why are you using two ci
> servers?
>

AFAIK, Edgent only uses travis, and doesn't currently run jobs on ASF
Jenkins.  At least if they do, I was unable to find a job with a name that
was easy to identify.


> Chris
>
>
>
> Von meinem Samsung Galaxy Smartphone gesendet.
>
>
> -------- Ursprüngliche Nachricht --------
> Von: "John D. Ament" <johndament@apache.org>
> Datum: 28.05.17 16:20 (GMT+01:00)
> An: dev@edgent.apache.org
> Betreff: Re: Understanding the snapshot and release process
>
> The maven based build seems to be much more maintainable.  I'm glad you
> guys came together to solve this problem :-)
>
> RE distribution of snapshots.  We only presently support publishing via our
> hosted jenkins, however Edgent currently builds in Travis.  So that would
> need to be changed, or some other solution.
>
> You also can drop the repositories/pluginRepositories sections of the new
> pom.  Its not needed.
>
>
> John
>
> On Sun, May 28, 2017 at 10:03 AM Christofer Dutz <
> christofer.dutz@c-ware.de>
> wrote:
>
> > Hi guys,
> >
> > Strange people these Germans … if it rains, they complain about it and
> how
> > much they would like to go swimming in the sun … now that it’s 32° and
> the
> > sun is shining, no one wants to go swimming, cause it’s too fu***g hot
> ;-)
> > … well this way I had some time to finish the maven migration … I just
> > pushed the changes to the “feature/maven” branch of my fork on github:
> > https://github.com/chrisdutz/incubator-edgent/tree/feature/maven
> >
> > Here a short list on what little monsters I found siting under some
> rocks:
> >
> > 1) Usage of test-jars --> Bad practice → Move code to separate test-util
> > jar module
> > 2) AppServiceTest in provides/direct relies on system-properties to load
> > jar produced by another module → Ideally this should be refactored to
> find
> > the jar on the classpath
> > 3) HttpServerTest in console/server relies on console.war
> > 4) console/servlets module compiled to something with a name completely
> > unrelated to the project
> > 5) test/svt requires samples project → Some modules rely on samples.
> While
> > I think, this is ok for Samples themselves, nothing outside the samples
> > should rely on samples (my opinion)
> > 6) Almost all tests relying on successful SSL handshakes in
> > wsclient-javax.websocket are failing (I’ll investigate this as soon as I
> > have time, I could imagine this to be related to non US JVM encryption
> > issues we are also seeing in the Flex project)
> > 7) Some unit-tests should more be integration-tests
> >
> > I didn’t migrate the stuff in “platform” as I didn’t quite understand
> what
> > they are used for and what they should do.
> >
> > Hope you like the changes. With a build like this it should be easy to
> > setup the CI to auto distribute SNAPSHOTs and execute releases on ASF
> > machines with a simple one or two-liner.
> >
> > Chris
> >
> >
> > Am 27.05.17, 19:16 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de
> >:
> >
> >     I did have some free time this weekend, so I thought I’d give the
> > Maven thing a try. And right now I haven finished, but I think I’m about
> > 80% done. So far there have been one or two little monsters under some
> > rocks … but they weren’t too nasty ;-)
> >
> >     Regarding the classpath stuff in the manifest: I really dislike this
> > way of setting the classpath as it usually causes a lot of problems when
> > using it. It requires libs to be in a predefined directory structure and
> > follow a given naming convention. Try to use this in a project built with
> > Maven (just picking this as I saw this would be a problem for a Maven
> built
> > project)
> >
> >     Regarding tests: I did see that there are several Unit tests I would
> > probably more call integration-tests. Especially the ones that require
> > loading of a war (console.war). So I guess you have both some classical
> > unit-tests as well as integration-tests … no need to treat them the same
> > way. Maven ususally has the unit-test and the integration-test phase for
> > exactly this.
> >
> >     Regarding jacoco: So I’ll activate that in the maven build too
> >
> >     Regarding automatic build: For Flex I setup the build to do a
> > multibranch pipeline build. So whenever someone checks something in
> > “develop”, “releae/*” or “feature/*” it automatically builds that
> > particular branch. However develop and “release/*” are the only ones
> > uploading SNAPSHOT versions to the ASF Nexus.
> >
> >     Regarding my commitment: I think Edgent is the tool I was looking for
> > in order to make my SCADA project feature-perfect. So I have a strong
> > interest in the project maturing. I do have plans for a longer
> involvement
> > in the project. Probably I would be trying to create an open-source “plc”
> > connector (which would probably have to be developed outside of Apache
> due
> > to libraries still being GPL). But I do plan on staying around some time
> ;-)
> >
> >     Regarding the usage: Maven or Gradle based application developers
> > could directly utilize SNAPSHOT and release artifacts as they are
> released.
> > Additionally, I would create a so-called assembly that produces the same
> > zip or tar-gz archives the users have been using. So there shouldn’t be
> too
> > much of a change.
> >
> >     So now I’ll try to finish the rest of the connector modules maven
> > conversion (
> >
> >     Chris
> >
> >     Am 25.05.17, 19:37 schrieb "Dale LaBossiere" <dml.apache@gmail.com>:
> >
> >         Awaiting others to chime in on this “switch to mvn” subject.
> >         A “don’t care” (+/- 0) response is preferred over silence.
> >
> >         In the mean time...
> >
> >         > On May 24, 2017, at 11:41 AM, Christofer Dutz <
> > christofer.dutz@c-ware.de> wrote:
> >         > ...
> >         > I just checked out everything and managed to get things
> imported
> > in IntelliJ after a little struggle … this is a good job you did :-)
> >         Can’t take all the credit (or blame :-)
> >         >
> >         > Right now, it seems as if there were Ant+Gradle+Eclipse build
> > files in there. While it might seem ...
> >         gradle is the only CLI way to build edgent.  Any ant build.xml
> > files are either (a) residual cruft that should be removed (my bad) or
> (b)
> > needed to leverage the ant machinery for invoking retrolambda in building
> > java7/android compatible versions of the jars (machinery that wasn’t
> > converted to pure gradle due to time/effort/value at the time).
> Attempting
> > to use ant at the top level tells you it doesn’t work :-)  As for Eclipse
> > .project/.classpath, yup those are live and haven’t been a maintenance
> > issue.
> >
> >         fwiw, I just removed all build.xml except the top level one and
> > those under platform and “gradle release” worked fine so I’ll create a PR
> > to clean them up as a first step.
> >
> >         > ...JavaDoc is generated automatically when running a release
> > build together with the usual Maven project reports (is even configured
> in
> > the apache parent POM together with rat, deployment etc.)
> >         Javadoc was complicated by creating groupings as well as
> excluding
> > it for non-API classes.
> >
> >         > ...I didn’t quite understand the “Manifest” thing, but the jar
> > plugin does generate this with reasonable defaults, and can be extended
> to
> > also export the dependencies into that (even if I don’t recommend that).
> >         An edgent jar’s manifest class-path includes references to its
> > immediate dependent edgent jars (not transitive) as well as references to
> > its external jar dependencies (transitively).  I agree that "compiled in"
> > references to specific versions of external dependencies may not be a
> great
> > idea / is perhaps best eliminated.
> >
> >         My recollection is that by default gradle did not generate any
> > manifest class-path.
> >
> >         e.g., just to make this a bit more concrete, this is from
> > edgent.connectors.kafka.jar/MANIFEST.MF (see
> connectors/kafka/build.gradle
> > for more info)
> >
> >         Class-Path: ../../../lib/edgent.api.topology.jar
> > ../../../ext/gson-2.2
> >          .4.jar ../../../ext/slf4j-api-1.7.12.jar
> > ../../../ext/metrics-core-3.
> >          1.2.jar ../ext/kafka_2.10-0.8.2.2.jar
> > ../ext/kafka-clients-0.8.2.2.ja
> >          r ../ext/log4j-1.2.16.jar ../ext/metrics-core-2.2.0.jar
> > ../ext/scala-
> >          library-2.10.4.jar ../ext/zkclient-0.3.jar
> > ../ext/zookeeper-3.4.6.jar
> >
> >         edgent.api.topology.jar itself has lots of edgent jar
> dependencies
> > (captured in its manifest) but that’s of no concern to the kafka
> > connector.  I’m not sure maintaining this is a requirement and, at least
> > for gradle, eliminating it would have greatly simplified things.
> >
> >         Test execution environment was also an issue.  perhaps more a
> > result of structural issues.  i.e., the gradle config treated a
> component’s
> > tests as unit tests, run against the associated component’s .class files.
> > The ant system treated them more like integration tests, run against the
> > jar files that would be bundled into a binary-release bundle; more like
> the
> > environment that Edgent-based applications would use.  That was very
> > important.  And due to some manual setup requirements for some connector
> > tests are excluded from execution by default.
> >
> >         > …
> >         > In the Flex project, I also setup the build to run SonarQube
> > analysis and automatically generate the documentation from markdown
> and/or
> > asciidoctor (which I think is very convenient) even automatically update
> > and deploy the project website.
> >
> >         We include use of jacoco
> >         We have travis integration for auto-PR validation.
> >         It would be nice to have some periodic build/regression testing
> > run on the main (master) branch.
> >
> >         > I could offer to create a fork on GitHub, create a feature
> > branch there and try to whip up a set of poms that add Maven as fourth
> > build system to the list … you could check it out and play around with
> it.
> > But I’d only do this, if there is any interest in it.
> >
> >         I don’t blame you!
> >
> >         One worry I have is that it ends up like the gradle effort… it
> was
> > ~trivial to get started but there was then a lot of effort required
> > identify and flesh out equivalence with the ant based build result
> > artifacts.
> >         If you’re on board for the long haul in doing a conversion then
> > that lessens my concern.
> >
> >         I’m on board to help but don’t want to inherit a “now finish it”
> > task :-)
> >
> >         A full switch to a ~simple mvn-based build system would be the
> > goal IMO — we’d want to toss gradle. The effort ultimately also entails
> > updating a bunch of doc.  Not a killer, just not to be overlooked.
> >
> >         It’s unclear to me how the Edgent-based app developers will be
> > affected by all of this.  OK, they build their Edgent-based app with some
> > tool that can utilize a maven repo.  Then what?
> >
> >         Today the story is simple: extract the binary-tgz (or a subset of
> > it) on the target (edge device), copy your app code jar/classes to the
> > target, set the CLASSPATH and go.  (assume we also eliminate
> > building/distributing a binary-tgz)
> >
> >         Can elaborate on that part of the story?
> >
> >         Thanks!
> >         — Dale
> >
> >
> >
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message