Mailing-List: contact dev-help@river.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@river.apache.org
Received-SPF: pass (nike.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\))
Subject: Build system and Java 8 was: Re: [Vote] (RIVER-432) Jar files in svn
 and src distributions
From: Greg Trasuk <trasukg@stratuscom.com>
In-Reply-To: <1392114957.6766.11.camel@Nokia-N900>
Date: Tue, 11 Feb 2014 11:01:37 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <299ED069-7474-4CAF-BFEF-D5536DD003C2@stratuscom.com>
References: <JIRA.12685437.1387394252165@arcas>
 <JIRA.12685437.1387394252165.31497.1388682352223@arcas>
 <39A69BFC-70DE-4471-A031-85C0BA225C59@stratuscom.com>
 <52C69018.6060101@qcg.nl>
 <2FFCF0E9-39DE-468C-A064-5DDDC19FCA03@stratuscom.com>
 <7DC70848-B027-43E3-922F-A4CAF27CED04@stratuscom.com>
 <1392114957.6766.11.camel@Nokia-N900>
To: dev@river.apache.org,
 Peter <jini@zeus.net.au>


Hi Peter:

I=92m not familiar with Java 8 yet.  How is the current build not =
supported?

Cheers,

Greg.

On Feb 11, 2014, at 5:35 AM, Peter <jini@zeus.net.au> wrote:

> +1 Peter.
>=20
> Note that the current build system does not support java 8.  =20
>=20
> Maintaining a build system is a significant burden ( I know I had to =
implement Java 5 support).
>=20
> We will need a new build system for java 8, this looks like a step in =
that direction.  In reality we need to adopt the build conventions used =
by Rio.
>=20
> Regards,
>=20
> Peter.
> ----- Original message -----
>>=20
>> As discussion has settled somewhat, I would like to call another vote =
to
>> accept the latest patch described in=20
>> https://issues.apache.org/jira/browse/RIVER-432
>>=20
>> The patch removes the archived jar files for Velocity and ASM and
>> replaces them with Apache Ivy scripts that download the jars from =
Maven
>> Central the first time a build is done.   =46rom then on, the jar =
files are
>> in Ivy=92s repository (for more info, see http://ant.apache.org/ivy).
>>=20
>> Voting will remain open at least until 2000 UTC Feb 13, 2014.
>>=20
>> Cheers,
>>=20
>> Greg.
>>=20
>>=20
>> On Jan 3, 2014, at 12:57 PM, Greg Trasuk <trasukg@stratuscom.com> =
wrote:
>>=20
>>>=20
>>> On Jan 3, 2014, at 5:25 AM, Simon IJskes - QCG <simon@qcg.nl> wrote:
>>>=20
>>>> In order to gain some time to discuss this first i will vote -1.
>>>>=20
>>>> First, we decided to NOT remove velocity builder.
>>>=20
>>> When I read the email chain, my impression was that we wanted to
>>> remove it (to quote you Sim, =93To be honest, I hate it=94), but =
there was
>>> a dependency on it in the =91extras=92 folder that was added in the =
trunk
>>> branch.   As there is no =91extras=92 in the 2.2 branch, and that is =
what
>>> this patch applies to, I thought it was fair to remove
>>> VelocityConfigurationBuilder from the 2.2 branch.     Perhaps we =
should
>>> revisit the ConfigurationBuilder approach in another thread.   For =
now
>>> I=92ll spin another patch that doesn=92t remove
>>> VelocityConfigurationBuilder.
>>>=20
>>>>=20
>>>> Second, no need to remove the jars as specified in your own =
comments
>>>> on RIVER-432.
>>>>=20
>>>> Pulling in external jars at compile time, shall we start here?
>>>>=20
>>>> They are already in the svn. They are already in the build scripts.
>>>> What does this patch fix? No legal problems?
>>>>=20
>>>=20
>>> Apache policy is somewhat unclear on this point.   One needs to =
examine
>>> the mailing lists for clues on what we should really do.   I will =
argue
>>> that:
>>>=20
>>> 1 - The fundamental distribution model of Apache is source code, not
>>> binaries. 2 - Distributing binaries is tolerated but not encouraged.=20=

>>> Since the svn repository can be seen as a distribution point, =
binaries
>>> in svn are also tolerated but not encouraged. 3 - Downloading
>>> dependency binaries at build time is technologically easy, provides
>>> the same guarantees as putting them in cvs, and avoids the question =
of
>>> effectively distributing someone else=92s code.
>>>=20
>>> All these together suggest that although we=92re technically OK to =
put
>>> dependency jars in a =93-deps=94 package (note that the status quo =
_is_
>>> unacceptable - at the very least, we need to restructure the
>>> dependencies into a =93-deps=94 binary package), there is some =
policy
>>> uncertainty which we avoid totally by having dependencies downloaded
>>> from a known-good source at build time.
>>>=20
>>> Let=92s examine these points:
>>>=20
>>> 1 - The fundamental distribution model of Apache is source code, not
>>> binaries.   Apache began with httpd.   Back in those days, =93Open =
Source=94
>>> software was distributed in source form only, simply because it was
>>> mostly intended for Unix systems (then later Linux).   I recall the
>>> first release of Perl coming down as a ten-part uunet news message.=20=

>>> Part of this distribution model was practical necessity - System
>>> differences made it necessary to compile your software on the =
hardware
>>> it was going to run on.   Part of it was open-source philosophy.=20
>>> Having the source code meant that you could take a look at it and
>>> verify that it wasn=92t hazardous to your operations.  =20
>>>=20
>>> In any case, the way we use to use open source software was
>>> (=93./configure; make; make install=94).   If the software had
>>> dependencies, you built them from source, for the same reasons.
>>>=20
>>> Now, as time has gone on, we=92ve gotten used to having the JVM as a
>>> common runtime, and jar files as a common binary distribution =
medium.=20
>>> But the Apache Foundation=92s mandate is to produce open source =
software
>>> that is freely usable under the Apache License.   That means source
>>> code is at the heart of Apache, despite the rest of the world=92s
>>> comfort with binaries.   Hence Roy=92s statements in (1):
>>>=20
>>>> Class files are not open source.   Jar files filled with class =
files
>>>> are not open source.   The fact that they are derived from open =
source
>>>> is applicable only to what we allow projects to be dependent upon,
>>>> not what we vote on as a release package.   Release votes are on
>>>> verified open source artifacts.   Binary packages are separate from
>>>> source packages. One cannot vote to approve a release containing a
>>>> mix of source and binary code because the binary is not open source
>>>> and cannot be verified to be safe for release (even if it was
>>>> derived from open source).
>>>>=20
>>>> I thought that was frigging obvious.   Why do I need to write
>>>> documentation to explain something that is fundamental to the open
>>>> source definition?
>>> He=92s talking about binary packages, not jar files in svn, but I =
read
>>> that (and many other emails) as a distaste for binary distributions.
>>>=20
>>> In fact, if you look at Apache httpd=92s download page, it doesn=92t
>>> appear that the Apache project publishes any Unix or Linux binaries.=20=

>>> They leave that to other organizations.
>>>=20
>>> 2 - Distributing binaries is tolerated but not encouraged.   Since =
the
>>> svn repository can be seen as a distribution point, binaries in svn
>>> are also tolerated but not encouraged.
>>>=20
>>> It=92s hard to find a single reference that encapsulates this =
outlook,
>>> but that=92s the impression I get from reading the various mailing
>>> lists.   For instance, Sam Ruby says (2):
>>>> IMO, our projects release source. So, our projects should not
>>>> maintain object/binary artifacts in their svn release tree,
>>>> regardless of license (category a or b).
>>> There is some debate on whether the svn tree should be considered a
>>> distribution point.   Incubator releases are regularly called out =
for
>>> not having =93NOTICE=94 and =93RELEASE=94 files at all reasonable =
checkout
>>> points in svn.   [LEGAL-26]
>>> (https://issues.apache.org/jira/browse/LEGAL-26) concerns this and
>>> remains unresolved.
>>>=20
>>> Doug Cutting (3) says:
>>>> On Mon, Sep 16, 2013 at 2:50 AM, Stephen Connolly
>>>> <stephen.alan.connolly@gmail.com> wrote:
>>>>> * Source control is not an Apache distribution and hence we do not
>>>>> need to have LICENSE and NOTICE files in source control, it can be
>>>>> a nice convenience, but there is no *requirement*.
>>>>=20
>>>> I think perhaps you're looking for clear lines where things are
>>>> actually a bit fuzzy.   Certainly releases are official =
distributions
>>>> and need LICENSE and NOTICE files.   That line is clear.   On the =
other
>>>> hand, we try to discourage folks from thinking that source control =
is
>>>> a distribution.   Rather we wish it to be considered our shared
>>>> workspace, containing works in progress, not yet always ready for
>>>> distribution to folks outside the foundation.   But, since we work =
in
>>>> public, folks from outside the foundation can see our shared
>>>> workspace and might occasionally mistake it for an official
>>>> distribution.   We'd like them to still see a LICENSE and NOTICE
>>>> file.   So it's not a hard-and-fast requirement that every tree =
that
>>>> can possibly be checked out have a LICENSE and NOTICE file at its
>>>> root, but it's a good practice for those trees that are likely to =
be
>>>> checked out have them, so that folks who might consume them are =
well
>>>> informed.
>>> Again, he=92s not talking directly about jar files in svn, however I
>>> think his statement that =93since we work in public, folks from =
outside
>>> the foundation can see our shared workspace and might occasionally
>>> mistake it for an official distribution=94 applies here.   =
Fundamentally,
>>> the decision on how and where to distribute =91velocity.jar=92 =
rightly
>>> belongs with the Velocity group and I don=92t think we ought to
>>> redistribute it.
>>>=20
>>> 3 - Downloading dependency binaries at build time is technologically
>>> easy, provides the same guarantees as putting them in cvs, and =
avoids
>>> the question of effectively distributing someone else=92s code.
>>>=20
>>> There doesn=92t seem to be clear policy in the ASF on this, as =
evidenced
>>> by the frequent debates on it, and the lack of documentation.   I=92ve=

>>> tried to lay out an argument that having jars in svn is not =
encouraged
>>> by the ASF (really, it=92s not in line with the ASF=92s charter), =
even if
>>> it isn=92t disallowed.   You may disagree, and I won=92t claim I=92ve =
made a
>>> strong argument, simply because the policy isn=92t clear.   So =
instead of
>>> going through arguments that amount to differences of opinion on
>>> Apache policy, let=92s use a technological solution that is simple,
>>> common, and avoids the question altogether, by automatically
>>> downloading the dependencies at build time.
>>>=20
>>> Projects that use Maven do this automatic download as standard
>>> practice (that=92s what Maven does, and that=92s what the Maven =
Central
>>> infrastructure is there to support).   We don=92t use Maven, which =
is
>>> fine (our customers have asked us to make our binaries available in
>>> Maven Central, and we=92ve done that).   Apache Ivy is a popular =
add-on
>>> to Apache Ant that provides similar dependency resolution to an
>>> Ant-based build.
>>>=20
>>> I was a little surprised how easy it was to persuade Ivy to get the
>>> required dependencies at build time.   The =93ivy.xml=94 file is 39 =
lines
>>> including the ASL header (which by the way I forgot to include in =
the
>>> patch - I=92ll fix that).   There are about 50 lines added to =
=91build.xml=92
>>> to download Ivy and then download the required jar files
>>>=20
>>> So, given that the status-quo seems to be unacceptable (Roy talks
>>> about not having jar files in the open-source trees, only in =93-deps=94=

>>> and =93tools=94 trees), we have two options:
>>>=20
>>> (a) - restructure the svn repository and the build to allow a =
separate
>>> =93-deps=94 distribution.   This wouldn=92t affect our binary =
distributions
>>> (note that I=92m no longer using the term =93binary release=94), but =
to
>>> build from source, a user would have to download a separate archive,
>>> unpack it, and then copy those files into the directory that was
>>> unpacked from the source release.   This option effectively still =
has
>>> us distributing dependent binaries, which is not the goal of the =
ASF,
>>> just with a disclaimer that says =93this isn=92t an ASF release, its =
just
>>> a binary distribution put together by a committer for your
>>> convenience, so don=92t feel you should trust it too much=94.
>>>=20
>>> (b) - use Ivy to get the jars from Maven Central automatically as =
part
>>> of the build.
>>>=20
>>> I think (b) is the option that causes the least hassle for our
>>> downstream consumers, and not much hassle for us.
>>>=20
>>>=20
>>>> Pulling external jars at compile time also makes it more difficult
>>>> to certify the software. In order to certify the software you need
>>>> to establish baseline that will be garanteed the same, even if you
>>>> pull it from the archive 10 years later.
>>>=20
>>> As I said above, Apache=92s focus is creating software that can be =
built
>>> from source, not distributing binaries (note that QCG or another
>>> company might have a different focus, and is perfectly able to
>>> distribute binaries under the Apache license).   I think a =
reasonably
>>> prudent user would ask =93How can I trust the =91velocity.jar=92 =
that=92s
>>> included in this binary?=94   And the answer would be either =
=93because I
>>> built it from source and installed it in my corporate repository=94
>>> (very cautious, but not unheard-of) or =93It was published by the
>>> Velocity group to a trusted repository, Maven Central=94 (more =
common).
>>>=20
>>> If you look in the =93ivy.xml=94 file you=92ll see that the =
dependencies are
>>> specified using Maven-style =93group-artifact-version=94 =
coordinates.   Old
>>> versions are maintained in Maven Central forever.   I suppose it=92s
>>> possible that a publisher could convince Maven Central to remove a
>>> version for some reason (security or licensing problems perhaps), =
but
>>> then, would we want to be distributing that version in a =93-deps=94
>>> package?
>>>=20
>>> I agree that it=92s not enough to just say =93you need to download
>>> such-and-such jar=94, hence the automatic download managed by =93Ivy=94=
 from
>>> Maven Central.
>>>=20
>>>> It is not a high level project that builds on several frameworks. =
It
>>>> is a lowlevel system library. The stuff below the stack is minimal.
>>>> The number of jars we use is limited. Why bother?
>>>>=20
>>>=20
>>> In the currently released branches, the dependencies are limited to
>>> ASM and Velocity.   Looking forward to the trunk branch and the
>>> qa_refactor branch, the number of external dependencies seem to be
>>> increasing (IMO I don=92t like that, because I also view River as a =
low
>>> level system library, but I=92m only one PMC member).   We need to =
get in
>>> front of the problem before we start distributing large numbers of
>>> dependencies.
>>>=20
>>> This point rolls in with the general question of jar files in =
version
>>> control.   I was always taught that version control was for source
>>> code, and putting binaries into version control was a bad idea.   In
>>> addition, there are practical problems - with older systems like =
cvs,
>>> even doing an update or commit effectively downloads the binaries,
>>> which slows things down if there are large binary files.   On newer
>>> distributed version control systems like git or Mercurial, the =
entire
>>> repository, including all versions of binary artifacts, comes down
>>> with the project checkout.   Currently, we have one version of
>>> relatively few jar files in our repository, so it=92s not a major =
issue.
>>>    But it gets worse as time goes on.   So I suggest we work out the
>>> technology now to avoid the problem.
>>>=20
>>>> Gr. Simon
>>>>=20
>>>=20
>>> Thanks for the questions, Sim.   I hope you=92ll come around to =
removing
>>> your =91-1=92.
>>>=20
>>> Cheers,
>>>=20
>>> Greg
>>>=20
>>> Footnotes
>>> =97=97=97=97=97=97
>>>=20
>>> (1) - Roy Fielding - http://s.apache.org/roy-binary-deps-1
>>> (2) - Sam Ruby - http://s.apache.org/r5
>>> (3) - Doug Cutting - http://s.apache.org/GNP
>>>=20
>>>> On 02-01-14 18:22, Greg Trasuk wrote:
>>>>>=20
>>>>> Hello all:
>>>>>=20
>>>>> Please have a look at the patch mentioned below and cast a vote on
>>>>> it.
>>>>>=20
>>>>> The main idea is to remove the dependency jar files from the
>>>>> source distribution.   As a side effect of using Ivy, it becomes
>>>>> reasonable to remove them from the svn archive as well.   Also, =
the
>>>>> Velocity dependency was there to support the
>>>>> VelocityConfigurationBuilder.   We had discussed removing that
>>>>> component, so rather than move that dependency to Ivy, I=92ve
>>>>> removed VelocityConfigurationBuilder.
>>>>>=20
>>>>> It=92s arguable whether the VelocityConfigurationBuider was part =
of
>>>>> the official Jini API (I see it as a utility, not API), so I don=92t=

>>>>> think this commit actually requires a vote.   However, it does =
seem
>>>>> like a significant change to the build process that ought to be
>>>>> reviewed.   So I propose to treat this as a =93lazy consensus=94 =
vote,
>>>>> and will commit the change to the 2.2 branch if there are no
>>>>> objections in 72 hours (i.e. 1730UTC 20140105).
>>>>>=20
>>>>> At the same time, based on discussions over on
>>>>> general@incubator.apache.org, I=92ll withdraw my assertion that we
>>>>> can=92t have jars in svn.   Those interested may want to check out
>>>>> the thread at
>>>>> =
http://mail-archives.apache.org/mod_mbox/incubator-general/201312.mbox/%3C=
01B04CC4-95B8-4A39-BC16-04BAA4269B65%40stratuscom.com%3E
>>>>>=20
>>>>> Cheers,
>>>>>=20
>>>>> Greg.
>>>>>=20
>>>>> On Jan 2, 2014, at 12:05 PM, Greg Trasuk (JIRA) <jira@apache.org>
>>>>> wrote:
>>>>>=20
>>>>>>=20
>>>>>> [
>>>>>> =
https://issues.apache.org/jira/browse/RIVER-432?page=3Dcom.atlassian.jira.=
plugin.system.issuetabpanels:all-tabpanel
>>>>>> ]
>>>>>>=20
>>>>>> Greg Trasuk updated RIVER-432:
>>>>>> ------------------------------
>>>>>>=20
>>>>>> Attachment: river-2_2_remove_jars.diff
>>>>>>=20
>>>>>> The attached patch for the 2.2 branch does the following:
>>>>>> - removes the 'asm' directory and 'tests/lib' directories which
>>>>>> currently contain the asm library, mockito, and junit jars. -
>>>>>> Modifies 'build.xml', 'common.xml', and adds 'ivy.xml' so that
>>>>>> the Apache Ivy ant plugin is downloaded at build time, and then
>>>>>> used to retrieve the libraries mentioned above from Maven
>>>>>> Central.   This removes the need to have the jar files in svn. -
>>>>>> Removes (as per discussion
>>>>>> =
http://mail-archives.apache.org/mod_mbox/river-dev/201211.mbox/%3C509B99E3=
.6080800%40qcg.nl%3E)
>>>>>> the VelocityConfigBuilder, and associated Velocity jars.   Note
>>>>>> that the 'extras' folder is not present in the 2.2 branch, so
>>>>>> Sim's last comments in the thread do not apply.
>>>>>>=20
>>>>>>> Jar files in svn and src distributions
>>>>>>> --------------------------------------
>>>>>>>=20
>>>>>>> Key: RIVER-432
>>>>>>> URL: https://issues.apache.org/jira/browse/RIVER-432
>>>>>>> Project: River
>>>>>>> Issue Type: Bug
>>>>>>> Reporter: Greg Trasuk
>>>>>>> Attachments: river-2_2_remove_jars.diff
>>>>>>>=20
>>>>>>>=20
>>>>>>> Recent traffic on the incubator lists has pointed out that
>>>>>>> including jar files for dependencies in the subversion
>>>>>>> repository and the source distributions is against Apache
>>>>>>> policy. In River, the following libraries appear in the
>>>>>>> Subversion repository and the source distributions (these are
>>>>>>> from trunk, a smaller set appear in the 2.2 branch):
>>>>>>> animal-sniffer asm bouncy-castle dnsjava high-scale-lib
>>>>>>> rc-libs
>>>>>>> velocity
>>>>>>> They all have to go.   What are we using them for?   As I
>>>>>>> understand it, we were going to remove the
>>>>>>> VelocityConfigurationBuilder, so that's not a problem.   Some
>>>>>>> of the others are available from Maven Central, so we can get
>>>>>>> them at build time using Ivy or another build tool.   Which
>>>>>>> ones are actually required?   And where did they come from?
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> --
>>>>>> This message was sent by Atlassian JIRA
>>>>>> (v6.1.5#6160)
>>>>>=20
>>>>=20
>>>>=20
>>>> --=20
>>>> QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl
>>>> Quality Consultancy Group b.v., Leiderdorp, Kvk Den Haag: 28088397
>>>=20
>>=20
>=20