spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: [ANNOUNCE] Announcing Spark 1.5.1
Date Mon, 12 Oct 2015 04:35:41 GMT
*to not include binaries.

On Sun, Oct 11, 2015 at 9:35 PM, Patrick Wendell <pwendell@gmail.com> wrote:

> I think Daniel is correct here. The source artifact incorrectly includes
> jars. It is inadvertent and not part of our intended release process. This
> was something I noticed in Spark 1.5.0 and filed a JIRA and was fixed by
> updating our build scripts to fix it. However, our build environment was
> not using the most current version of the build scripts. See related links:
>
> https://issues.apache.org/jira/browse/SPARK-10511
> https://github.com/apache/spark/pull/8774/files
>
> I can update our build environment and we can repackage the Spark 1.5.1
> source tarball. To not include sources.
>
> - Patrick
>
> On Sun, Oct 11, 2015 at 8:53 AM, Sean Owen <sowen@cloudera.com> wrote:
>
>> Daniel: we did not vote on a tag. Please again read the VOTE email I
>> linked to you:
>>
>>
>> http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-5-1-RC1-tt14310.html#none
>>
>> among other things, it contains a link to the concrete source (and
>> binary) distribution under vote:
>>
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>>
>> You can still examine it, sure.
>>
>> Dependencies are *not* bundled in the source release. You're again
>> misunderstanding what you are seeing. Read my email again.
>>
>> I am still pretty confused about what the problem is. This is entirely
>> business as usual for ASF projects. I'll follow up with you offline if
>> you have any more doubts.
>>
>> On Sun, Oct 11, 2015 at 4:49 PM, Daniel Gruno <humbedooh@apache.org>
>> wrote:
>> > Here's my issue:
>> >
>> > How am I to audit that the dependencies you bundle are in fact what you
>> > claim they are?  How do I know they don't contain malware or - in light
>> > of recent events - emissions test rigging? ;)
>> >
>> > I am not interested in a git tag - that means nothing in the ASF voting
>> > process, you cannot vote on a tag, only on a release candidate. The VCS
>> > in use is irrelevant in this issue. If you can point me to a release
>> > candidate archive that was voted upon and does not contain binary
>> > applications, all is well.
>> >
>> > If there is no such thing, and we cannot come to an understanding, I
>> > will exercise my ASF Members' rights and bring this to the attention of
>> > the board of directors and ask for a clarification of the legality of
>> this.
>> >
>> > I find it highly irregular. Perhaps it is something some projects do in
>> > the Java community, but that doesn't make it permissible in my view.
>> >
>> > With regards,
>> > Daniel.
>> >
>> >
>> > On 10/11/2015 05:42 PM, Sean Owen wrote:
>> >> Still confused. Why are you saying we didn't vote on an archive? refer
>> >> to the email I linked, which includes both the git tag and a link to
>> >> all generated artifacts (also in my email).
>> >>
>> >> So, there are two things at play here:
>> >>
>> >> First, I am not sure what you mean that a source distro can't have
>> >> binary files. It's supposed to have the source code of Spark, and
>> >> shouldn't contain binary Spark. Nothing you listed are Spark binaries.
>> >> However, a distribution might have a lot of things in it that support
>> >> the source build, like copies of tools, test files, etc.  That
>> >> explains I think the first couple lines that you identified.
>> >>
>> >> Still, I am curious why you are saying that would invalidate a source
>> >> release? I have never heard anything like that.
>> >>
>> >> Second, I do think there are some binaries in here that aren't
>> >> supposed to be there, like the build/ directory stuff. IIRC these were
>> >> included accidentally and won't be in the next release. At least, I
>> >> don't see why they need to be bundled. These are just local copies of
>> >> third party tools though, and don't really matter. As it happens, the
>> >> licenses that get distributed with the source distro even cover all of
>> >> this stuff. I think that's not supposed to be there, but, also don't
>> >> see it's 'invalid' as a result.
>> >>
>> >>
>> >> On Sun, Oct 11, 2015 at 4:33 PM, Daniel Gruno <humbedooh@apache.org>
>> wrote:
>> >>> On 10/11/2015 05:29 PM, Sean Owen wrote:
>> >>>> Of course, but what's making you think this was a binary-only
>> >>>> distribution?
>> >>>
>> >>> I'm not saying binary-only, I am saying your source release contains
>> >>> binary programs, which would invalidate a release vote. Is there a
>> >>> release candidate package, that is voted on (saying you have a git tag
>> >>> does not satisfy this criteria, you need to vote on an actual archive
>> of
>> >>> files, otherwise there is no cogent proof of the release being from
>> that
>> >>> specific git tag).
>> >>>
>> >>> Here's what I found in your source release:
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/sql/hive/src/test/resources/data/files/TestSerDe.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>>
>> spark-1.5.1/sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/sql/hive/src/test/resources/TestUDTF.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/R/pkg/inst/test_support/sparktestjar_2.10-1.0.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/zinc-0.3.5.3/lib/scala-reflect.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/zinc-0.3.5.3/lib/sbt-interface.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/zinc-0.3.5.3/lib/compiler-interface-sources.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/zinc-0.3.5.3/lib/incremental-compiler.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/zinc-0.3.5.3/lib/scala-compiler.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/zinc-0.3.5.3/lib/zinc.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/zinc-0.3.5.3/lib/scala-library.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>>
>> spark-1.5.1/build/scala-2.10.4/misc/scala-devel/plugins/continuations.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/lib/scala-reflect.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/lib/akka-actors.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/lib/typesafe-config.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/lib/scala-actors-migration.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/lib/scala-actors.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/lib/scalap.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/lib/scala-swing.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/lib/scala-compiler.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/lib/scala-library.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/src/scala-reflect-src.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/src/scala-swing-src.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/src/scalap-src.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/src/scala-actors-src.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/src/scala-partest-src.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/src/scala-library-src.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/src/fjbg-src.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/src/scala-compiler-src.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/scala-2.10.4/src/msil-src.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/apache-maven-3.3.3/lib/guava-18.0.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/apache-maven-3.3.3/lib/wagon-http-2.9-shaded.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/apache-maven-3.3.3/lib/jsr250-api-1.0.jar
>> >>>
>> >>> Binary application (application/jar; charset=binary) found in
>> >>> spark-1.5.1/build/apache-maven-3.3.3/lib/javax.inject-1.jar
>> >>>
>> >>>
>> >>>
>> >>> The downloads page points you directly to the source
>> >>>> distro: http://spark.apache.org/downloads.html
>> >>>>
>> >>>> Look for the last vote, and you'll find it was of course a vote
on
>> >>>> source (and binary) artifacts:
>> >>>>
>> http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-5-1-RC1-tt14310.html#none
>> >>>>
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>> >>>>
>> >>>> On Sun, Oct 11, 2015 at 4:23 PM, Daniel Gruno <humbedooh@apache.org>
>> wrote:
>> >>>>> On 10/11/2015 05:12 PM, Sean Owen wrote:
>> >>>>>> The Spark releases include a source distribution and several
binary
>> >>>>>> distributions. This is pretty normal for Apache projects.
What are
>> you
>> >>>>>> referring to here?
>> >>>>>
>> >>>>> Surely the _source_ distribution does not contain binaries?
How
>> else can
>> >>>>> you vote on a release if you don't know what it contains?
>> >>>>>
>> >>>>> You can produce convenience downloads that contain binary files,
>> yes,
>> >>>>> but surely you need a source-only package which is the one you
vote
>> on,
>> >>>>> that does not contain any binaries. Do you have such a thing?
And
>> where
>> >>>>> may I find it?
>> >>>>>
>> >>>>> With regards,
>> >>>>> Daniel.
>> >>>>>
>> >>>>>>
>> >>>>>> On Sun, Oct 11, 2015 at 3:26 PM, Daniel Gruno <
>> humbedooh@apache.org> wrote:
>> >>>>>>> Out of curiosity: How can you vote on a release that
contains 34
>> binary files? Surely a source code release should only contain source code
>> and not binaries, as you cannot verify the content of these.
>> >>>>>>>
>> >>>>>>> Looking forward to a response.
>> >>>>>>>
>> >>>>>>> With regards,
>> >>>>>>> Daniel.
>> >>>>>>>
>> >>>>>>> On 10/2/2015, 4:42:31 AM, Reynold Xin <rxin@databricks.com>
>> wrote:
>> >>>>>>>> Hi All,
>> >>>>>>>>
>> >>>>>>>> Spark 1.5.1 is a maintenance release containing
stability fixes.
>> This
>> >>>>>>>> release is based on the branch-1.5 maintenance branch
of Spark.
>> We
>> >>>>>>>> *strongly recommend* all 1.5.0 users to upgrade
to this release.
>> >>>>>>>>
>> >>>>>>>> The full list of bug fixes is here:
>> http://s.apache.org/spark-1.5.1
>> >>>>>>>>
>> >>>>>>>> http://spark.apache.org/releases/spark-release-1-5-1.html
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> (note: it can take a few hours for everything to
be propagated,
>> so you
>> >>>>>>>> might get 404 on some download links, but everything
should be
>> in maven
>> >>>>>>>> central already)
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> ---------------------------------------------------------------------
>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>> >>>>>>>
>> >>>>>
>> >>>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>

Mime
View raw message