spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <lian.cs....@gmail.com>
Subject Re: [VOTE] Release Apache Spark 1.1.0 (RC2)
Date Fri, 29 Aug 2014 18:52:35 GMT
Just noticed one thing: although --with-hive is deprecated by -Phive,
make-distribution.sh still relies on $SPARK_HIVE (which was controlled by
--with-hive) to determine whether to include datanucleus jar files. This
means we have to do something like SPARK_HIVE=true ./make-distribution.sh
... to enable Hive support. Otherwise datanucleus jars are not included in
lib/.

This issue is similar to SPARK-3234
<https://issues.apache.org/jira/browse/SPARK-3234>, both
SPARK_HADOOP_VERSION and SPARK_HIVE are controlled by some deprecated
command line options.
​


On Fri, Aug 29, 2014 at 11:18 AM, Patrick Wendell <pwendell@gmail.com>
wrote:

> Oh darn - I missed this update. GRR, unfortunately I think this means
> I'll need to cut a new RC. Thanks for catching this Nick.
>
> On Fri, Aug 29, 2014 at 10:18 AM, Nicholas Chammas
> <nicholas.chammas@gmail.com> wrote:
> > [Let me know if I should be posting these comments in a different
> thread.]
> >
> > Should the default Spark version in spark-ec2 be updated for this
> release?
> >
> > Nick
> >
> >
> >
> > On Fri, Aug 29, 2014 at 12:55 PM, Patrick Wendell <pwendell@gmail.com>
> > wrote:
> >>
> >> Hey Nicholas,
> >>
> >> Thanks for this, we can merge in doc changes outside of the actual
> >> release timeline, so we'll make sure to loop those changes in before
> >> we publish the final 1.1 docs.
> >>
> >> - Patrick
> >>
> >> On Fri, Aug 29, 2014 at 9:24 AM, Nicholas Chammas
> >> <nicholas.chammas@gmail.com> wrote:
> >> > There were several formatting and typographical errors in the SQL docs
> >> > that
> >> > I've fixed in this PR. Dunno if we want to roll that into the release.
> >> >
> >> >
> >> > On Fri, Aug 29, 2014 at 12:17 PM, Patrick Wendell <pwendell@gmail.com
> >
> >> > wrote:
> >> >>
> >> >> Okay I'll plan to add cdh4 binary as well for the final release!
> >> >>
> >> >> ---
> >> >> sent from my phone
> >> >> On Aug 29, 2014 8:26 AM, "Ye Xianjin" <advancedxy@gmail.com>
wrote:
> >> >>
> >> >> > We just used CDH 4.7 for our production cluster. And I believe
we
> >> >> > won't
> >> >> > use CDH 5 in the next year.
> >> >> >
> >> >> > Sent from my iPhone
> >> >> >
> >> >> > > On 2014年8月29日, at 14:39, Matei Zaharia <matei.zaharia@gmail.com>
> >> >> > > wrote:
> >> >> > >
> >> >> > > Personally I'd actually consider putting CDH4 back if there
are
> >> >> > > still
> >> >> > users on it. It's always better to be inclusive, and the
> convenience
> >> >> > of
> >> >> > a
> >> >> > one-click download is high. Do we have a sense on what % of CDH
> users
> >> >> > still
> >> >> > use CDH4?
> >> >> > >
> >> >> > > Matei
> >> >> > >
> >> >> > > On August 28, 2014 at 11:31:13 PM, Sean Owen (sowen@cloudera.com
> )
> >> >> > > wrote:
> >> >> > >
> >> >> > > (Copying my reply since I don't know if it goes to the mailing
> >> >> > > list)
> >> >> > >
> >> >> > > Great, thanks for explaining the reasoning. You're saying
these
> >> >> > > aren't
> >> >> > > going into the final release? I think that moots any issue
> >> >> > > surrounding
> >> >> > > distributing them then.
> >> >> > >
> >> >> > > This is all I know of from the ASF:
> >> >> > > https://community.apache.org/projectIndependence.html I don't
> read
> >> >> > > it
> >> >> > > as expressly forbidding this kind of thing although you can
see
> how
> >> >> > > it
> >> >> > > bumps up against the spirit. There's not a bright line --
what
> >> >> > > about
> >> >> > > Tomcat providing binaries compiled for Windows for example?
does
> >> >> > > that
> >> >> > > favor an OS vendor?
> >> >> > >
> >> >> > > From this technical ASF perspective only the releases matter
--
> do
> >> >> > > what you want with snapshots and RCs. The only issue there
is
> maybe
> >> >> > > releasing something different than was in the RC; is that
at all
> >> >> > > confusing? Just needs a note.
> >> >> > >
> >> >> > > I think this theoretical issue doesn't exist if these binaries
> >> >> > > aren't
> >> >> > > released, so I see no reason to not proceed.
> >> >> > >
> >> >> > > The rest is a different question about whether you want to
spend
> >> >> > > time
> >> >> > > maintaining this profile and candidate. The vendor already
> manages
> >> >> > > their build I think and -- and I don't know -- may even prefer
> not
> >> >> > > to
> >> >> > > have a different special build floating around. There's also
the
> >> >> > > theoretical argument that this turns off other vendors from
> >> >> > > adopting
> >> >> > > Spark if it's perceived to be too connected to other vendors.
I'd
> >> >> > > like
> >> >> > > to maximize Spark's distribution and there's some argument
you do
> >> >> > > this
> >> >> > > by not making vendor profiles. But as I say a different question
> to
> >> >> > > just think about over time...
> >> >> > >
> >> >> > > (oh and PS for my part I think it's a good thing that CDH4
> binaries
> >> >> > > were removed. I wasn't arguing for resurrecting them)
> >> >> > >
> >> >> > >> On Fri, Aug 29, 2014 at 7:26 AM, Patrick Wendell
> >> >> > >> <pwendell@gmail.com>
> >> >> > wrote:
> >> >> > >> Hey Sean,
> >> >> > >>
> >> >> > >> The reason there are no longer CDH-specific builds is
that all
> >> >> > >> newer
> >> >> > >> versions of CDH and HDP work with builds for the upstream
Hadoop
> >> >> > >> projects. I dropped CDH4 in favor of a newer Hadoop version
> (2.4)
> >> >> > >> and
> >> >> > >> the Hadoop-without-Hive (also 2.4) build.
> >> >> > >>
> >> >> > >> For MapR - we can't officially post those artifacts on
ASF web
> >> >> > >> space
> >> >> > >> when we make the final release, we can only link to them
as
> being
> >> >> > >> hosted by MapR specifically since they use non-compatible
> >> >> > >> licenses.
> >> >> > >> However, I felt that providing these during a testing
period was
> >> >> > >> alright, with the goal of increasing test coverage. I
couldn't
> >> >> > >> find
> >> >> > >> any policy against posting these on personal web space
during RC
> >> >> > >> voting. However, we can remove them if there is one.
> >> >> > >>
> >> >> > >> Dropping CDH4 was more because it is now pretty old,
but we can
> >> >> > >> add
> >> >> > >> it
> >> >> > >> back if people want. The binary packaging is a slightly
separate
> >> >> > >> question from release votes, so I can always add more
binary
> >> >> > >> packages
> >> >> > >> whenever. And on this, my main concern is covering the
most
> >> >> > >> popular
> >> >> > >> Hadoop versions to lower the bar for users to build and
test
> >> >> > >> Spark.
> >> >> > >>
> >> >> > >> - Patrick
> >> >> > >>
> >> >> > >>> On Thu, Aug 28, 2014 at 11:04 PM, Sean Owen <
> sowen@cloudera.com>
> >> >> > wrote:
> >> >> > >>> +1 I tested the source and Hadoop 2.4 release. Checksums
and
> >> >> > >>> signatures are OK. Compiles fine with Java 8 on OS
X. Tests...
> >> >> > >>> don't
> >> >> > >>> fail any more than usual.
> >> >> > >>>
> >> >> > >>> FWIW I've also been using the 1.1.0-SNAPSHOT for
some time in
> >> >> > >>> another
> >> >> > >>> project and have encountered no problems.
> >> >> > >>>
> >> >> > >>>
> >> >> > >>> I notice that the 1.1.0 release removes the CDH4-specific
> build,
> >> >> > >>> but
> >> >> > >>> adds two MapR-specific builds. Compare with
> >> >> > >>> https://dist.apache.org/repos/dist/release/spark/spark-1.0.2/
> I
> >> >> > >>> commented on the commit:
> >> >> > >>>
> >> >> >
> >> >> >
> >> >> >
> https://github.com/apache/spark/commit/ceb19830b88486faa87ff41e18d03ede713a73cc
> >> >> > >>>
> >> >> > >>> I'm in favor of removing all vendor-specific builds.
This
> change
> >> >> > >>> *looks* a bit funny as there was no JIRA (?) and
appears to
> swap
> >> >> > >>> one
> >> >> > >>> vendor for another. Of course there's nothing untoward
going
> on,
> >> >> > >>> but
> >> >> > >>> what was the reasoning? It's best avoided, and MapR
already
> >> >> > >>> distributes Spark just fine, no?
> >> >> > >>>
> >> >> > >>> This is a gray area with ASF projects. I mention
it as well
> >> >> > >>> because
> >> >> > >>> it
> >> >> > >>> came up with Apache Flink recently
> >> >> > >>> (
> >> >> >
> >> >> >
> >> >> >
> http://mail-archives.eu.apache.org/mod_mbox/incubator-flink-dev/201408.mbox/%3CCANC1h_u%3DN0YKFu3pDaEVYz5ZcQtjQnXEjQA2ReKmoS%2Bye7%3Do%3DA%40mail.gmail.com%3E
> >> >> > )
> >> >> > >>> Another vendor rightly noted this could look like
favoritism.
> >> >> > >>> They
> >> >> > >>> changed to remove vendor releases.
> >> >> > >>>
> >> >> > >>>> On Fri, Aug 29, 2014 at 3:14 AM, Patrick Wendell
> >> >> > >>>> <pwendell@gmail.com>
> >> >> > wrote:
> >> >> > >>>> Please vote on releasing the following candidate
as Apache
> Spark
> >> >> > version 1.1.0!
> >> >> > >>>>
> >> >> > >>>> The tag to be voted on is v1.1.0-rc2 (commit
711aebb3):
> >> >> > >>>>
> >> >> >
> >> >> >
> >> >> >
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=711aebb329ca28046396af1e34395a0df92b5327
> >> >> > >>>>
> >> >> > >>>> The release files, including signatures, digests,
etc. can be
> >> >> > >>>> found
> >> >> > at:
> >> >> > >>>> http://people.apache.org/~pwendell/spark-1.1.0-rc2/
> >> >> > >>>>
> >> >> > >>>> Release artifacts are signed with the following
key:
> >> >> > >>>> https://people.apache.org/keys/committer/pwendell.asc
> >> >> > >>>>
> >> >> > >>>> The staging repository for this release can be
found at:
> >> >> > >>>>
> >> >> >
> >> >> >
> https://repository.apache.org/content/repositories/orgapachespark-1029/
> >> >> > >>>>
> >> >> > >>>> The documentation corresponding to this release
can be found
> at:
> >> >> > >>>> http://people.apache.org/~pwendell/spark-1.1.0-rc2-docs/
> >> >> > >>>>
> >> >> > >>>> Please vote on releasing this package as Apache
Spark 1.1.0!
> >> >> > >>>>
> >> >> > >>>> The vote is open until Monday, September 01,
at 03:11 UTC and
> >> >> > >>>> passes
> >> >> > if
> >> >> > >>>> a majority of at least 3 +1 PMC votes are cast.
> >> >> > >>>>
> >> >> > >>>> [ ] +1 Release this package as Apache Spark 1.1.0
> >> >> > >>>> [ ] -1 Do not release this package because ...
> >> >> > >>>>
> >> >> > >>>> To learn more about Apache Spark, please see
> >> >> > >>>> http://spark.apache.org/
> >> >> > >>>>
> >> >> > >>>> == Regressions fixed since RC1 ==
> >> >> > >>>> LZ4 compression issue:
> >> >> > https://issues.apache.org/jira/browse/SPARK-3277
> >> >> > >>>>
> >> >> > >>>> == What justifies a -1 vote for this release?
==
> >> >> > >>>> This vote is happening very late into the QA
period compared
> >> >> > >>>> with
> >> >> > >>>> previous votes, so -1 votes should only occur
for significant
> >> >> > >>>> regressions from 1.0.2. Bugs already present
in 1.0.X will not
> >> >> > >>>> block
> >> >> > >>>> this release.
> >> >> > >>>>
> >> >> > >>>> == What default changes should I be aware of?
==
> >> >> > >>>> 1. The default value of "spark.io.compression.codec"
is now
> >> >> > >>>> "snappy"
> >> >> > >>>> --> Old behavior can be restored by switching
to "lzf"
> >> >> > >>>>
> >> >> > >>>> 2. PySpark now performs external spilling during
aggregations.
> >> >> > >>>> --> Old behavior can be restored by setting
> >> >> > >>>> "spark.shuffle.spill"
> >> >> > >>>> to
> >> >> > "false".
> >> >> > >>>>
> >> >> > >>>>
> >> >> > >>>>
> >> >> > >>>>
> ---------------------------------------------------------------------
> >> >> > >>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> >> > >>>> For additional commands, e-mail: dev-help@spark.apache.org
> >> >> > >
> >> >> > >
> >> >> > >
> ---------------------------------------------------------------------
> >> >> > > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> >> > > For additional commands, e-mail: dev-help@spark.apache.org
> >> >> > >
> >> >> >
> >> >
> >> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message