spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Chan ...@ooyala.com>
Subject Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)
Date Thu, 12 Dec 2013 16:59:00 GMT
I'd be personally fine with a standard workflow of assemble-deps +
packaging just the Spark files as separate packages, if it speeds up
everyone's development time.


On Wed, Dec 11, 2013 at 1:10 PM, Mark Hamstra <mark@clearstorydata.com>wrote:

> I don't know how to make sense of the numbers, but here's what I've got
> from a very small sample size.
>
> For both v0.8.0-incubating and v0.8.1-incubating, building separate
> assemblies is faster than `./sbt/sbt assembly` and the times for building
> separate assemblies for 0.8.0 and 0.8.1 are about the same.
>
> For v0.8.0-incubating, `./sbt/sbt assembly` takes about 2.5x as long as the
> sum of the separate assemblies.
> For v0.8.1-incubating, `./sbt/sbt assembly` takes almost 8x as long as the
> sum of the separate assemblies.
>
> Weird.
>
>
> On Wed, Dec 11, 2013 at 11:49 AM, Patrick Wendell <pwendell@gmail.com
> >wrote:
>
> > I'll +1 myself also.
> >
> > For anyone who has the slow build problem: does this issue happen when
> > building v0.8.0-incubating also? Trying to figure out whether it's
> > related to something we added in 0.8.1 or if it's a long standing
> > issue.
> >
> > - Patrick
> >
> > On Wed, Dec 11, 2013 at 10:39 AM, Matei Zaharia <matei.zaharia@gmail.com
> >
> > wrote:
> > > Woah, weird, but definitely good to know.
> > >
> > > If you’re doing Spark development, there’s also a more convenient
> option
> > added by Shivaram in the master branch. You can do sbt assemble-deps to
> > package *just* the dependencies of each project in a special assembly
> JAR,
> > and then use sbt compile to update the code. This will use the classes
> > directly out of the target/scala-2.9.3/classes directories. You have to
> > redo assemble-deps only if your external dependencies change.
> > >
> > > Matei
> > >
> > > On Dec 11, 2013, at 1:04 AM, Prashant Sharma <scrapcodes@gmail.com>
> > wrote:
> > >
> > >> I hope this PR https://github.com/apache/incubator-spark/pull/252 can
> > help.
> > >> Again this is not a blocker for the release from my side either.
> > >>
> > >>
> > >> On Wed, Dec 11, 2013 at 2:14 PM, Mark Hamstra <
> mark@clearstorydata.com
> > >wrote:
> > >>
> > >>> Interesting, and confirmed: On my machine where `./sbt/sbt assembly`
> > takes
> > >>> a long, long, looooong time to complete (a MBP, in my case), building
> > three
> > >>> separate assemblies (`./sbt/sbt assembly/assembly`, `./sbt/sbt
> > >>> examples/assembly`, `./sbt/sbt tools/assembly`) takes much, much less
> > time.
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Dec 11, 2013 at 12:02 AM, Prashant Sharma <
> > scrapcodes@gmail.com
> > >>>> wrote:
> > >>>
> > >>>> forgot to mention, after running sbt/sbt assembly/assembly running
> > >>> sbt/sbt
> > >>>> examples/assembly takes just 37s. Not to mention my hardware is
not
> > >>> really
> > >>>> great.
> > >>>>
> > >>>>
> > >>>> On Wed, Dec 11, 2013 at 1:28 PM, Prashant Sharma <
> > scrapcodes@gmail.com
> > >>>>> wrote:
> > >>>>
> > >>>>> Hi Patrick and Matei,
> > >>>>>
> > >>>>> Was trying out this and followed the quick start guide which
says
> do
> > >>>>> sbt/sbt assembly, like few others I was also stuck for few
minutes
> on
> > >>>>> linux. On the other hand if I use sbt/sbt assembly/assembly
it is
> > much
> > >>>>> faster.
> > >>>>>
> > >>>>> Should we change the documentation to reflect this. It will
not be
> > >>> great
> > >>>>> for first time users to get stuck there.
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Dec 11, 2013 at 9:54 AM, Matei Zaharia <
> > >>> matei.zaharia@gmail.com
> > >>>>> wrote:
> > >>>>>
> > >>>>>> +1
> > >>>>>>
> > >>>>>> Built and tested it on Mac OS X.
> > >>>>>>
> > >>>>>> Matei
> > >>>>>>
> > >>>>>>
> > >>>>>> On Dec 10, 2013, at 4:49 PM, Patrick Wendell <pwendell@gmail.com>
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> Please vote on releasing the following candidate as
Apache Spark
> > >>>>>>> (incubating) version 0.8.1.
> > >>>>>>>
> > >>>>>>> The tag to be voted on is v0.8.1-incubating (commit
b87d31d):
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=commit;h=b87d31dd8eb4b4e47c0138e9242d0dd6922c8c4e
> > >>>>>>>
> > >>>>>>> The release files, including signatures, digests, etc
can be
> found
> > >>> at:
> > >>>>>>> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc4/
> > >>>>>>>
> > >>>>>>> Release artifacts are signed with the following key:
> > >>>>>>> https://people.apache.org/keys/committer/pwendell.asc
> > >>>>>>>
> > >>>>>>> The staging repository for this release can be found
at:
> > >>>>>>>
> > >>>>
> > https://repository.apache.org/content/repositories/orgapachespark-040/
> > >>>>>>>
> > >>>>>>> The documentation corresponding to this release can
be found at:
> > >>>>>>>
> > http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc4-docs/
> > >>>>>>>
> > >>>>>>> For information about the contents of this release
see:
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=blob;f=CHANGES.txt;h=ce0aeab524505b63c7999e0371157ac2def6fe1c;hb=branch-0.8
> > >>>>>>>
> > >>>>>>> Please vote on releasing this package as Apache Spark
> > >>>> 0.8.1-incubating!
> > >>>>>>>
> > >>>>>>> The vote is open until Saturday, December 14th at 01:00
UTC and
> > >>>>>>> passes if a majority of at least 3 +1 PPMC votes are
cast.
> > >>>>>>>
> > >>>>>>> [ ] +1 Release this package as Apache Spark 0.8.1-incubating
> > >>>>>>> [ ] -1 Do not release this package because ...
> > >>>>>>>
> > >>>>>>> To learn more about Apache Spark, please see
> > >>>>>>> http://spark.incubator.apache.org/
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> s
> > >>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> s
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> s
> > >
> >
>



-- 
--
Evan Chan
Staff Engineer
ev@ooyala.com  |

<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message