spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Chan ...@ooyala.com>
Subject Re: Spark 0.9.1 release
Date Mon, 24 Mar 2014 07:11:47 GMT
@Tathagata,  the PR is here:
https://github.com/apache/spark/pull/215

On Mon, Mar 24, 2014 at 12:02 AM, Tathagata Das
<tathagata.das1565@gmail.com> wrote:
> @Shivaram, That is a useful patch but I am bit afraid merge it in.
> Randomizing the executor has performance implications, especially for Spark
> Streaming. The non-randomized ordering of allocating machines to tasks was
> subtly helping to speed up certain window-based shuffle operations.  For
> example, corresponding shuffle partitions in multiple shuffles using the
> same partitioner were likely to be co-located, that is, shuffle partition 0
> were likely to be on the same machine for multiple shuffles. While this is
> the not a reliable mechanism to rely on, randomization may lead to
> performance degradation. So I am afraid to merge this one without
> understanding the consequences.
>
> @Evan, I have already cut a release! You can submit the PR and we can merge
> it branch-0.9. If we have to cut another release, then we can include it.
>
>
>
> On Sun, Mar 23, 2014 at 11:42 PM, Evan Chan <ev@ooyala.com> wrote:
>
>> I also have a really minor fix for SPARK-1057  (upgrading fastutil),
>> could that also make it in?
>>
>> -Evan
>>
>>
>> On Sun, Mar 23, 2014 at 11:01 PM, Shivaram Venkataraman
>> <shivaram@eecs.berkeley.edu> wrote:
>> > Sorry this request is coming in a bit late, but would it be possible to
>> > backport SPARK-979[1] to branch-0.9 ? This is the patch for randomizing
>> > executor offers and I would like to use this in a release sooner rather
>> > than later.
>> >
>> > Thanks
>> > Shivaram
>> >
>> > [1]
>> >
>> https://github.com/apache/spark/commit/556c56689bbc32c6cec0d07b57bd3ec73ceb243e#diff-8ef3258646b0e6a4793d6ad99848eacd
>> >
>> >
>> > On Thu, Mar 20, 2014 at 10:18 PM, Bhaskar Dutta <bhaskie@gmail.com>
>> wrote:
>> >
>> >> Thank You! We plan to test out 0.9.1 on YARN once it is out.
>> >>
>> >> Regards,
>> >> Bhaskar
>> >>
>> >> On Fri, Mar 21, 2014 at 12:42 AM, Tom Graves <tgraves_cs@yahoo.com>
>> wrote:
>> >>
>> >> > I'll pull [SPARK-1053] Should not require SPARK_YARN_APP_JAR when
>> running
>> >> > on YARN - JIRA and  [SPARK-1051] On Yarn, executors don't doAs as
>> >> > submitting user - JIRA in.  The pyspark one I would consider more of
>> an
>> >> > enhancement so might not be appropriate for a point release.
>> >> >
>> >> >
>> >> >  [SPARK-1053] Should not require SPARK_YARN_APP_JAR when running on
>> YA...
>> >> > org.apache.spark.SparkException: env SPARK_YARN_APP_JAR is not set
at
>> >> >
>> >>
>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:49)
>> >> > at org.apache.spark.schedule...
>> >> > View on spark-project.atlassian.net Preview by Yahoo
>> >> >
>> >> >
>> >> >  [SPARK-1051] On Yarn, executors don't doAs as submitting user - JIRA
>> >> > This means that they can't write/read from files that the yarn user
>> >> > doesn't have permissions to but the submitting user does.
>> >> > View on spark-project.atlassian.net Preview by Yahoo
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Thursday, March 20, 2014 1:35 PM, Bhaskar Dutta <bhaskie@gmail.com
>> >
>> >> > wrote:
>> >> >
>> >> > It will be great if
>> >> > "SPARK-1101<https://spark-project.atlassian.net/browse/SPARK-1101>:
>> >> > Umbrella
>> >> > for hardening Spark on YARN" can get into 0.9.1.
>> >> >
>> >> > Thanks,
>> >> > Bhaskar
>> >> >
>> >> >
>> >> > On Thu, Mar 20, 2014 at 5:37 AM, Tathagata Das
>> >> > <tathagata.das1565@gmail.com>wrote:
>> >> >
>> >> > >  Hello everyone,
>> >> > >
>> >> > > Since the release of Spark 0.9, we have received a number of
>> important
>> >> > bug
>> >> > > fixes and we would like to make a bug-fix release of Spark 0.9.1.
We
>> >> are
>> >> > > going to cut a release candidate soon and we would love it if
people
>> >> test
>> >> > > it out. We have backported several bug fixes into the 0.9 and
>> updated
>> >> > JIRA
>> >> > > accordingly<
>> >> > >
>> >> >
>> >>
>> https://spark-project.atlassian.net/browse/SPARK-1275?jql=project%20in%20(SPARK%2C%20BLINKDB%2C%20MLI%2C%20MLLIB%2C%20SHARK%2C%20STREAMING%2C%20GRAPH%2C%20TACHYON)%20AND%20fixVersion%20%3D%200.9.1%20AND%20status%20in%20(Resolved%2C%20Closed)
>> >> > > >.
>> >> > > Please let me know if there are fixes that were not backported
but
>> you
>> >> > > would like to see them in 0.9.1.
>> >> > >
>> >> > > Thanks!
>> >> > >
>> >> > > TD
>> >> > >
>> >> >
>> >>
>>
>>
>>
>> --
>> --
>> Evan Chan
>> Staff Engineer
>> ev@ooyala.com  |
>>



-- 
--
Evan Chan
Staff Engineer
ev@ooyala.com  |

Mime
View raw message