spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ye Xianjin <>
Subject Re: [VOTE] Release Apache Spark 1.1.0 (RC2)
Date Fri, 29 Aug 2014 11:02:22 GMT
We just used CDH 4.7 for our production cluster. And I believe we won't use CDH 5 in the next

Sent from my iPhone

> On 2014年8月29日, at 14:39, Matei Zaharia <> wrote:
> Personally I'd actually consider putting CDH4 back if there are still users on it. It's
always better to be inclusive, and the convenience of a one-click download is high. Do we
have a sense on what % of CDH users still use CDH4?
> Matei
> On August 28, 2014 at 11:31:13 PM, Sean Owen ( wrote:
> (Copying my reply since I don't know if it goes to the mailing list) 
> Great, thanks for explaining the reasoning. You're saying these aren't 
> going into the final release? I think that moots any issue surrounding 
> distributing them then. 
> This is all I know of from the ASF: 
> I don't read it 
> as expressly forbidding this kind of thing although you can see how it 
> bumps up against the spirit. There's not a bright line -- what about 
> Tomcat providing binaries compiled for Windows for example? does that 
> favor an OS vendor? 
> From this technical ASF perspective only the releases matter -- do 
> what you want with snapshots and RCs. The only issue there is maybe 
> releasing something different than was in the RC; is that at all 
> confusing? Just needs a note. 
> I think this theoretical issue doesn't exist if these binaries aren't 
> released, so I see no reason to not proceed. 
> The rest is a different question about whether you want to spend time 
> maintaining this profile and candidate. The vendor already manages 
> their build I think and -- and I don't know -- may even prefer not to 
> have a different special build floating around. There's also the 
> theoretical argument that this turns off other vendors from adopting 
> Spark if it's perceived to be too connected to other vendors. I'd like 
> to maximize Spark's distribution and there's some argument you do this 
> by not making vendor profiles. But as I say a different question to 
> just think about over time... 
> (oh and PS for my part I think it's a good thing that CDH4 binaries 
> were removed. I wasn't arguing for resurrecting them) 
>> On Fri, Aug 29, 2014 at 7:26 AM, Patrick Wendell <> wrote:

>> Hey Sean, 
>> The reason there are no longer CDH-specific builds is that all newer 
>> versions of CDH and HDP work with builds for the upstream Hadoop 
>> projects. I dropped CDH4 in favor of a newer Hadoop version (2.4) and 
>> the Hadoop-without-Hive (also 2.4) build. 
>> For MapR - we can't officially post those artifacts on ASF web space 
>> when we make the final release, we can only link to them as being 
>> hosted by MapR specifically since they use non-compatible licenses. 
>> However, I felt that providing these during a testing period was 
>> alright, with the goal of increasing test coverage. I couldn't find 
>> any policy against posting these on personal web space during RC 
>> voting. However, we can remove them if there is one. 
>> Dropping CDH4 was more because it is now pretty old, but we can add it 
>> back if people want. The binary packaging is a slightly separate 
>> question from release votes, so I can always add more binary packages 
>> whenever. And on this, my main concern is covering the most popular 
>> Hadoop versions to lower the bar for users to build and test Spark. 
>> - Patrick 
>>> On Thu, Aug 28, 2014 at 11:04 PM, Sean Owen <> wrote:

>>> +1 I tested the source and Hadoop 2.4 release. Checksums and 
>>> signatures are OK. Compiles fine with Java 8 on OS X. Tests... don't 
>>> fail any more than usual. 
>>> FWIW I've also been using the 1.1.0-SNAPSHOT for some time in another 
>>> project and have encountered no problems. 
>>> I notice that the 1.1.0 release removes the CDH4-specific build, but 
>>> adds two MapR-specific builds. Compare with 
>>> I 
>>> commented on the commit: 

>>> I'm in favor of removing all vendor-specific builds. This change 
>>> *looks* a bit funny as there was no JIRA (?) and appears to swap one 
>>> vendor for another. Of course there's nothing untoward going on, but 
>>> what was the reasoning? It's best avoided, and MapR already 
>>> distributes Spark just fine, no? 
>>> This is a gray area with ASF projects. I mention it as well because it 
>>> came up with Apache Flink recently 
>>> (
>>> Another vendor rightly noted this could look like favoritism. They 
>>> changed to remove vendor releases. 
>>>> On Fri, Aug 29, 2014 at 3:14 AM, Patrick Wendell <>
>>>> Please vote on releasing the following candidate as Apache Spark version
>>>> The tag to be voted on is v1.1.0-rc2 (commit 711aebb3): 

>>>> The release files, including signatures, digests, etc. can be found at: 
>>>> Release artifacts are signed with the following key: 
>>>> The staging repository for this release can be found at: 
>>>> The documentation corresponding to this release can be found at: 
>>>> Please vote on releasing this package as Apache Spark 1.1.0! 
>>>> The vote is open until Monday, September 01, at 03:11 UTC and passes if 
>>>> a majority of at least 3 +1 PMC votes are cast. 
>>>> [ ] +1 Release this package as Apache Spark 1.1.0 
>>>> [ ] -1 Do not release this package because ... 
>>>> To learn more about Apache Spark, please see 
>>>> == Regressions fixed since RC1 == 
>>>> LZ4 compression issue: 
>>>> == What justifies a -1 vote for this release? == 
>>>> This vote is happening very late into the QA period compared with 
>>>> previous votes, so -1 votes should only occur for significant 
>>>> regressions from 1.0.2. Bugs already present in 1.0.X will not block 
>>>> this release. 
>>>> == What default changes should I be aware of? == 
>>>> 1. The default value of "" is now "snappy" 
>>>> --> Old behavior can be restored by switching to "lzf" 
>>>> 2. PySpark now performs external spilling during aggregations. 
>>>> --> Old behavior can be restored by setting "spark.shuffle.spill" to "false".

>>>> --------------------------------------------------------------------- 
>>>> To unsubscribe, e-mail:
>>>> For additional commands, e-mail:
> --------------------------------------------------------------------- 
> To unsubscribe, e-mail: 
> For additional commands, e-mail: 

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message