Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 08C10185EE for ; Tue, 22 Dec 2015 01:49:14 +0000 (UTC) Received: (qmail 38134 invoked by uid 500); 22 Dec 2015 01:49:12 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 38018 invoked by uid 500); 22 Dec 2015 01:49:11 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 38007 invoked by uid 99); 22 Dec 2015 01:49:11 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Dec 2015 01:49:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 11E68C0A84 for ; Tue, 22 Dec 2015 01:49:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.001 X-Spam-Level: **** X-Spam-Status: No, score=4.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, HTML_OBFUSCATE_05_10=0.001, KAM_LIVE=1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=databricks-com.20150623.gappssmtp.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id wBNhQrOvnOe5 for ; Tue, 22 Dec 2015 01:48:58 +0000 (UTC) Received: from mail-lf0-f47.google.com (mail-lf0-f47.google.com [209.85.215.47]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 07C7F203B9 for ; Tue, 22 Dec 2015 01:48:58 +0000 (UTC) Received: by mail-lf0-f47.google.com with SMTP id p203so121996624lfa.0 for ; Mon, 21 Dec 2015 17:48:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=databricks-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=ZrFXEke333F2QlEYuKRnxJehRtn+8agGcHV+SQ84SKM=; b=f9p6+ZIlkYfB5aALH1f5FNbQzQEX8cVb009MbrJw7LxMUomV0i0Ilk4hioyeBioE9r Mion+9A6oEpOTYaPpsPrPepy1sTV497dmrf/2KcAz9tx52Rj0iBaCbLyU6AQbSzBS5WN KnqXzm2nHaYDBQcwzAnTcRWgSU4QA1zyzSG9/o7/gduCYse9ORb6a7TrD0DoXZEX42Zu 2OYeNdiQlvg0gHsD9m5CYIFzjNLx4mxTTJ9IfZoxQNE2njn93LUTq3yznPBtOnILzjdv kh9wSjKuuv42vsV986xu7htCVBWLTQoTjmFXGaA6qgYzAjRhRjmIjME5K1EGobCfsXIQ d9pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=ZrFXEke333F2QlEYuKRnxJehRtn+8agGcHV+SQ84SKM=; b=DuK/2ARGyXXVWknMrgi5tJ9lR8pBOisYG+WE9qm3avTGFcY70xUfIYGN5BIYHvhWWe lRbtC21+uYcZaijvUGMDqr/R8WWXkStoW4tv3Nhp6ZtjytkeLIdWeZ3qSPhGnHoi6MFX 8tfjTpiuR8UJsXb6b+CKryQBMx0rNpqKjkC49wTBbjbadJCJX41aHdnI3wVBDamzWXAH mXUUDvEkiB06/BmZwU0T5a/ldsIbbZpTBP+K6i0P52iWpP9/Cny8dDHRSUQMwYcfW39M DnVIh+rZmC2twSN6cwxYuhuLTpFFnyt7OIbSCmS8pEniWtHWqjNa3mi1RUhFzbOamwc/ 7u3g== X-Gm-Message-State: ALoCoQlzBPDen4orxLagr6K/kzeaE4wECNbglfDW/rlUEX2abOHXZ1+MNNoDZ7UEscMjX7BSPGVwPAccBir4ITpaYV3FKpX6Jw== X-Received: by 10.25.88.209 with SMTP id m200mr3369591lfb.99.1450748937175; Mon, 21 Dec 2015 17:48:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.141.204 with HTTP; Mon, 21 Dec 2015 17:48:37 -0800 (PST) In-Reply-To: References: From: Michael Armbrust Date: Mon, 21 Dec 2015 17:48:37 -0800 Message-ID: Subject: Re: [VOTE] Release Apache Spark 1.6.0 (RC3) To: "dev@spark.apache.org" Content-Type: multipart/alternative; boundary=001a1140ef920f15fe052772cf50 --001a1140ef920f15fe052772cf50 Content-Type: text/plain; charset=UTF-8 It's come to my attention that there have been several bug fixes merged since RC3: - SPARK-12404 - Fix serialization error for Datasets with Timestamps/Arrays/Decimal - SPARK-12218 - Fix incorrect pushdown of filters to parquet - SPARK-12395 - Fix join columns of outer join for DataFrame using - SPARK-12413 - Fix mesos HA Normally, these would probably not be sufficient to hold the release, however with the holidays going on in the US this week, we don't have the resources to finalize 1.6 until next Monday. Given this delay anyway, I propose that we cut one final RC with the above fixes and plan for the actual release first thing next week. I'll post RC4 shortly and cancel this vote if there are no objections. Since this vote nearly passed with no major issues, I don't anticipate any problems with RC4. Michael On Sat, Dec 19, 2015 at 11:44 PM, Jeff Zhang wrote: > +1 (non-binding) > > All the test passed, and run it on HDP 2.3.2 sandbox successfully. > > On Sun, Dec 20, 2015 at 10:43 AM, Luciano Resende > wrote: > >> +1 (non-binding) >> >> Tested Standalone mode, SparkR and couple Stream Apps, all seem ok. >> >> On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust > > wrote: >> >>> Please vote on releasing the following candidate as Apache Spark version >>> 1.6.0! >>> >>> The vote is open until Saturday, December 19, 2015 at 18:00 UTC and >>> passes if a majority of at least 3 +1 PMC votes are cast. >>> >>> [ ] +1 Release this package as Apache Spark 1.6.0 >>> [ ] -1 Do not release this package because ... >>> >>> To learn more about Apache Spark, please see http://spark.apache.org/ >>> >>> The tag to be voted on is *v1.6.0-rc3 >>> (168c89e07c51fa24b0bb88582c739cec0acb44d7) >>> * >>> >>> The release files, including signatures, digests, etc. can be found at: >>> http://people.apache.org/~pwendell/spark-releases/spark-1.6.0-rc3-bin/ >>> >>> Release artifacts are signed with the following key: >>> https://people.apache.org/keys/committer/pwendell.asc >>> >>> The staging repository for this release can be found at: >>> https://repository.apache.org/content/repositories/orgapachespark-1174/ >>> >>> The test repository (versioned as v1.6.0-rc3) for this release can be >>> found at: >>> https://repository.apache.org/content/repositories/orgapachespark-1173/ >>> >>> The documentation corresponding to this release can be found at: >>> http://people.apache.org/~pwendell/spark-releases/spark-1.6.0-rc3-docs/ >>> >>> ======================================= >>> == How can I help test this release? == >>> ======================================= >>> If you are a Spark user, you can help us test this release by taking an >>> existing Spark workload and running on this release candidate, then >>> reporting any regressions. >>> >>> ================================================ >>> == What justifies a -1 vote for this release? == >>> ================================================ >>> This vote is happening towards the end of the 1.6 QA period, so -1 votes >>> should only occur for significant regressions from 1.5. Bugs already >>> present in 1.5, minor regressions, or bugs related to new features will not >>> block this release. >>> >>> =============================================================== >>> == What should happen to JIRA tickets still targeting 1.6.0? == >>> =============================================================== >>> 1. It is OK for documentation patches to target 1.6.0 and still go into >>> branch-1.6, since documentations will be published separately from the >>> release. >>> 2. New features for non-alpha-modules should target 1.7+. >>> 3. Non-blocker bug fixes should target 1.6.1 or 1.7.0, or drop the >>> target version. >>> >>> >>> ================================================== >>> == Major changes to help you focus your testing == >>> ================================================== >>> >>> Notable changes since 1.6 RC2 >>> - SPARK_VERSION has been set correctly >>> - SPARK-12199 ML Docs are publishing correctly >>> - SPARK-12345 Mesos cluster mode has been fixed >>> >>> Notable changes since 1.6 RC1 >>> Spark Streaming >>> >>> - SPARK-2629 >>> trackStateByKey has been renamed to mapWithState >>> >>> Spark SQL >>> >>> - SPARK-12165 >>> SPARK-12189 Fix >>> bugs in eviction of storage memory by execution. >>> - SPARK-12258 correct >>> passing null into ScalaUDF >>> >>> Notable Features Since 1.5Spark SQL >>> >>> - SPARK-11787 Parquet >>> Performance - Improve Parquet scan performance when using flat >>> schemas. >>> - SPARK-10810 >>> Session Management - Isolated devault database (i.e USE mydb) even >>> on shared clusters. >>> - SPARK-9999 Dataset >>> API - A type-safe API (similar to RDDs) that performs many >>> operations on serialized binary data and code generation (i.e. Project >>> Tungsten). >>> - SPARK-10000 Unified >>> Memory Management - Shared memory for execution and caching instead >>> of exclusive division of the regions. >>> - SPARK-11197 SQL >>> Queries on Files - Concise syntax for running SQL queries over files >>> of any supported format without registering a table. >>> - SPARK-11745 Reading >>> non-standard JSON files - Added options to read non-standard JSON >>> files (e.g. single-quotes, unquoted attributes) >>> - SPARK-10412 Per-operator >>> Metrics for SQL Execution - Display statistics on a peroperator >>> basis for memory usage and spilled data size. >>> - SPARK-11329 Star >>> (*) expansion for StructTypes - Makes it easier to nest and unest >>> arbitrary numbers of columns >>> - SPARK-10917 , >>> SPARK-11149 In-memory >>> Columnar Cache Performance - Significant (up to 14x) speed up when >>> caching data that contains complex types in DataFrames or SQL. >>> - SPARK-11111 Fast >>> null-safe joins - Joins using null-safe equality (<=>) will now >>> execute using SortMergeJoin instead of computing a cartisian product. >>> - SPARK-11389 SQL >>> Execution Using Off-Heap Memory - Support for configuring query >>> execution to occur using off-heap memory to avoid GC overhead >>> - SPARK-10978 Datasource >>> API Avoid Double Filter - When implemeting a datasource with filter >>> pushdown, developers can now tell Spark SQL to avoid double evaluating a >>> pushed-down filter. >>> - SPARK-4849 Advanced >>> Layout of Cached Data - storing partitioning and ordering schemes in >>> In-memory table scan, and adding distributeBy and localSort to DF API >>> - SPARK-9858 Adaptive >>> query execution - Intial support for automatically selecting the >>> number of reducers for joins and aggregations. >>> - SPARK-9241 Improved >>> query planner for queries having distinct aggregations - Query plans >>> of distinct aggregations are more robust when distinct columns have high >>> cardinality. >>> >>> Spark Streaming >>> >>> - API Updates >>> - SPARK-2629 New >>> improved state management - mapWithState - a DStream >>> transformation for stateful stream processing, supercedes >>> updateStateByKey in functionality and performance. >>> - SPARK-11198 Kinesis >>> record deaggregation - Kinesis streams have been upgraded to use >>> KCL 1.4.0 and supports transparent deaggregation of KPL-aggregated records. >>> - SPARK-10891 Kinesis >>> message handler function - Allows arbitraray function to be >>> applied to a Kinesis record in the Kinesis receiver before to customize >>> what data is to be stored in memory. >>> - SPARK-6328 Python >>> Streamng Listener API - Get streaming statistics (scheduling >>> delays, batch processing times, etc.) in streaming. >>> >>> >>> - UI Improvements >>> - Made failures visible in the streaming tab, in the timelines, >>> batch list, and batch details page. >>> - Made output operations visible in the streaming tab as progress >>> bars. >>> >>> MLlibNew algorithms/models >>> >>> - SPARK-8518 Survival >>> analysis - Log-linear model for survival analysis >>> - SPARK-9834 Normal >>> equation for least squares - Normal equation solver, providing >>> R-like model summary statistics >>> - SPARK-3147 Online >>> hypothesis testing - A/B testing in the Spark Streaming framework >>> - SPARK-9930 New >>> feature transformers - ChiSqSelector, QuantileDiscretizer, SQL >>> transformer >>> - SPARK-6517 Bisecting >>> K-Means clustering - Fast top-down clustering variant of K-Means >>> >>> API improvements >>> >>> - ML Pipelines >>> - SPARK-6725 Pipeline >>> persistence - Save/load for ML Pipelines, with partial coverage >>> of spark.mlalgorithms >>> - SPARK-5565 LDA >>> in ML Pipelines - API for Latent Dirichlet Allocation in ML >>> Pipelines >>> - R API >>> - SPARK-9836 R-like >>> statistics for GLMs - (Partial) R-like stats for ordinary least >>> squares via summary(model) >>> - SPARK-9681 Feature >>> interactions in R formula - Interaction operator ":" in R formula >>> - Python API - Many improvements to Python API to approach feature >>> parity >>> >>> Misc improvements >>> >>> - SPARK-7685 , >>> SPARK-9642 Instance >>> weights for GLMs - Logistic and Linear Regression can take instance >>> weights >>> - SPARK-10384 , >>> SPARK-10385 Univariate >>> and bivariate statistics in DataFrames - Variance, stddev, >>> correlations, etc. >>> - SPARK-10117 LIBSVM >>> data source - LIBSVM as a SQL data sourceDocumentation improvements >>> - SPARK-7751 @since >>> versions - Documentation includes initial version when classes and >>> methods were added >>> - SPARK-11337 Testable >>> example code - Automated testing for code in user guide examples >>> >>> Deprecations >>> >>> - In spark.mllib.clustering.KMeans, the "runs" parameter has been >>> deprecated. >>> - In spark.ml.classification.LogisticRegressionModel and >>> spark.ml.regression.LinearRegressionModel, the "weights" field has been >>> deprecated, in favor of the new name "coefficients." This helps >>> disambiguate from instance (row) weights given to algorithms. >>> >>> Changes of behavior >>> >>> - spark.mllib.tree.GradientBoostedTrees validationTol has changed >>> semantics in 1.6. Previously, it was a threshold for absolute change in >>> error. Now, it resembles the behavior of GradientDescent convergenceTol: >>> For large errors, it uses relative error (relative to the previous error); >>> for small errors (< 0.01), it uses absolute error. >>> - spark.ml.feature.RegexTokenizer: Previously, it did not convert >>> strings to lowercase before tokenizing. Now, it converts to lowercase by >>> default, with an option not to. This matches the behavior of the simpler >>> Tokenizer transformer. >>> - Spark SQL's partition discovery has been changed to only discover >>> partition directories that are children of the given path. (i.e. if >>> path="/my/data/x=1" then x=1 will no longer be considered a >>> partition but only children of x=1.) This behavior can be overridden >>> by manually specifying the basePath that partitioning discovery >>> should start with (SPARK-11678 >>> ). >>> - When casting a value of an integral type to timestamp (e.g. >>> casting a long value to timestamp), the value is treated as being in >>> seconds instead of milliseconds (SPARK-11724 >>> ). >>> - With the improved query planner for queries having distinct >>> aggregations (SPARK-9241 >>> ), the plan of a >>> query having a single distinct aggregation has been changed to a more >>> robust version. To switch back to the plan generated by Spark 1.5's >>> planner, please set spark.sql.specializeSingleDistinctAggPlanning to >>> true (SPARK-12077 >>> ). >>> >>> >> >> >> -- >> Luciano Resende >> http://people.apache.org/~lresende >> http://twitter.com/lresende1975 >> http://lresende.blogspot.com/ >> > > > > -- > Best Regards > > Jeff Zhang > --001a1140ef920f15fe052772cf50 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
It's come to my attention that there have been se= veral bug fixes merged since RC3:

=C2=A0 - SPARK-1= 2404 - Fix serialization error for Datasets with Timestamps/Arrays/Decimal<= br>=C2=A0 - SPARK-12218 - Fix incorrect pushdown of filters to parquet
= =C2=A0 - SPARK-12395 - Fix join columns of outer join for DataFrame using=C2=A0 - SPARK-12413 - Fix mesos HA

Normally, th= ese would probably not be sufficient to hold the release, however with the = holidays going on in the US this week, we don't have the resources to f= inalize 1.6 until next Monday.=C2=A0 Given this delay anyway, I propose tha= t we cut one final RC with the above fixes and plan for the actual release = first thing next week.

I'll post RC4 shortly a= nd cancel this vote if there are no objections.=C2=A0 Since this vote nearl= y passed with no major issues, I don't anticipate any problems with RC4= .

Michael

<= div class=3D"gmail_quote">On Sat, Dec 19, 2015 at 11:44 PM, Jeff Zhang <zjf= fdu@gmail.com> wrote:
+1= (non-binding)

All the test passed, and run it on HDP 2.= 3.2 sandbox successfully.
<= br>
On Sun, Dec 20, 2015 at 10:43 AM, Luciano Res= ende <luckbr1975@gmail.com> wrote:
=
+1 (non-binding)

Tested Stand= alone mode, SparkR and couple Stream Apps, all seem ok.

On Wed, Dec 16, 2015 = at 1:32 PM, Michael Armbrust <michael@databricks.com> w= rote:
Please vote on releasi= ng the following candidate as Apache Spark version 1.6.0!

The vote= is open until Saturday, December 19, 2015=C2=A0at 18:00 UTC and passes if a majority=C2=A0of at least 3 +1 PMC votes= are cast.

[ ] +1 Release this package as Apache = Spark 1.6.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see=C2=A0http://spark.apache.org/
=

The tag to be voted on is=C2=A0v1.6.0-rc3 (168c89e07c51fa24b0bb88582c739cec0acb44d7)<= /font>

The release files, including signatures, digests, etc. can b= e found at:

Release artifacts= are signed with the following key:

The staging repository for this release can be found at:
<= div style=3D"font-size:12.8px">https://repository.apache.org/content/repositories/orgap= achespark-1174/

The test repository (versioned as = v1.6.0-rc3) for this release can be found at:

The documentation corresponding to this release can be f= ound at:

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=3D=3D How c= an I help test this release? =3D=3D
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D
If you are a Spark user, you can help us test this release b= y taking an existing Spark workload and running on this release candidate, = then reporting any regressions.

=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=3D=3D What justifies a -1 = vote for this release? =3D=3D
<= font face=3D"monospace, monospace">=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
= This vote is happening towards the end = of the 1.6 QA period, so -1 votes should only occur for significant regress= ions from 1.5. Bugs already present in 1.5, minor regressions, or bugs rela= ted to new features will not block this release.

=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
= =3D=3D What should happen to JIRA ticke= ts still targeting 1.6.0? =3D=3D
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D
1. It is OK for documentation patches to target 1.6.0 and still go= into branch-1.6, since documentations will be published separately from th= e release.
2. New features for non-alpha-modules should target 1.7+.
3. Non-blocker bug fixes should target 1.6.1 or 1.7.0, or drop the targ= et version.


=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=3D=3D Major changes to help y= ou focus your testing =3D=3D
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Notable changes since 1.6= RC2


- SPARK_VERSION has= been set correctly
- SPARK-12199=C2=A0ML Docs are publishing correctly<= br>- SPARK-12345 Mesos cluster mode has been fixed

<= span style=3D"font-weight:normal;font-family:monospace,monospace;font-size:= small;line-height:1.225">Notable changes since 1.6 RC1

Spark Streaming

Spark S= QL

  • SPARK-12165=C2=A0<= a href=3D"https://issues.apache.org/jira/browse/SPARK-12189" style=3D"color= :rgb(71,135,242);text-decoration:none;line-height:1.6" target=3D"_blank">SP= ARK-12189=C2=A0Fix bugs in eviction of storage memory by execution.
  • SPARK-12258=C2=A0correct passing null into ScalaUDF

Notable Features Since 1.5

Spark SQL

  • SPARK-11787=C2=A0Parquet Performance=C2=A0- Improve Parquet= scan performance when using flat schemas.
  • SPARK-10810=C2=A0Session=C2=A0Management=C2=A0- Isolated devault database (i.e=C2=A0<= code style=3D"color:rgb(51,51,51);line-height:1.6;padding:0px;border-radius= :0px;white-space:nowrap;border:none;margin:0px;background-image:none;backgr= ound-color:initial;background-repeat:initial">USE mydb) even on shared clusters.=
  • SPARK-9999=C2=A0=C2=A0Dataset API=C2=A0- A type-safe API (simi= lar to RDDs) that performs many operations on serialized binary data and co= de generation (i.e. Project Tungsten).
  • SPARK-10000=C2=A0Unified Memory Manag= ement=C2=A0- Shared memory for execution and caching instead of excl= usive division of the regions.
  • SPARK-= 11197=C2=A0SQL Queries on Files= =C2=A0- Concise syntax for running SQL queries over files of any supported = format without registering a table.
  • = S= PARK-11745=C2=A0Reading non-standard JS= ON files=C2=A0- Added options to read non-standard JSON files (e.g. = single-quotes, unquoted attributes)
  • = S= PARK-10412=C2=A0Per-operator Metrics fo= r SQL Execution=C2=A0- Display statistics on a peroperator basis for= memory usage and spilled data size.
  • = SPARK-11329=C2=A0Star (*) expansion for= StructTypes=C2=A0- Makes it easier to nest and unest arbitrary numb= ers of columns
  • SPARK-10917,=C2=A0= S= PARK-11149=C2=A0In-memory Columnar Cach= e Performance=C2=A0- Significant (up to 14x) speed up when caching d= ata that contains complex types in DataFrames or SQL.
  • SPARK-11111=C2=A0Fast= null-safe joins=C2=A0- Joins using null-safe equality (<=3D>) will now execute usin= g SortMergeJoin instead of computing a cartisian product.
  • SPARK-11389=C2=A0S= QL Execution Using Off-Heap Memory=C2=A0- Support for configuring qu= ery execution to occur using off-heap memory to avoid GC overhead
  • SPARK-10978=C2=A0Datasource API Avoid Double Filter=C2=A0- When implemeting a = datasource with filter pushdown, developers can now tell Spark SQL to avoid= double evaluating a pushed-down filter.
  • SPARK-4849=C2=A0=C2=A0Advanced Layou= t of Cached Data=C2=A0- storing partitioning and ordering schemes in= In-memory table scan, and adding distributeBy and localSort to DF API
  • <= font face=3D"monospace, monospace">SPARK-9858=C2=A0=C2=A0Adaptive query execution=C2=A0- Intial support for a= utomatically selecting the number of reducers for joins and aggregations.
  • SPARK-9241=C2=A0=C2=A0Improved query planner for queries having distinct agg= regations=C2=A0- Query plans of distinct aggregations are more robus= t when distinct columns have high cardinality.

Spark Streaming

  • API Updates
    • SPARK-2629=C2=A0=C2=A0<= span style=3D"line-height:1.6">New improved state management=C2=A0-= =C2=A0mapWithState=C2= =A0- a DStream transformation for stateful stream processing, supercedes=C2= =A0updateStateByKey=C2= =A0in functionality and performance.
    • SPARK-11198=C2= =A0Kinesis record deaggregation=C2= =A0- Kinesis streams have been upgraded to use KCL 1.4.0 and supports trans= parent deaggregation of KPL-aggregated records.
    • SPARK-10= 891=C2=A0Kinesis message handler functi= on=C2=A0- Allows arbitraray function to be applied to a Kinesis reco= rd in the Kinesis receiver before to customize what data is to be stored in= memory.
    • SPARK-6328=C2=A0=C2=A0Python Streamng Listener API=C2=A0- Get streaming statistic= s (scheduling delays, batch processing times, etc.) in streaming.
  • UI Improvements
    • Made failures visible in the stre= aming tab, in the timelines, batch list, and batch details page.
    • Made output operations visible in the streaming tab as progress ba= rs.

MLlib

New algorith= ms/models

  • SPARK-8518=C2=A0= =C2=A0Survival analysis=C2=A0- L= og-linear model for survival analysis
  • SPARK-9834=C2=A0= =C2=A0Normal equation for least squares=C2=A0- Normal equation solver, providing R-like model summary statistic= s
  • SPARK-3147=C2=A0=C2=A0Online hypothesis testing=C2=A0- A/B testing in the Spark Streamin= g framework
  • SPARK-9930=C2=A0=C2=A0New feature transformers=C2=A0- ChiSqSelector, QuantileD= iscretizer, SQL transformer
  • SPARK-6517=C2=A0=C2=A0Bisecting K-Means clustering=C2=A0- Fast= top-down clustering variant of K-Means

API improvements<= /font>

  • ML Pipelines
    • SPARK-6725=C2=A0=C2=A0Pipeline persistence=C2=A0- Save/load for ML Pipelines= , with partial coverage of=C2=A0spark.mlalgorithms
    • SPARK-5565=C2=A0=C2=A0LDA in ML Pipelines=C2=A0- API for Latent= Dirichlet Allocation in ML Pipelines
  • R API
    • SPARK-9836=C2=A0=C2=A0R-like statistics for GLMs=C2=A0- (Partial)= R-like stats for ordinary least squares via summary(model)
    • SPARK-9681=C2=A0=C2=A0Feature interac= tions in R formula=C2=A0- Interaction operator ":" in R fo= rmula
  • <= font face=3D"monospace, monospace">Python A= PI=C2=A0- Many improvements to Python API to approach feature parity=

Misc improvements

  • SPARK-7685=C2=A0,=C2=A0SPARK-9642=C2=A0=C2=A0Instance weights for GLMs=C2=A0- Logistic and L= inear Regression can take instance weights
  • SPARK-10384,=C2=A0SPARK-10385=C2=A0Univariate and = bivariate statistics in DataFrames=C2=A0- Variance, stddev, correlat= ions, etc.
  • SPARK-10117=C2=A0LIBSVM data source=C2=A0- LIBSVM as a SQL data source<= h4 style=3D"margin:16px 0px 0px;line-height:1.6;color:inherit">Documentation improve= ments
  • SPARK-7751=C2=A0=C2=A0@since versions=C2=A0- Documentation includes initial ver= sion when classes and methods were added
  • SPARK-11337= =C2=A0Testable example code=C2=A0- A= utomated testing for code in user guide examples

Deprecations

  • In s= park.mllib.clustering.KMeans, the "runs" parameter has been depre= cated.
  • In spark.ml.classification.LogisticRegressionMode= l and spark.ml.regression.LinearRegressionModel, the "weights" fi= eld has been deprecated, in favor of the new name "coefficients."= This helps disambiguate from instance (row) weights given to algorithms.

Changes of behavior

  • spark.mllib.tree.GradientBoostedTrees validationT= ol has changed semantics in 1.6. Previously, it was a threshold for absolut= e change in error. Now, it resembles the behavior of GradientDescent conver= genceTol: For large errors, it uses relative error (relative to the previou= s error); for small errors (< 0.01), it uses absolute error.
    • =
  • spark.ml.feature.RegexTokenizer: Previously, it did not convert str= ings to lowercase before tokenizing. Now, it converts to lowercase by defau= lt, with an option not to. This matches the behavior of the simpler Tokeniz= er transformer.
  • <= font face=3D"monospace, monospace">Spark SQL's partition discovery has = been changed to only discover partition directories that are children of th= e given path. (i.e. if=C2=A0pa= th=3D"/my/data/x=3D1"=C2=A0then=C2=A0x=3D1=C2=A0will no longer be considered a pa= rtition but only children of=C2=A0x=3D1.) This behavior can be overridden by manually specifying t= he=C2=A0basePath=C2=A0t= hat partitioning discovery should start with (SPARK-11678).
    • <= li style=3D"margin-left:15px;line-height:1.6">When casting a value of an integral type to timestamp (e.g. casting = a long value to timestamp), the value is treated as being in seconds instea= d of milliseconds (SPARK-11724).
  • With the improved qu= ery planner for queries having distinct aggregations (SPARK-9241), the pl= an of a query having a single distinct aggregation has been changed to a mo= re robust version. To switch back to the plan generated by Spark 1.5's = planner, please set=C2=A0spark.sql.specializeSin= gleDistinctAggPlanning=C2=A0to=C2=A0true<= /code>=C2=A0(SPARK-12077).



--



<= font color=3D"#888888">--
Best Regards

Jeff Zhang

--001a1140ef920f15fe052772cf50--