Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8C10A186E0 for ; Tue, 24 Nov 2015 06:52:30 +0000 (UTC) Received: (qmail 47161 invoked by uid 500); 24 Nov 2015 06:52:26 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 46990 invoked by uid 500); 24 Nov 2015 06:52:26 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 45999 invoked by uid 99); 24 Nov 2015 06:52:26 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2015 06:52:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 9DB65C06CA for ; Tue, 24 Nov 2015 06:52:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.751 X-Spam-Level: *** X-Spam-Status: No, score=3.751 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, KAM_INFOUSMEBIZ=0.75, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=databricks-com.20150623.gappssmtp.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id wtw5ffFD3Dxm for ; Tue, 24 Nov 2015 06:52:11 +0000 (UTC) Received: from mail-ob0-f173.google.com (mail-ob0-f173.google.com [209.85.214.173]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id E4B5220562 for ; Tue, 24 Nov 2015 06:52:10 +0000 (UTC) Received: by obdgf3 with SMTP id gf3so6219509obd.3 for ; Mon, 23 Nov 2015 22:52:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=databricks-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=J8cy+9TnXw9oEmfaGec1nyl0Xc72RNboSHm8IkObZZM=; b=hT9gGnSYWjwi7TB4stYr5KCobDzR1V11VpAfHL40GfnzpbxWFVo9qvQS+NSFTGzoba bh0M3dIYe0HtSgx5rmvwxBRxwDKrsUhUFITVt2Fjh7j5NxmO3dzL9F+AvA+uRIbIKRwJ iJoAoIpIrY2We8Bl1tAmTntAHNDYgJWc5JfycxT5JtoIs7e2oxC6gahSFnm0pKGAOEq4 XUgyGfIpSFN0U+p9KIAkIw7p7mNuF94YFFxYisgGte42zDSph+nid4O74Pd+gTt/1WsV o7NhMl7aJBa7uRq5vLqouoJ8U7J+YuESKBfqkxvAnC9y77/VglxPL1dWiIXJgKvE2dkM yEyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=J8cy+9TnXw9oEmfaGec1nyl0Xc72RNboSHm8IkObZZM=; b=LbQCXZWl4zC+i2EVbroO1lMt1jPUVL/DjwAGS98PDUjKER5tfvG8gZ4NZ72DvLNFn8 UCafwqVzwblGMegYSKiHCnxuphNtn/lxhneCbAd1G0bGxT17iDS1jlSS37Ax+szjttD4 IpWVnezes1RrS1Q7FXny7HUtINMx+U6PcazuzADdrE/YMw7GMHHS+7t8s2jOCq3d7OwB U91FsMn+5dTnPUhN9296A1Sg6sN5ciwV8mdYoHF6/6ykMIU7flr5LteNJ8cgpn5r25CY kbDVmI1+SEyj9wyjD/dz3XgxwdHS8eeWuILV8fKCPZQ3f+Uq8vxlrYAExWlORgCL/yMH 0i3A== X-Gm-Message-State: ALoCoQl5DuEC114tD8zuSmQ0D0dNZ1jHovFRE4rD/XUWQr1cJBQVIaN71v6IT+kAPwyKqjyo/gMq X-Received: by 10.60.144.66 with SMTP id sk2mr871863oeb.15.1448347930314; Mon, 23 Nov 2015 22:52:10 -0800 (PST) MIME-Version: 1.0 Received: by 10.202.49.18 with HTTP; Mon, 23 Nov 2015 22:51:50 -0800 (PST) In-Reply-To: References: <8D051899-9583-44D2-A838-258E29E8CD9B@gmail.com> <12E2B2F6075044F68EAA8B8153B97023@gmail.com> <9D5B00849D2CDA4386BDA89E83F69E6C194BD57E@G4W3292.americas.hpqcorp.net> <80833ADD533E324CA05C160E41B6366103D157BD@shsmsx102.ccr.corp.intel.com> <80833ADD533E324CA05C160E41B6366103D15A12@shsmsx102.ccr.corp.intel.com> From: Reynold Xin Date: Mon, 23 Nov 2015 22:51:50 -0800 Message-ID: Subject: Re: A proposal for Spark 2.0 To: Mark Hamstra Cc: Kostas Sakellis , "dev@spark.apache.org" Content-Type: multipart/alternative; boundary=047d7b2ed51de5e3c5052543c74b --047d7b2ed51de5e3c5052543c74b Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I actually think the next one (after 1.6) should be Spark 2.0. The reason is that I already know we have to break some part of the DataFrame/Dataset API as part of the Dataset design. (e.g. DataFrame.map should return Dataset rather than RDD). In that case, I'd rather break this sooner (in one release) than later (in two releases). so the damage is smaller. I don't think whether we call Dataset/DataFrame experimental or not matters too much for 2.0. We can still call Dataset experimental in 2.0 and then mark them as stable in 2.1. Despite being "experimental", there has been no breaking changes to DataFrame from 1.3 to 1.6. On Wed, Nov 18, 2015 at 3:43 PM, Mark Hamstra wrote: > Ah, got it; by "stabilize" you meant changing the API, not just bug > fixing. We're on the same page now. > > On Wed, Nov 18, 2015 at 3:39 PM, Kostas Sakellis > wrote: > >> A 1.6.x release will only fix bugs - we typically don't change APIs in z >> releases. The Dataset API is experimental and so we might be changing th= e >> APIs before we declare it stable. This is why I think it is important to >> first stabilize the Dataset API with a Spark 1.7 release before moving t= o >> Spark 2.0. This will benefit users that would like to use the new Datase= t >> APIs but can't move to Spark 2.0 because of the backwards incompatible >> changes, like removal of deprecated APIs, Scala 2.11 etc. >> >> Kostas >> >> >> On Fri, Nov 13, 2015 at 12:26 PM, Mark Hamstra >> wrote: >> >>> Why does stabilization of those two features require a 1.7 release >>> instead of 1.6.1? >>> >>> On Fri, Nov 13, 2015 at 11:40 AM, Kostas Sakellis >>> wrote: >>> >>>> We have veered off the topic of Spark 2.0 a little bit here - yes we >>>> can talk about RDD vs. DS/DF more but lets refocus on Spark 2.0. I'd l= ike >>>> to propose we have one more 1.x release after Spark 1.6. This will all= ow us >>>> to stabilize a few of the new features that were added in 1.6: >>>> >>>> 1) the experimental Datasets API >>>> 2) the new unified memory manager. >>>> >>>> I understand our goal for Spark 2.0 is to offer an easy transition but >>>> there will be users that won't be able to seamlessly upgrade given wha= t we >>>> have discussed as in scope for 2.0. For these users, having a 1.x rele= ase >>>> with these new features/APIs stabilized will be very beneficial. This = might >>>> make Spark 1.7 a lighter release but that is not necessarily a bad thi= ng. >>>> >>>> Any thoughts on this timeline? >>>> >>>> Kostas Sakellis >>>> >>>> >>>> >>>> On Thu, Nov 12, 2015 at 8:39 PM, Cheng, Hao >>>> wrote: >>>> >>>>> Agree, more features/apis/optimization need to be added in DF/DS. >>>>> >>>>> >>>>> >>>>> I mean, we need to think about what kind of RDD APIs we have to >>>>> provide to developer, maybe the fundamental API is enough, like, the >>>>> ShuffledRDD etc.. But PairRDDFunctions probably not in this category= , as >>>>> we can do the same thing easily with DF/DS, even better performance. >>>>> >>>>> >>>>> >>>>> *From:* Mark Hamstra [mailto:mark@clearstorydata.com] >>>>> *Sent:* Friday, November 13, 2015 11:23 AM >>>>> *To:* Stephen Boesch >>>>> >>>>> *Cc:* dev@spark.apache.org >>>>> *Subject:* Re: A proposal for Spark 2.0 >>>>> >>>>> >>>>> >>>>> Hmmm... to me, that seems like precisely the kind of thing that argue= s >>>>> for retaining the RDD API but not as the first thing presented to new= Spark >>>>> developers: "Here's how to use groupBy with DataFrames.... Until the >>>>> optimizer is more fully developed, that won't always get you the best >>>>> performance that can be obtained. In these particular circumstances,= ..., >>>>> you may want to use the low-level RDD API while setting >>>>> preservesPartitioning to true. Like this...." >>>>> >>>>> >>>>> >>>>> On Thu, Nov 12, 2015 at 7:05 PM, Stephen Boesch >>>>> wrote: >>>>> >>>>> My understanding is that the RDD's presently have more support for >>>>> complete control of partitioning which is a key consideration at scal= e. >>>>> While partitioning control is still piecemeal in DF/DS it would see= m >>>>> premature to make RDD's a second-tier approach to spark dev. >>>>> >>>>> >>>>> >>>>> An example is the use of groupBy when we know that the source relatio= n >>>>> (/RDD) is already partitioned on the grouping expressions. AFAIK the= spark >>>>> sql still does not allow that knowledge to be applied to the optimize= r - so >>>>> a full shuffle will be performed. However in the native RDD we can us= e >>>>> preservesPartitioning=3Dtrue. >>>>> >>>>> >>>>> >>>>> 2015-11-12 17:42 GMT-08:00 Mark Hamstra : >>>>> >>>>> The place of the RDD API in 2.0 is also something I've been wondering >>>>> about. I think it may be going too far to deprecate it, but changing >>>>> emphasis is something that we might consider. The RDD API came well = before >>>>> DataFrames and DataSets, so programming guides, introductory how-to >>>>> articles and the like have, to this point, also tended to emphasize R= DDs -- >>>>> or at least to deal with them early. What I'm thinking is that with = 2.0 >>>>> maybe we should overhaul all the documentation to de-emphasize and >>>>> reposition RDDs. In this scheme, DataFrames and DataSets would be >>>>> introduced and fully addressed before RDDs. They would be presented = as the >>>>> normal/default/standard way to do things in Spark. RDDs, in contrast= , >>>>> would be presented later as a kind of lower-level, closer-to-the-meta= l API >>>>> that can be used in atypical, more specialized contexts where DataFra= mes or >>>>> DataSets don't fully fit. >>>>> >>>>> >>>>> >>>>> On Thu, Nov 12, 2015 at 5:17 PM, Cheng, Hao >>>>> wrote: >>>>> >>>>> I am not sure what the best practice for this specific problem, but >>>>> it=E2=80=99s really worth to think about it in 2.0, as it is a painfu= l issue for >>>>> lots of users. >>>>> >>>>> >>>>> >>>>> By the way, is it also an opportunity to deprecate the RDD API (or >>>>> internal API only?)? As lots of its functionality overlapping with >>>>> DataFrame or DataSet. >>>>> >>>>> >>>>> >>>>> Hao >>>>> >>>>> >>>>> >>>>> *From:* Kostas Sakellis [mailto:kostas@cloudera.com] >>>>> *Sent:* Friday, November 13, 2015 5:27 AM >>>>> *To:* Nicholas Chammas >>>>> *Cc:* Ulanov, Alexander; Nan Zhu; witgo@qq.com; dev@spark.apache.org; >>>>> Reynold Xin >>>>> >>>>> >>>>> *Subject:* Re: A proposal for Spark 2.0 >>>>> >>>>> >>>>> >>>>> I know we want to keep breaking changes to a minimum but I'm hoping >>>>> that with Spark 2.0 we can also look at better classpath isolation wi= th >>>>> user programs. I propose we build on >>>>> spark.{driver|executor}.userClassPathFirst, setting it true by defaul= t, and >>>>> not allow any spark transitive dependencies to leak into user code. F= or >>>>> backwards compatibility we can have a whitelist if we want but I'd be= good >>>>> if we start requiring user apps to explicitly pull in all their >>>>> dependencies. From what I can tell, Hadoop 3 is also moving in this >>>>> direction. >>>>> >>>>> >>>>> >>>>> Kostas >>>>> >>>>> >>>>> >>>>> On Thu, Nov 12, 2015 at 9:56 AM, Nicholas Chammas < >>>>> nicholas.chammas@gmail.com> wrote: >>>>> >>>>> With regards to Machine learning, it would be great to move useful >>>>> features from MLlib to ML and deprecate the former. Current structure= of >>>>> two separate machine learning packages seems to be somewhat confusing= . >>>>> >>>>> With regards to GraphX, it would be great to deprecate the use of RDD >>>>> in GraphX and switch to Dataframe. This will allow GraphX evolve with >>>>> Tungsten. >>>>> >>>>> On that note of deprecating stuff, it might be good to deprecate some >>>>> things in 2.0 without removing or replacing them immediately. That wa= y 2.0 >>>>> doesn=E2=80=99t have to wait for everything that we want to deprecate= to be >>>>> replaced all at once. >>>>> >>>>> Nick >>>>> >>>>> =E2=80=8B >>>>> >>>>> >>>>> >>>>> On Thu, Nov 12, 2015 at 12:45 PM Ulanov, Alexander < >>>>> alexander.ulanov@hpe.com> wrote: >>>>> >>>>> Parameter Server is a new feature and thus does not match the goal of >>>>> 2.0 is =E2=80=9Cto fix things that are broken in the current API and = remove certain >>>>> deprecated APIs=E2=80=9D. At the same time I would be happy to have t= hat feature. >>>>> >>>>> >>>>> >>>>> With regards to Machine learning, it would be great to move useful >>>>> features from MLlib to ML and deprecate the former. Current structure= of >>>>> two separate machine learning packages seems to be somewhat confusing= . >>>>> >>>>> With regards to GraphX, it would be great to deprecate the use of RDD >>>>> in GraphX and switch to Dataframe. This will allow GraphX evolve with >>>>> Tungsten. >>>>> >>>>> >>>>> >>>>> Best regards, Alexander >>>>> >>>>> >>>>> >>>>> *From:* Nan Zhu [mailto:zhunanmcgill@gmail.com] >>>>> *Sent:* Thursday, November 12, 2015 7:28 AM >>>>> *To:* witgo@qq.com >>>>> *Cc:* dev@spark.apache.org >>>>> *Subject:* Re: A proposal for Spark 2.0 >>>>> >>>>> >>>>> >>>>> Being specific to Parameter Server, I think the current agreement is >>>>> that PS shall exist as a third-party library instead of a component o= f the >>>>> core code base, isn=E2=80=99t? >>>>> >>>>> >>>>> >>>>> Best, >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Nan Zhu >>>>> >>>>> http://codingcat.me >>>>> >>>>> >>>>> >>>>> On Thursday, November 12, 2015 at 9:49 AM, witgo@qq.com wrote: >>>>> >>>>> Who has the idea of machine learning? Spark missing some features for >>>>> machine learning, For example, the parameter server. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> =E5=9C=A8 2015=E5=B9=B411=E6=9C=8812=E6=97=A5=EF=BC=8C05:32=EF=BC=8CM= atei Zaharia =E5=86=99=E9=81=93=EF=BC=9A >>>>> >>>>> >>>>> >>>>> I like the idea of popping out Tachyon to an optional component too t= o >>>>> reduce the number of dependencies. In the future, it might even be us= eful >>>>> to do this for Hadoop, but it requires too many API changes to be wor= th >>>>> doing now. >>>>> >>>>> >>>>> >>>>> Regarding Scala 2.12, we should definitely support it eventually, but >>>>> I don't think we need to block 2.0 on that because it can be added la= ter >>>>> too. Has anyone investigated what it would take to run on there? I im= agine >>>>> we don't need many code changes, just maybe some REPL stuff. >>>>> >>>>> >>>>> >>>>> Needless to say, but I'm all for the idea of making "major" releases >>>>> as undisruptive as possible in the model Reynold proposed. Keeping ev= eryone >>>>> working with the same set of releases is super important. >>>>> >>>>> >>>>> >>>>> Matei >>>>> >>>>> >>>>> >>>>> On Nov 11, 2015, at 4:58 AM, Sean Owen wrote: >>>>> >>>>> >>>>> >>>>> On Wed, Nov 11, 2015 at 12:10 AM, Reynold Xin >>>>> wrote: >>>>> >>>>> to the Spark community. A major release should not be very different >>>>> from a >>>>> >>>>> minor release and should not be gated based on new features. The main >>>>> >>>>> purpose of a major release is an opportunity to fix things that are >>>>> broken >>>>> >>>>> in the current API and remove certain deprecated APIs (examples >>>>> follow). >>>>> >>>>> >>>>> >>>>> Agree with this stance. Generally, a major release might also be a >>>>> >>>>> time to replace some big old API or implementation with a new one, bu= t >>>>> >>>>> I don't see obvious candidates. >>>>> >>>>> >>>>> >>>>> I wouldn't mind turning attention to 2.x sooner than later, unless >>>>> >>>>> there's a fairly good reason to continue adding features in 1.x to a >>>>> >>>>> 1.7 release. The scope as of 1.6 is already pretty darned big. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> 1. Scala 2.11 as the default build. We should still support Scala >>>>> 2.10, but >>>>> >>>>> it has been end-of-life. >>>>> >>>>> >>>>> >>>>> By the time 2.x rolls around, 2.12 will be the main version, 2.11 wil= l >>>>> >>>>> be quite stable, and 2.10 will have been EOL for a while. I'd propose >>>>> >>>>> dropping 2.10. Otherwise it's supported for 2 more years. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> 2. Remove Hadoop 1 support. >>>>> >>>>> >>>>> >>>>> I'd go further to drop support for <2.2 for sure (2.0 and 2.1 were >>>>> >>>>> sort of 'alpha' and 'beta' releases) and even <2.6. >>>>> >>>>> >>>>> >>>>> I'm sure we'll think of a number of other small things -- shading a >>>>> >>>>> bunch of stuff? reviewing and updating dependencies in light of >>>>> >>>>> simpler, more recent dependencies to support from Hadoop etc? >>>>> >>>>> >>>>> >>>>> Farming out Tachyon to a module? (I felt like someone proposed this?) >>>>> >>>>> Pop out any Docker stuff to another repo? >>>>> >>>>> Continue that same effort for EC2? >>>>> >>>>> Farming out some of the "external" integrations to another repo (? >>>>> >>>>> controversial) >>>>> >>>>> >>>>> >>>>> See also anything marked version "2+" in JIRA. >>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> >>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org >>>>> >>>>> For additional commands, e-mail: dev-help@spark.apache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> >>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org >>>>> >>>>> For additional commands, e-mail: dev-help@spark.apache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> >>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org >>>>> >>>>> For additional commands, e-mail: dev-help@spark.apache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >> > --047d7b2ed51de5e3c5052543c74b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I actually think the next one (after 1.6) should be Spark = 2.0. The reason is that I already know we have to break some part of the Da= taFrame/Dataset API as part of the Dataset design. (e.g. DataFrame.map shou= ld return Dataset rather than RDD). In that case, I'd rather break this= sooner (in one release) than later (in two releases). so the damage is sma= ller.

I don't think whether we call Dataset/DataFram= e experimental or not matters too much for 2.0. We can still call Dataset e= xperimental in 2.0 and then mark them as stable in 2.1. Despite being "= ;experimental", there has been no breaking changes to DataFrame from 1= .3 to 1.6.



On Wed, Nov 18, 2015 at 3:43 PM, Mark Ha= mstra <mark@clearstorydata.com> wrote:
Ah, got it; by "stabilize" yo= u meant changing the API, not just bug fixing.=C2=A0 We're on the same = page now.

On Wed, Nov 18, 2015 at 3:39 PM, Kost= as Sakellis <kostas@cloudera.com> wrote:
A 1.6.x release will only fix bugs -= we typically don't change APIs in z releases. The Dataset API is exper= imental and so we might be changing the APIs before we declare it stable. T= his is why I think it is important to first stabilize the Dataset API with = a Spark 1.7 release before moving to Spark 2.0. This will benefit users tha= t would like to use the new Dataset APIs but can't move to Spark 2.0 be= cause of the backwards incompatible changes, like removal of deprecated API= s, Scala 2.11 etc.

Kostas

<= /font>

On Fri, Nov 13, 2015 at 12:26 PM, Mark Hamstra <mark@cle= arstorydata.com> wrote:
Why does stabilization of those two features require a 1.7 re= lease instead of 1.6.1?

On Fri, Nov 13, 2015 at 11:40 AM, Kostas Sakellis <kostas@cloudera.com> wrote:
We have veered off the topic of Spark 2.0 a little = bit here - yes we can talk about RDD vs. DS/DF more but lets refocus on Spa= rk 2.0. I'd like to propose we have one more 1.x release after Spark 1.= 6. This will allow us to stabilize a few of the new features that were adde= d in 1.6:

1) the experimental Datasets API
2) the new unified memory manager.

I understand= our goal for Spark 2.0 is to offer an easy transition but there will be us= ers that won't be able to seamlessly upgrade given what we have discuss= ed as in scope for 2.0. For these users, having a 1.x release with these ne= w features/APIs stabilized will be very beneficial. This might make Spark 1= .7 a lighter release but that is not necessarily a bad thing.
Any thoughts on this timeline?

Kostas Sakellis


<= /font>

On Thu, Nov 12, 2015 at 8:39 PM, Cheng, Hao &= lt;hao.cheng@intel= .com> wrote:

Agree, more features/apis/optimizatio= n need to be added in DF/DS.

=C2=A0

I mean, we need to think about what k= ind of RDD APIs we have to provide to developer, maybe the fundamental API = is enough, like, the ShuffledRDD etc..=C2=A0 But PairRDDFunctions probably not in this category, as we can do the same thing easily with DF/= DS, even better performance.

=C2=A0

From: Mark Hamstra [mailto:mark@clearstorydata.c= om]
Sent: Friday, November 13, 2015 11:23 AM
To: Stephen Boesch


Cc: dev@sp= ark.apache.org
Subject: Re: A proposal for Spark 2.0

=C2=A0

Hmmm... to me, that seems like precisely the kind of= thing that argues for retaining the RDD API but not as the first thing pre= sented to new Spark developers: "Here's how to use groupBy with Da= taFrames.... Until the optimizer is more fully developed, that won't always get you the best performance that can be = obtained.=C2=A0 In these particular circumstances, ..., you may want to use= the low-level RDD API while setting preservesPartitioning to true.=C2=A0 L= ike this...."

=C2=A0

On Thu, Nov 12, 2015 at 7:05 PM, Stephen Boesch <= javadba@gmail.com> wrote:

My understanding is that =C2=A0the RDD's present= ly have more support for complete control of partitioning which is a key co= nsideration at scale.=C2=A0 While partitioning control is still piecemeal i= n =C2=A0DF/DS =C2=A0it would seem premature to make RDD's a second-tier approach to spark dev. =C2=A0 =C2=A0

=C2=A0

An example is the use of groupBy when we know that t= he source relation (/RDD) is already partitioned on the grouping expression= s.=C2=A0 AFAIK the spark sql still does not allow that knowledge to be appl= ied to the optimizer - so a full shuffle will be performed. However in the native RDD we can use preservesPartition= ing=3Dtrue.

=C2=A0

2015-11-12 17:42 GMT-08:00 Mark Hamstra <mark@clearstorydata.c= om>:

The place of the RDD API in 2.0 is also something I&= #39;ve been wondering about.=C2=A0 I think it may be going too far to depre= cate it, but changing emphasis is something that we might consider.=C2=A0 T= he RDD API came well before DataFrames and DataSets, so programming guides, introductory how-to articles and the like have, to = this point, also tended to emphasize RDDs -- or at least to deal with them = early.=C2=A0 What I'm thinking is that with 2.0 maybe we should overhau= l all the documentation to de-emphasize and reposition RDDs.=C2=A0 In this scheme, DataFrames and DataSets would be in= troduced and fully addressed before RDDs.=C2=A0 They would be presented as = the normal/default/standard way to do things in Spark.=C2=A0 RDDs, in contr= ast, would be presented later as a kind of lower-level, closer-to-the-metal API that can be used in atypical, more specialized con= texts where DataFrames or DataSets don't fully fit.=C2=A0=

=C2=A0

On Thu, Nov 12, 2015 at 5:17 PM, Cheng, Hao <hao.cheng@intel.com> wrote:

I am not sure what the best practice = for this specific problem, but it=E2=80=99s really worth to think about it in 2.0, as it is a painful issue for lots of users.

=C2=A0

By the way, is it also an opportunity= to deprecate the RDD API (or internal API only?)? As lots of its functionality overlapping with DataFrame or DataSet.<= /u>

=C2=A0

Hao

=C2=A0

From: Kostas Sakellis [mailto:kostas@cloudera.com]
Sent: Friday, November 13, 2015 5:27 AM
To: Nicholas Chammas
Cc: Ulanov, Alexander; Nan Zhu; witgo@qq.com; dev@spark.apache.org; Reynold Xin


Subject: Re: A proposal for Spark 2.0

=C2=A0

I know we want to keep breaking changes to a minimum= but I'm hoping that with Spark 2.0 we can also look at better classpat= h isolation with user programs. I propose we build on spark.{driver|executor}.userClassPathFirst, setting it true by default, an= d not allow any spark transitive dependencies to leak into user code. For b= ackwards compatibility we can have a whitelist if we want but I'd be go= od if we start requiring user apps to explicitly pull in all their dependencies. From what I can tell, Hadoop 3 = is also moving in this direction.

=C2=A0

Kostas

=C2=A0

On Thu, Nov 12, 2015 at 9:56 AM, Nicholas Chammas &l= t;nicholas.= chammas@gmail.com> wrote:

With rega= rds to Machine learning, it would be great to move useful features from MLl= ib to ML and deprecate the former. Current structure of two separate machin= e learning packages seems to be somewhat confusing.

With rega= rds to GraphX, it would be great to deprecate the use of RDD in GraphX and = switch to Dataframe. This will allow GraphX evolve with Tungsten.=

On that note of deprecating stuff, it m= ight be good to deprecate some things in 2.0 without removing or replacing = them immediately. That way 2.0 doesn=E2=80=99t have to wait for everything = that we want to deprecate to be replaced all at once.

Nick

=E2=80=8B<= /u>

=C2=A0

On Thu, Nov 12, 2015 at 12:45 PM Ulanov, Alexander &= lt;alexander.= ulanov@hpe.com> wrote:

Parameter Server is a new feature and= thus does not match the goal of 2.0 is =E2=80=9Cto fix things that are broken in the current API and remove certain deprecated APIs=E2=80=9D. At = the same time I would be happy to have that feature.

=C2=A0

With regards to Machine learning, it = would be great to move useful features from MLlib to ML and deprecate the former. Current structure of two separate machine learning packages se= ems to be somewhat confusing.

With regards to GraphX, it would be g= reat to deprecate the use of RDD in GraphX and switch to Dataframe. This will allow GraphX evolve with Tungsten.

=C2=A0

Best regards, Alexander=

=C2=A0

From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
Sent: Thursday, November 12, 2015 7:28 AM
To: witgo@qq.com
Cc: dev@sp= ark.apache.org
Subject: Re: A proposal for Spark 2.0

=C2=A0

Being specific to P= arameter Server, I think the current agreement is that PS shall=C2=A0exist = as a third-party library instead of a component of the core code base, isn=E2=80=99t?

=C2=A0

Best,=

=C2=A0

--=C2=A0

Nan Zhu

=C2=A0

On Thursday, November 12, 2015 at 9:49 AM,= witgo@qq.com wrote:

Who has the idea of machine learning? Spark missing = some features for machine learning, For example, the parameter server.

=C2=A0

=C2=A0

=E5=9C=A8 2015=E5=B9=B411=E6=9C=8812=E6=97=A5=EF=BC=8C05:32=EF=BC=8CMatei Zaharia <m= atei.zaharia@gmail.com> =E5=86=99=E9= =81=93=EF=BC=9A

=C2=A0

I like the idea of popping out Tachyon to an optiona= l component too to reduce the number of dependencies. In the future, it mig= ht even be useful to do this for Hadoop, but it requires too many API changes to be worth doing now.

=C2=A0

Regarding Scala 2.12, we should definitely support i= t eventually, but I don't think we need to block 2.0 on that because it= can be added later too. Has anyone investigated what it would take to run on there? I imagine we don't need many code chang= es, just maybe some REPL stuff.

=C2=A0

Needless to say, but I'm all for the idea of mak= ing "major" releases as undisruptive as possible in the model Rey= nold proposed. Keeping everyone working with the same set of releases is super important.

=C2=A0

Matei

=C2=A0

On Nov 11, 2015, at 4:58 AM, Sean Owen <sowen@cloudera.com>= wrote:

=C2=A0

On Wed, Nov 11, 2015 at 12:10 AM, Reynold Xin <rxin@databricks.com<= /a>> wrote:

to the Spark community. A major release should not b= e very different from a

minor release and should not be gated based on new f= eatures. The main

purpose of a major release is an opportunity to fix = things that are broken

in the current API and remove certain deprecated API= s (examples follow).

=C2=A0

Agree with this stance. Generally, a major release m= ight also be a

time to replace some big old API or implementation w= ith a new one, but

I don't see obvious candidates.

=C2=A0

I wouldn't mind turning attention to 2.x sooner = than later, unless

there's a fairly good reason to continue adding = features in 1.x to a

1.7 release. The scope as of 1.6 is already pretty d= arned big.

=C2=A0

=C2=A0

1. Scala 2.11 as the default build. We should still = support Scala 2.10, but

it has been end-of-life.

=C2=A0

By the time 2.x rolls around, 2.12 will be the main = version, 2.11 will

be quite stable, and 2.10 will have been EOL for a w= hile. I'd propose

dropping 2.10. Otherwise it's supported for 2 mo= re years.

=C2=A0

=C2=A0

2. Remove Hadoop 1 support.

=C2=A0

I'd go further to drop support for <2.2 for s= ure (2.0 and 2.1 were

sort of 'alpha' and 'beta' releases)= and even <2.6.

=C2=A0

I'm sure we'll think of a number of other sm= all things -- shading a

bunch of stuff? reviewing and updating dependencies = in light of

simpler, more recent dependencies to support from Ha= doop etc?

=C2=A0

Farming out Tachyon to a module? (I felt like someon= e proposed this?)

Pop out any Docker stuff to another repo?<= /u>

Continue that same effort for EC2?

Farming out some of the "external" integra= tions to another repo (?

controversial)

=C2=A0

See also anything marked version "2+" in J= IRA.

=C2=A0

----------------------------------------------------= -----------------

To unsubscribe, e-mail: dev-u= nsubscribe@spark.apache.org

For additional commands, e-mail: dev-help@spa= rk.apache.org

=C2=A0

=C2=A0

----------------------------------------------------= -----------------

To unsubscribe, e-mail: dev-u= nsubscribe@spark.apache.org

For additional commands, e-mail: dev-help@spa= rk.apache.org

=C2=A0

=C2=A0

=C2=A0

=C2=A0

----------------------------------------------------= -----------------

To unsubscribe, e-mail: dev-u= nsubscribe@spark.apache.org

For additional commands, e-mail: dev-help@spa= rk.apache.org

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0






--047d7b2ed51de5e3c5052543c74b--