From dev-return-24828-archive-asf-public=cust-asf.ponee.io@spark.apache.org Fri Jun 8 03:29:01 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id DA249180663 for ; Fri, 8 Jun 2018 03:28:59 +0200 (CEST) Received: (qmail 96402 invoked by uid 500); 8 Jun 2018 01:28:58 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 96384 invoked by uid 99); 8 Jun 2018 01:28:57 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jun 2018 01:28:57 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 2FEA4C005C for ; Fri, 8 Jun 2018 01:28:57 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.87 X-Spam-Level: * X-Spam-Status: No, score=1.87 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id yPOd6tQ0l0J8 for ; Fri, 8 Jun 2018 01:28:54 +0000 (UTC) Received: from mail-lf0-f68.google.com (mail-lf0-f68.google.com [209.85.215.68]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 8B6985F1E6 for ; Fri, 8 Jun 2018 01:28:53 +0000 (UTC) Received: by mail-lf0-f68.google.com with SMTP id y20-v6so17556662lfy.0 for ; Thu, 07 Jun 2018 18:28:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=vxhQ4bXzKHp4f0ji+MfihbNinj9qE2HjURAbc9GLZ5E=; b=luUDtKKaKEuO2TTMtLixwuoXayqF6n3EfT4445HsZYnSDQNrrPbcfgHYj/ewvAx4E/ rZ/n3LpI91QU0qZWBLU3LEWtkrRVnzrfvN59dqtjefwASOriE8qq7kjesyuIJwLQrNAk oaXINBK8R0uGcUe3ROFYfOWJxKzAaKqNmjc8G1HCBgw2o2xor8mDu/ZRNX3xUAkEgPa7 ujlIt6qXTAo2o4dRZPuAvojiVG96ius8R5a3WhbrKhVWObi09QkTRgZfqW3T5YkVyHVL sdVLJNMKecLPuSN2NOn8RVSJo3UY+lLA8TksF2AAh1kZKk1tx8xU0GiTrSx0wfeoCESo s2tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=vxhQ4bXzKHp4f0ji+MfihbNinj9qE2HjURAbc9GLZ5E=; b=P/V9XuXFqfUYLW6/IisKy6aUvKGECj+wjYGytHD1H3Kjl18G6Qs4f1GC5bsp7RxihG qQBX4at8SRqBVnT+ZWJYlx2rYfCCH0xZKV7RDoN9BeIb4k4q9O29f4Uk+9mpByjTACdr DrpQ9scPXicM34E0z40W1jMR8lnj5RYiIglmmcLBoDeEZn03qJ5K0R01aEV465gqg4a7 j24s4C+YR/QdtHYsocJR3SGZm/Xr0/vLRFwmY1l4B502HMIgvxdtVs0pk0RCUcbu7c/A 3c6dDY3aMD3yP6xDo6oHrim5XhvLgbQ8uiYllmWXQ0lp2EsFWset5mTuJDx1CnU+jb53 5kjw== X-Gm-Message-State: APt69E0HRuzpQxBQ8807Z293iSWs3t0TKJOkYQzrTiv+Y06e9qXh/gYU Mo51nNRzjGk+XmYBHGWFaxkyPJhAHMAAAjSeRiU= X-Google-Smtp-Source: ADUXVKKEjUPI8XeiRd82qYcszLiyao2rtzlruLKQRJatPl7JrzweQwB19HYhu0ky5w3GFBgVhpCx9V1agdLp6Scx1jk= X-Received: by 2002:a2e:4302:: with SMTP id q2-v6mr3103595lja.55.1528421333049; Thu, 07 Jun 2018 18:28:53 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a19:df58:0:0:0:0:0 with HTTP; Thu, 7 Jun 2018 18:28:52 -0700 (PDT) In-Reply-To: References: <655CB8A1-A742-4650-B7F0-3DAAD049D2C4@apple.com> <36DD5BA5-CCA6-40AE-9EAE-4F7A78CB4C66@apple.com> From: Wenchen Fan Date: Thu, 7 Jun 2018 18:28:52 -0700 Message-ID: Subject: Re: Scala 2.12 support To: Marcelo Vanzin Cc: Holden Karau , DB Tsai , Dean Wampler , Felix Cheung , Som Snytt , Sean Owen , Reynold Xin , dev Content-Type: multipart/alternative; boundary="0000000000009fa62a056e17529f" --0000000000009fa62a056e17529f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable One more point: There was a time that we maintain 2 Spark REPL codebase for Scala 2.10 and 2.11, maybe we can do the same for Scala 2.11 and 2.12? if it's too hard to find a common way to do that between different Scala versions. On Thu, Jun 7, 2018 at 6:20 PM, Marcelo Vanzin wrote: > But DB's shell output is on the most recent 2.11, not 2.12, right? > > On Thu, Jun 7, 2018 at 5:54 PM, Holden Karau wrote= : > > I agree that's a little odd, could we not add the bacspace terminal > > character? Regardless even if not, I don't think that should be a block= er > > for 2.12 support especially since it doesn't degrade the 2.11 experienc= e. > > > > On Thu, Jun 7, 2018, 5:53 PM DB Tsai wrote: > >> > >> If we decide to initialize Spark in `initializeSynchronous()` in Scala > >> 2.11.12, it will look like the following which is odd. > >> > >> Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java > >> 1.8.0_161) > >> Type in expressions to have them evaluated. > >> Type :help for more information. > >> > >> scala> Spark context Web UI available at http://192.168.1.169:4040 > >> Spark context available as 'sc' (master =3D local[*], app id =3D > >> local-1528180279528). > >> Spark session available as 'spark=E2=80=99. > >> scala> > >> > >> DB Tsai | Siri Open Source Technologies [not a contribution] | =EF= =A3=BF > >> Apple, Inc > >> > >> On Jun 7, 2018, at 5:49 PM, Holden Karau wrote: > >> > >> Tests can just be changed to accept either output too :p > >> > >> On Thu, Jun 7, 2018, 5:19 PM Dean Wampler > wrote: > >>> > >>> Do the tests expect a particular console output order? That would ann= oy > >>> them. ;) You could sort the expected and output lines, then diff... > >>> > >>> Dean Wampler, Ph.D. > >>> VP, Fast Data Engineering at Lightbend > >>> Author: Programming Scala, 2nd Edition, Fast Data Architectures for > >>> Streaming Applications, and other content from O'Reilly > >>> @deanwampler > >>> http://polyglotprogramming.com > >>> https://github.com/deanwampler > >>> > >>> On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau > >>> wrote: > >>>> > >>>> If the difference is the order of the welcome message I think that > >>>> should be fine. > >>>> > >>>> On Thu, Jun 7, 2018, 4:43 PM Dean Wampler > wrote: > >>>>> > >>>>> I'll point the Scala team to this issue, but it's unlikely to get > fixed > >>>>> any time soon. > >>>>> > >>>>> dean > >>>>> > >>>>> Dean Wampler, Ph.D. > >>>>> VP, Fast Data Engineering at Lightbend > >>>>> Author: Programming Scala, 2nd Edition, Fast Data Architectures for > >>>>> Streaming Applications, and other content from O'Reilly > >>>>> @deanwampler > >>>>> http://polyglotprogramming.com > >>>>> https://github.com/deanwampler > >>>>> > >>>>> On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai wrote: > >>>>>> > >>>>>> Thanks Felix for bringing this up. > >>>>>> > >>>>>> Currently, in Scala 2.11.8, we initialize the Spark by overriding > >>>>>> loadFIles() before REPL sees any file since there is no good hook > in Scala > >>>>>> to load our initialization code. > >>>>>> > >>>>>> In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles(= ) > >>>>>> method was removed. > >>>>>> > >>>>>> Alternatively, one way we can do in the newer version of Scala is = by > >>>>>> overriding initializeSynchronous() suggested by Som Snytt; I have = a > working > >>>>>> PR with this approach, > >>>>>> https://github.com/apache/spark/pull/21495 , and this approach > should > >>>>>> work for older version of Scala too. > >>>>>> > >>>>>> However, in the newer version of Scala, the first thing that the > REPL > >>>>>> calls is printWelcome, so in the newer version of Scala, welcome > message > >>>>>> will be shown and then the URL of the SparkUI in this approach. > This will > >>>>>> cause UI inconsistencies between different versions of Scala. > >>>>>> > >>>>>> We can also initialize the Spark in the printWelcome which I feel > more > >>>>>> hacky. It will only work for newer version of Scala since in order > version > >>>>>> of Scala, printWelcome is called in the end of the initialization > process. > >>>>>> If we decide to go this route, basically users can not use Scala > older than > >>>>>> 2.11.9. > >>>>>> > >>>>>> I think this is also a blocker for us to move to newer version of > >>>>>> Scala 2.12.x since the newer version of Scala 2.12.x has the same > issue. > >>>>>> > >>>>>> In my opinion, Scala should fix the root cause and provide a stabl= e > >>>>>> hook for 3rd party developers to initialize their custom code. > >>>>>> > >>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] | = =EF=A3=BF > >>>>>> Apple, Inc > >>>>>> > >>>>>> > On Jun 7, 2018, at 6:43 AM, Felix Cheung < > felixcheung_m@hotmail.com> > >>>>>> > wrote: > >>>>>> > > >>>>>> > +1 > >>>>>> > > >>>>>> > Spoke to Dean as well and mentioned the problem with 2.11.12 > >>>>>> > https://github.com/scala/bug/issues/10913 > >>>>>> > > >>>>>> > _____________________________ > >>>>>> > From: Sean Owen > >>>>>> > Sent: Wednesday, June 6, 2018 12:23 PM > >>>>>> > Subject: Re: Scala 2.12 support > >>>>>> > To: Holden Karau > >>>>>> > Cc: Dean Wampler , Reynold Xin > >>>>>> > , dev > >>>>>> > > >>>>>> > > >>>>>> > If it means no change to 2.11 support, seems OK to me for Spark > >>>>>> > 2.4.0. The 2.12 support is separate and has never been mutually > compatible > >>>>>> > with 2.11 builds anyway. (I also hope, suspect that the changes > are minimal; > >>>>>> > tests are already almost entirely passing with no change to the > closure > >>>>>> > cleaner when built for 2.12) > >>>>>> > > >>>>>> > On Wed, Jun 6, 2018 at 1:33 PM Holden Karau > > >>>>>> > wrote: > >>>>>> > Just chatted with Dean @ the summit and it sounds like from > Adriaan > >>>>>> > there is a fix in 2.13 for the API change issue that could be > back ported to > >>>>>> > 2.12 so how about we try and get this ball rolling? > >>>>>> > > >>>>>> > It sounds like it would also need a closure cleaner change, whic= h > >>>>>> > could be backwards compatible but since it=E2=80=99s such a core > component and we > >>>>>> > might want to be cautious with it, we could when building for > 2.11 use the > >>>>>> > old cleaner code and for 2.12 use the new code so we don=E2=80= =99t break > anyone. > >>>>>> > > >>>>>> > How do folks feel about this? > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > >>>>> > >>> > >> > > > > > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org > > --0000000000009fa62a056e17529f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
One more point: There was a time that we maintain 2 Spark = REPL codebase for Scala 2.10 and 2.11, maybe we can do the same for Scala 2= .11 and 2.12? if it's too hard to find a common way to do that between = different Scala versions.

On Thu, Jun 7, 2018 at 6:20 PM, Marcelo Vanzin <vanzin@clo= udera.com> wrote:
But DB&#= 39;s shell output is on the most recent 2.11, not 2.12, right?

On Thu, Jun 7, 2018 at 5:54 PM, Holden Karau <holden@pigscanfly.ca> wrote:
> I agree that's a little odd, could we not add the bacspace termina= l
> character? Regardless even if not, I don't think that should be a = blocker
> for 2.12 support especially since it doesn't degrade the 2.11 expe= rience.
>
> On Thu, Jun 7, 2018, 5:53 PM DB Tsai <d_tsai@apple.com> wrote:
>>
>> If we decide to initialize Spark in `initializeSynchronous()` in S= cala
>> 2.11.12, it will look like the following which is odd.
>>
>> Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Ja= va
>> 1.8.0_161)
>> Type in expressions to have them evaluated.
>> Type :help for more information.
>>
>> scala> Spark context Web UI available at http://192.168.1.169:4= 040
>> Spark context available as 'sc' (master =3D local[*], app = id =3D
>> local-1528180279528).
>> Spark session available as 'spark=E2=80=99.
>> scala>
>>
>> DB Tsai=C2=A0 |=C2=A0 Siri Open Source Technologies [not a contrib= ution]=C2=A0 |=C2=A0 =EF=A3=BF
>> Apple, Inc
>>
>> On Jun 7, 2018, at 5:49 PM, Holden Karau <holden@pigscanfly.ca> wrote:
>>
>> Tests can just be changed to accept either output too :p
>>
>> On Thu, Jun 7, 2018, 5:19 PM Dean Wampler <deanwampler@gmail.com> wrote:
>>>
>>> Do the tests expect a particular console output order? That wo= uld annoy
>>> them. ;) You could sort the expected and output lines, then di= ff...
>>>
>>> Dean Wampler, Ph.D.
>>> VP, Fast Data Engineering at Lightbend
>>> Author: Programming Scala, 2nd Edition, Fast Data Architecture= s for
>>> Streaming Applications, and other content from O'Reilly >>> @deanwampler
>>> http://polyglotprogramming.com
>>> https://github.com/deanwampler
>>>
>>> On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau <holden@pigscanfly.ca>
>>> wrote:
>>>>
>>>> If the difference is the order of the welcome message I th= ink that
>>>> should be fine.
>>>>
>>>> On Thu, Jun 7, 2018, 4:43 PM Dean Wampler <deanwampler@gmail.com> wrote:
>>>>>
>>>>> I'll point the Scala team to this issue, but it= 9;s unlikely to get fixed
>>>>> any time soon.
>>>>>
>>>>> dean
>>>>>
>>>>> Dean Wampler, Ph.D.
>>>>> VP, Fast Data Engineering at Lightbend
>>>>> Author: Programming Scala, 2nd Edition, Fast Data Arch= itectures for
>>>>> Streaming Applications, and other content from O'R= eilly
>>>>> @deanwampler
>>>>> http://polyglotprogramming.com
>>>>> https://github.com/deanwampler
>>>>>
>>>>> On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai <d_tsai@apple.com> wrote:
>>>>>>
>>>>>> Thanks Felix for bringing this up.
>>>>>>
>>>>>> Currently, in Scala 2.11.8, we initialize the Spar= k by overriding
>>>>>> loadFIles() before REPL sees any file since there = is no good hook in Scala
>>>>>> to load our initialization code.
>>>>>>
>>>>>> In Scala 2.11.12 and newer version of the Scala 2.= 12.x, loadFIles()
>>>>>> method was removed.
>>>>>>
>>>>>> Alternatively, one way we can do in the newer vers= ion of Scala is by
>>>>>> overriding initializeSynchronous() suggested by So= m Snytt; I have a working
>>>>>> PR with this approach,
>>>>>> https://github.com/apache/sp= ark/pull/21495 , and this approach should
>>>>>> work for older version of Scala too.
>>>>>>
>>>>>> However, in the newer version of Scala, the first = thing that the REPL
>>>>>> calls is printWelcome, so in the newer version of = Scala, welcome message
>>>>>> will be shown and then the URL of the SparkUI in t= his approach. This will
>>>>>> cause UI inconsistencies between different version= s of Scala.
>>>>>>
>>>>>> We can also initialize the Spark in the printWelco= me which I feel more
>>>>>> hacky. It will only work for newer version of Scal= a since in order version
>>>>>> of Scala, printWelcome is called in the end of the= initialization process.
>>>>>> If we decide to go this route, basically users can= not use Scala older than
>>>>>> 2.11.9.
>>>>>>
>>>>>> I think this is also a blocker for us to move to n= ewer version of
>>>>>> Scala 2.12.x since the newer version of Scala 2.12= .x has the same issue.
>>>>>>
>>>>>> In my opinion, Scala should fix the root cause and= provide a stable
>>>>>> hook for 3rd party developers to initialize their = custom code.
>>>>>>
>>>>>> DB Tsai=C2=A0 |=C2=A0 Siri Open Source Technologie= s [not a contribution]=C2=A0 |=C2=A0 =EF=A3=BF
>>>>>> Apple, Inc
>>>>>>
>>>>>> > On Jun 7, 2018, at 6:43 AM, Felix Cheung <= felixcheung_m@hotmail.com&= gt;
>>>>>> > wrote:
>>>>>> >
>>>>>> > +1
>>>>>> >
>>>>>> > Spoke to Dean as well and mentioned the probl= em with 2.11.12
>>>>>> > https://github.com/scala/bug/= issues/10913
>>>>>> >
>>>>>> > _____________________________
>>>>>> > From: Sean Owen <srowen@gmail.com>
>>>>>> > Sent: Wednesday, June 6, 2018 12:23 PM
>>>>>> > Subject: Re: Scala 2.12 support
>>>>>> > To: Holden Karau <holden@pigscanfly.ca>
>>>>>> > Cc: Dean Wampler <deanwampler@gmail.com>, Reynold Xin
>>>>>> > <rx= in@databricks.com>, dev <= dev@spark.apache.org>
>>>>>> >
>>>>>> >
>>>>>> > If it means no change to 2.11 support, seems = OK to me for Spark
>>>>>> > 2.4.0. The 2.12 support is separate and has n= ever been mutually compatible
>>>>>> > with 2.11 builds anyway. (I also hope, suspec= t that the changes are minimal;
>>>>>> > tests are already almost entirely passing wit= h no change to the closure
>>>>>> > cleaner when built for 2.12)
>>>>>> >
>>>>>> > On Wed, Jun 6, 2018 at 1:33 PM Holden Karau &= lt;holden@pigscanfly.ca>
>>>>>> > wrote:
>>>>>> > Just chatted with Dean @ the summit and it so= unds like from Adriaan
>>>>>> > there is a fix in 2.13 for the API change iss= ue that could be back ported to
>>>>>> > 2.12 so how about we try and get this ball ro= lling?
>>>>>> >
>>>>>> > It sounds like it would also need a closure c= leaner change, which
>>>>>> > could be backwards compatible but since it=E2= =80=99s such a core component and we
>>>>>> > might want to be cautious with it, we could w= hen building for 2.11 use the
>>>>>> > old cleaner code and for 2.12 use the new cod= e so we don=E2=80=99t break anyone.
>>>>>> >
>>>>>> > How do folks feel about this?
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>
>>
>



--
Marcelo

-----------------------------------------------------------------= ----
To unsubscribe e-mail: = dev-unsubscribe@spark.apache.org


--0000000000009fa62a056e17529f--