Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C114618D1C for ; Tue, 5 Jan 2016 22:33:50 +0000 (UTC) Received: (qmail 74059 invoked by uid 500); 5 Jan 2016 22:33:49 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 73962 invoked by uid 500); 5 Jan 2016 22:33:49 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 73951 invoked by uid 99); 5 Jan 2016 22:33:48 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2016 22:33:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 311F31A085B for ; Tue, 5 Jan 2016 22:33:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.001 X-Spam-Level: *** X-Spam-Status: No, score=3.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=tresata-com.20150623.gappssmtp.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 3ZccsA_NU4p8 for ; Tue, 5 Jan 2016 22:33:35 +0000 (UTC) Received: from mail-wm0-f52.google.com (mail-wm0-f52.google.com [74.125.82.52]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id A90FB201BC for ; Tue, 5 Jan 2016 22:33:34 +0000 (UTC) Received: by mail-wm0-f52.google.com with SMTP id f206so50916816wmf.0 for ; Tue, 05 Jan 2016 14:33:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tresata-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=YMCkmKn3nT8UHicbxn9BkMgZkRCkzw05cYU+S1D4CjA=; b=Y3MmCqwSDNHmZ75bM9Y8cOJj0hYmsS9RBXvmojFIjRrVKM7fAEDUv+VKXZXwxgr0q/ n24OR6SWo0f4BypYbl5ssfAG374+wJEQeH1BVK2mB5vAzNlWjYUMLNOGwVcysvIgLUPR lDp+ITcT+e5IiUlmJcHOz3q/kWvIEHR1GLbA6DwZBqWC4Pwz2nmtyQy4gIYN0rf3Y9Zt BNDDTcatHvLKenUA5f4Ae2V71WLLNC6aAL6NdFom1qVKM1dcXk8me8qLPB1qc3larPb3 1xtc6uhdI9lJwlJ+lU6faQhMG8ntZOwSkWR6gGpObsW78Q5HGNpOL/Nb5ARAleLz3DH7 2Hjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=YMCkmKn3nT8UHicbxn9BkMgZkRCkzw05cYU+S1D4CjA=; b=WT5GtPWrKGzB7Ec6aQnPFbVYq/Jsp7lK/oJ2AMR6ZV4eYq20vQNomZ0m+GmLaamO6d 2i7Ul9hJUIDk6VBli3HZKYKdIVAf6J9gT5P8rR5Zw4OScJyUW1qGk4ebgcqL2R2J3cjE SGQaZE2vrAvcDn1US8pxCWJpB/TYdonUkzL+LulJvK1hoiVyEyPpGQhJVW0G5UEfVwDB gSUfvCw2CYPdXZV0vTEzY2MVwYcOxXJE/tzYHozm3W80PZQJF8gZKga/ErOruYa2r41q yAG0lNDx1V1RY6mqsNCoRYBFsdVmltm11RCSdyQ8Ei9i24za7VlVNSrzbVg1jWb4NP8b OxbA== X-Gm-Message-State: ALoCoQnw9caNi6DKR/1Q4K6DCHsWJ2tzyI0VDKSdEH9R5wh9CNPESBaUaT7JAECS4sTLfrtHNAJ12Z2BeLCJ3XsQZwEePoZhEA== MIME-Version: 1.0 X-Received: by 10.28.22.199 with SMTP id 190mr6811470wmw.54.1452033206572; Tue, 05 Jan 2016 14:33:26 -0800 (PST) Received: by 10.194.33.169 with HTTP; Tue, 5 Jan 2016 14:33:26 -0800 (PST) X-Originating-IP: [204.148.13.62] In-Reply-To: References: <6718e1b9.d919.15211d83d82.Coremail.allenzhang010@126.com> <2BAAFF8A-BACF-451E-A60C-8C25A28C3CF3@esbet.es> Date: Tue, 5 Jan 2016 17:33:26 -0500 Message-ID: Subject: Re: [discuss] dropping Python 2.6 support From: Koert Kuipers To: Nicholas Chammas Cc: Julio Antonio Soto de Vicente , Juliet Hougland , Allen Zhang , Meethu Mathew , Reynold Xin , "dev@spark.apache.org" , user Content-Type: multipart/alternative; boundary=001a1146d9f07af4a205289dd3ab --001a1146d9f07af4a205289dd3ab Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable yeah, the practical concern is that we have no control over java or python version on large company clusters. our current reality for the vast majority of them is java 7 and python 2.6, no matter how outdated that is. i dont like it either, but i cannot change it. we currently don't use pyspark so i have no stake in this, but if we did i can assure you we would not upgrade to spark 2.x if python 2.6 was dropped. no point in developing something that doesnt run for majority of customers. On Tue, Jan 5, 2016 at 5:19 PM, Nicholas Chammas wrote: > As I pointed out in my earlier email, RHEL will support Python 2.6 until > 2020. So I'm assuming these large companies will have the option of ridin= g > out Python 2.6 until then. > > Are we seriously saying that Spark should likewise support Python 2.6 for > the next several years? Even though the core Python devs stopped supporti= ng > it in 2013? > > If that's not what we're suggesting, then when, roughly, can we drop > support? What are the criteria? > > I understand the practical concern here. If companies are stuck using 2.6= , > it doesn't matter to them that it is deprecated. But balancing that conce= rn > against the maintenance burden on this project, I would say that "upgrade > to Python 2.7 or stay on Spark 1.6.x" is a reasonable position to take. > There are many tiny annoyances one has to put up with to support 2.6. > > I suppose if our main PySpark contributors are fine putting up with those > annoyances, then maybe we don't need to drop support just yet... > > Nick > 2016=EB=85=84 1=EC=9B=94 5=EC=9D=BC (=ED=99=94) =EC=98=A4=ED=9B=84 2:27, = Julio Antonio Soto de Vicente =EB=8B=98=EC=9D=B4 > =EC=9E=91=EC=84=B1: > >> Unfortunately, Koert is right. >> >> I've been in a couple of projects using Spark (banking industry) where >> CentOS + Python 2.6 is the toolbox available. >> >> That said, I believe it should not be a concern for Spark. Python 2.6 is >> old and busted, which is totally opposite to the Spark philosophy IMO. >> >> >> El 5 ene 2016, a las 20:07, Koert Kuipers escribi=C3= =B3: >> >> rhel/centos 6 ships with python 2.6, doesnt it? >> >> if so, i still know plenty of large companies where python 2.6 is the >> only option. asking them for python 2.7 is not going to work >> >> so i think its a bad idea >> >> On Tue, Jan 5, 2016 at 1:52 PM, Juliet Hougland < >> juliet.hougland@gmail.com> wrote: >> >>> I don't see a reason Spark 2.0 would need to support Python 2.6. At thi= s >>> point, Python 3 should be the default that is encouraged. >>> Most organizations acknowledge the 2.7 is common, but lagging behind th= e >>> version they should theoretically use. Dropping python 2.6 >>> support sounds very reasonable to me. >>> >>> On Tue, Jan 5, 2016 at 5:45 AM, Nicholas Chammas < >>> nicholas.chammas@gmail.com> wrote: >>> >>>> +1 >>>> >>>> Red Hat supports Python 2.6 on REHL 5 until 2020 >>>> , >>>> but otherwise yes, Python 2.6 is ancient history and the core Python >>>> developers stopped supporting it in 2013. REHL 5 is not a good enough >>>> reason to continue support for Python 2.6 IMO. >>>> >>>> We should aim to support Python 2.7 and Python 3.3+ (which I believe w= e >>>> currently do). >>>> >>>> Nick >>>> >>>> On Tue, Jan 5, 2016 at 8:01 AM Allen Zhang >>>> wrote: >>>> >>>>> plus 1, >>>>> >>>>> we are currently using python 2.7.2 in production environment. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> =E5=9C=A8 2016-01-05 18:11:45=EF=BC=8C"Meethu Mathew" =E5=86=99=E9=81=93=EF=BC=9A >>>>> >>>>> +1 >>>>> We use Python 2.7 >>>>> >>>>> Regards, >>>>> >>>>> Meethu Mathew >>>>> >>>>> On Tue, Jan 5, 2016 at 12:47 PM, Reynold Xin >>>>> wrote: >>>>> >>>>>> Does anybody here care about us dropping support for Python 2.6 in >>>>>> Spark 2.0? >>>>>> >>>>>> Python 2.6 is ancient, and is pretty slow in many aspects (e.g. json >>>>>> parsing) when compared with Python 2.7. Some libraries that Spark de= pend on >>>>>> stopped supporting 2.6. We can still convince the library maintainer= s to >>>>>> support 2.6, but it will be extra work. I'm curious if anybody still= uses >>>>>> Python 2.6 to run Spark. >>>>>> >>>>>> Thanks. >>>>>> >>>>>> >>>>>> >>>>> >>> >> --001a1146d9f07af4a205289dd3ab Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
yeah, the practical concern is that we have no control over = java or python version on large company clusters. our current reality for t= he vast majority of them is java 7 and python 2.6, no matter how outdated t= hat is.

i dont like it either, but i cannot change it.


On Tue, Jan 5, 2016 at 5:19 PM, Nicholas Chammas &l= t;nicholas.= chammas@gmail.com> wrote:
A= s I pointed out in my earlier email, RHEL will support Python 2.6 until 202= 0. So I'm assuming these large companies will have the option of riding= out Python 2.6 until then.

Are we seriously saying that Spark shou= ld likewise support Python 2.6 for the next several years? Even though the = core Python devs stopped supporting it in 2013?

If that's not wh= at we're suggesting, then when, roughly, can we drop support? What are = the criteria?

I understand the practical concern here. If companies = are stuck using 2.6, it doesn't matter to them that it is deprecated. B= ut balancing that concern against the maintenance burden on this project, I= would say that "upgrade to Python 2.7 or stay on Spark 1.6.x" is= a reasonable position to take. There are many tiny annoyances one has to p= ut up with to support 2.6.

I suppose if our main PySpark contributo= rs are fine putting up with those annoyances, then maybe we don't need = to drop support just yet...

Nick
2016=EB=85=84 1=EC=9B=94 5=EC=9D=BC (=ED=99=94) =EC=98=A4=ED=9B= =84 2:27, Julio Antonio Soto de Vicente <julio@esbet.es>=EB=8B=98=EC=9D=B4 =EC=9E=91=EC= =84=B1:
Unfortunately, Koert is right.

= I've been in a couple of projects using Spark (banking industry) where = CentOS + Python 2.6 is the toolbox available.=C2=A0

That said, I believe it should not be a concern for Spark. Python 2.6 is = old and busted, which is totally opposite to the Spark philosophy IMO.
<= /div>


El 5 ene 20= 16, a las 20:07, Koert Kuipers <koert@tresata.com> escribi=C3=B3:

rhel/centos 6 ships with python 2.6= , doesnt it?

if so, i still know plenty of large companies where= python 2.6 is the only option. asking them for python 2.7 is not going to = work

so i think its a bad idea

On Tue, Jan 5, 2016 at 1:52 PM, Jul= iet Hougland <juliet.hougland@gmail.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
I don't see a reaso= n Spark 2.0 would need to support Python 2.6. At this point, Python 3 shoul= d be the default that is encouraged.
Most organizations acknowled= ge the 2.7 is common, but lagging behind the version they should theoretica= lly use. Dropping python 2.6
support sounds very reasonable to me= .

On Tue, Jan 5, 2016 at 5:45 AM, Nicholas Chammas <nicholas.cham= mas@gmail.com> wrote:
+1

Red Hat supports Python 2.6 on REHL 5 until 2020, but otherwis= e yes, Python 2.6 is ancient history and the core Python developers stopped= supporting it in 2013. REHL 5 is not a good enough reason to continue supp= ort for Python 2.6 IMO.

We should aim to support Python = 2.7 and Python 3.3+ (which I believe we currently do).

Nick

On Tue, Jan 5, 2016 at 8:01 AM Allen Zhang <allenzhang010@126.com> wrote= :
plus 1,

we are currently using python 2.7.2 in production environment.

On Tue, Jan 5, 2016 at 12:47 PM, Reynold Xin <rxin@databricks.com&= gt; wrote:
Does a= nybody here care about us dropping support for Python 2.6 in Spark 2.0?=C2= =A0

Python 2.6 is ancient, and is pretty slow in many as= pects (e.g. json parsing) when compared with Python 2.7. Some libraries tha= t Spark depend on stopped supporting 2.6. We can still convince the library= maintainers to support 2.6, but it will be extra work. I'm curious if = anybody still uses Python 2.6 to run Spark.

Thanks= .






--001a1146d9f07af4a205289dd3ab--