Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
From: Felix Cheung <felixcheung_m@hotmail.com>
To: Reynold Xin <rxin@databricks.com>
CC: "dev@spark.apache.org" <dev@spark.apache.org>
Subject: Re: [discuss] Spark 2.x release cadence
Thread-Topic: [discuss] Spark 2.x release cadence
Thread-Index: AQHSGPJcLhWnuUDAnkSsl38rwFI09aCN0rUAgABzax0=
Date: Wed, 28 Sep 2016 03:54:29 +0000
Message-ID: <BLUPR04MB772C48D8D62D386B00BEC7991CF0@BLUPR04MB772.namprd04.prod.outlook.com>
References: <CAPh_B=Z2ZDgUCbGkyVup3vDHJg921L3UpSN5bqiYsP5XO6f9gw@mail.gmail.com>,<CAAsvFPnMQ320b86BYayjJyBDx7g_37YfBcK8pUjgEoneDvOYQg@mail.gmail.com>
In-Reply-To: <CAAsvFPnMQ320b86BYayjJyBDx7g_37YfBcK8pUjgEoneDvOYQg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: SoftFail (protection.outlook.com: domain of transitioning
 hotmail.com discourages use of 10.152.66.51 as permitted sender)
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [Z9tqGj+KO/z5XbVvkCaxahLxgyMEngp1]
x-eopattributedmessage: 0
x-microsoft-exchange-diagnostics: 1;BN3NAM01HT188;6:wvHZYRNmzyN80YJD6pZuZRL8MIq2MpSzYso8ZEO1i3kcjf9et/6OKcqcBSj33fn/pwmJq1oWERlLkbbCSEHaEoaez2WVuOo2VgDwK75N5Boew66iZmF9zCWAQzFotQ43xvgUSZYJgg4XdOrYjolLW7RQhq+oBE4478ea0nacpHNYqRVAHpa2EOxro56QTvZsK/SftoutW/ZQmdusMg2iKYYadE/PgtZHZCgEb1PRkhe4pcnxcqg/oXhTfbhcPwtCQiGV1Ik95TBml454Zs8WLLnKgBat/RXTpTL9Wz6YfuY=;5:oLXaCLIdIOGZdiMXzWa2S6/IZGVr5zZCoiLWKQGofXZWuqGdPT1BBhtqKrIGUr26QYU3IVNGhOn3tdHHfiVQSnSePrxUSJ2FuNKJCup64j1Z5ppnvaJCxEdrq9kjvSOVNjXlXFXXNUWi4ug4m3oa4g==;24:TmajWfCqOv3YBF5pvUV0hvulZOdzVTwBZXH2hK0EblOHoggg1JQ8yExt1E3HnOZugF4j7KtNdA8MyB6bZd8ikIUPQtNcOsOw1Z5aHKQxPpw=;7:qbRnfeP74f6tmSbvUQTtwYRiGMn5YvJDSYBHhN30R0XUfWyPM8A80LF5gwgrcDgJbLB2YiYT1gHWKOCwFD4XKeuVV/ooD4pclUsbEqMR5MH9tZGEO3cbVLi4gVLb/e51NpWq5DaqGPM6ehbB10bySK0A09IqEB+D431Hrh7rOg+fNMD1LL2scdatLnYxB4nswHHrvx2A1benR4hiWJraPdRxBDeiINxa/x/woA51fPbVZOZ3B88PdUbqq6s8iwnrrKsbscMuM70aEVPNjpYdC6G6o5fWtFiiSzUVnCywFolDgbbbBgfEB4we9J0xSQiT
x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(10019020)(98900003);DIR:OUT;SFP:1102;SCL:1;SRVR:BN3NAM01HT188;H:BLUPR04MB772.namprd04.prod.outlook.com;FPR:;SPF:None;LANG:en;
x-ms-office365-filtering-correlation-id: 09613da2-94c9-454f-669e-08d3e7532612
x-microsoft-antispam: UriScan:;BCL:1;PCL:0;RULEID:(1601124038)(1603103081)(1601125047);SRVR:BN3NAM01HT188;
x-exchange-antispam-report-cfa-test: BCL:1;PCL:0;RULEID:(432015012)(82015046);SRVR:BN3NAM01HT188;BCL:1;PCL:0;RULEID:;SRVR:BN3NAM01HT188;
x-forefront-prvs: 0079056367
spamdiagnosticoutput: 1:5
spamdiagnosticmetadata: :1
Content-Type: multipart/alternative;
	boundary="_000_BLUPR04MB772C48D8D62D386B00BEC7991CF0BLUPR04MB772namprd_"
MIME-Version: 1.0
X-OriginatorOrg: hotmail.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Sep 2016 03:54:29.5476
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3NAM01HT188
X-OriginalArrivalTime: 28 Sep 2016 03:54:31.0005 (UTC) FILETIME=[046B78D0:01D2193C]
archived-at: Wed, 28 Sep 2016 03:54:46 -0000

--_000_BLUPR04MB772C48D8D62D386B00BEC7991CF0BLUPR04MB772namprd_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

+1 on longer release cycle at schedule and more maintenance releases.


_____________________________
From: Mark Hamstra <mark@clearstorydata.com<mailto:mark@clearstorydata.com>=
>
Sent: Tuesday, September 27, 2016 2:01 PM
Subject: Re: [discuss] Spark 2.x release cadence
To: Reynold Xin <rxin@databricks.com<mailto:rxin@databricks.com>>
Cc: <dev@spark.apache.org<mailto:dev@spark.apache.org>>


+1

And I'll dare say that for those with Spark in production, what is more imp=
ortant is that maintenance releases come out in a timely fashion than that =
new features are released one month sooner or later.

On Tue, Sep 27, 2016 at 12:06 PM, Reynold Xin <rxin@databricks.com<mailto:r=
xin@databricks.com>> wrote:
We are 2 months past releasing Spark 2.0.0, an important milestone for the =
project. Spark 2.0.0 deviated (took 6 month from the regular release cadenc=
e we had for the 1.x line, and we never explicitly discussed what the relea=
se cadence should look like for 2.x. Thus this email.

During Spark 1.x, roughly every three months we make a new 1.x feature rele=
ase (e.g. 1.5.0 comes out three months after 1.4.0). Development happened p=
rimarily in the first two months, and then a release branch was cut at the =
end of month 2, and the last month was reserved for QA and release preparat=
ion.

During 2.0.0 development, I really enjoyed the longer release cycle because=
 there was a lot of major changes happening and the longer time was critica=
l for thinking through architectural changes as well as API design. While I=
 don't expect the same degree of drastic changes in a 2.x feature release, =
I do think it'd make sense to increase the length of release cycle so we ca=
n make better designs.

My strawman proposal is to maintain a regular release cadence, as we did in=
 Spark 1.x, and increase the cycle from 3 months to 4 months. This effectiv=
ely gives us ~50% more time to develop (in reality it'd be slightly less th=
an 50% since longer dev time also means longer QA time). As for maintenance=
 releases, I think those should still be cut on-demand, similar to Spark 1.=
x, but more aggressively.

To put this into perspective, 4-month cycle means we will release Spark 2.1=
.0 at the end of Nov or early Dec (and branch cut / code freeze at the end =
of Oct).

I am curious what others think.


--_000_BLUPR04MB772C48D8D62D386B00BEC7991CF0BLUPR04MB772namprd_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
</head>
<body>
<div id=3D"compose" contenteditable=3D"true" style=3D"padding-left: 20px; p=
adding-right: 20px; padding-bottom: 8px;">
<div>&#43;1 on longer release cycle at schedule and more maintenance releas=
es.</div>
<div><br>
<div class=3D"acompli_signature"></div>
<br>
</div>
</div>
<div class=3D"gmail_quote">_____________________________<br>
From: Mark Hamstra &lt;<a dir=3D"ltr" href=3D"mailto:mark@clearstorydata.co=
m" x-apple-data-detectors=3D"true" x-apple-data-detectors-type=3D"link" x-a=
pple-data-detectors-result=3D"0">mark@clearstorydata.com</a>&gt;<br>
Sent: Tuesday, September 27, 2016 2:01 PM<br>
Subject: Re: [discuss] Spark 2.x release cadence<br>
To: Reynold Xin &lt;<a dir=3D"ltr" href=3D"mailto:rxin@databricks.com" x-ap=
ple-data-detectors=3D"true" x-apple-data-detectors-type=3D"link" x-apple-da=
ta-detectors-result=3D"1">rxin@databricks.com</a>&gt;<br>
Cc: &lt;<a dir=3D"ltr" href=3D"mailto:dev@spark.apache.org" x-apple-data-de=
tectors=3D"true" x-apple-data-detectors-type=3D"link" x-apple-data-detector=
s-result=3D"2">dev@spark.apache.org</a>&gt;<br>
<br>
<br>
<meta content=3D"text/html; charset=3Dutf-8">
<div dir=3D"ltr">&#43;1
<div><br>
</div>
<div>And I'll dare say that for those with Spark in production, what is mor=
e important is that maintenance releases come out in a timely fashion than =
that new features are released one month sooner or later.</div>
</div>
<div class=3D"gmail_extra"><br>
<div class=3D"gmail_quote">On Tue, Sep 27, 2016 at 12:06 PM, Reynold Xin <s=
pan dir=3D"ltr">
&lt;<a href=3D"mailto:rxin@databricks.com">rxin@databricks.com</a>&gt;</spa=
n> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
<div dir=3D"ltr">We are 2 months past releasing Spark 2.0.0, an important m=
ilestone for the project. Spark 2.0.0 deviated (took 6 month from the regul=
ar release cadence we had for the 1.x line, and we never explicitly discuss=
ed what the release cadence should
 look like for 2.x. Thus this email.
<div>
<div><br>
</div>
</div>
<div>During Spark 1.x, roughly every three months we make a new 1.x feature=
 release (e.g. 1.5.0 comes out three months after 1.4.0). Development happe=
ned primarily in the first two months, and then a release branch was cut at=
 the end of month 2, and the last
 month was reserved for QA and release preparation.</div>
<div><br>
</div>
<div>During 2.0.0 development, I really enjoyed the longer release cycle be=
cause there was a lot of major changes happening and the longer time was cr=
itical for thinking through architectural changes as well as API design. Wh=
ile I don't expect the same degree
 of drastic changes in a 2.x feature release, I do think it'd make sense to=
 increase the length of release cycle so we can make better designs.</div>
<div><br>
</div>
<div>My strawman proposal is to maintain a regular release cadence, as we d=
id in Spark 1.x, and increase the cycle from 3 months to 4 months. This eff=
ectively gives us ~50% more time to develop (in reality it'd be slightly le=
ss than 50% since longer dev time
 also means longer QA time). As for maintenance releases, I think those sho=
uld still be cut on-demand, similar to Spark 1.x, but more aggressively.</d=
iv>
<div><br>
</div>
<div>To put this into perspective, 4-month cycle means we will release Spar=
k 2.1.0 at the end of Nov or early Dec (and branch cut / code freeze at the=
 end of Oct).</div>
<div><br>
</div>
<div>I am curious what others think.</div>
<div><br>
</div>
<div><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<br>
</div>
</body>
</html>

--_000_BLUPR04MB772C48D8D62D386B00BEC7991CF0BLUPR04MB772namprd_--