spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilan Filonenko <>
Subject Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Date Wed, 15 Aug 2018 19:56:35 GMT
Correct, the OS change and updates would require more testing, from what
Shane has told me, and could potentially surface some issue that could
delay a major release.

So yes, the release manager would need to run the tests manually and after
the release we would switch to a fully integrated Jenkins that would run
those same tests on the newly updated workers.

On Wed, Aug 15, 2018 at 3:47 PM, Reynold Xin <> wrote:

> Personally I'd love for R support to be in 2.4, but I don't consider
> something "Done" unless tests are running ... Is the proposal: the release
> manager manually run the R tests when preparing the release, and switch
> over to fully integrated Jenkins after 2.4.0 is released?
> On Wed, Aug 15, 2018 at 2:45 PM Reynold Xin <> wrote:
>> What's the reason we don't want to do the OS updates right now? Is it due
>> to the unpredictability of potential issues that might happen and end up
>> delaying 2.4 release?
>> On Wed, Aug 15, 2018 at 2:33 PM Erik Erlandson <>
>> wrote:
>>> The SparkR support PR is finished, along with integration testing,
>>> however Shane has requested that the integration testing not be enabled
>>> until after the 2.4 release because it requires the OS updates he wants to
>>> test *after* the release.
>>> The integration testing can be run locally, and so the question at hand
>>> is: would the PMC be willing to consider inclusion of the SparkR for 2.4,
>>> based on local verification of the testing? The PySpark PR was merged under
>>> similar circumstances: the testing was verified locally and the PR was
>>> merged before the testing was enabled for jenkins.
>>> Cheers,
>>> Erik

View raw message