spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <patr...@databricks.com>
Subject Re: Signing releases with pwendell or release manager's key?
Date Tue, 19 Sep 2017 01:23:14 GMT
One thing we could do is modify the release tooling to allow the key to be
injected each time, thus allowing any RM to insert their own key at build
time.

Patrick

On Mon, Sep 18, 2017 at 4:56 PM Ryan Blue <rblue@netflix.com> wrote:

> I don't understand why it is necessary to share a release key. If this is
> something that can be automated in a Jenkins job, then can it be a script
> with a reasonable set of build requirements for Mac and Ubuntu? That's the
> approach I've seen the most in other projects.
>
> I'm also not just concerned about release managers. Having a key stored
> persistently on outside infrastructure adds the most risk, as Luciano noted
> as well. We should also start publishing checksums in the Spark VOTE
> thread, which are currently missing. The risk I'm concerned about is that
> if the key were compromised, it would be possible to replace binaries with
> perfectly valid ones, at least on some mirrors. If the Apache copy were
> replaced, then we wouldn't even be able to catch that it had happened.
> Given the high profile of Spark and the number of companies that run it, I
> think we need to take extra care to make sure that can't happen, even if it
> is an annoyance for the release managers.
>
> On Sun, Sep 17, 2017 at 10:12 PM, Patrick Wendell <patrick@databricks.com>
> wrote:
>
>> Sparks release pipeline is automated and part of that automation includes
>> securely injecting this key for the purpose of signing. I asked the ASF to
>> provide a service account key several years ago but they suggested that we
>> use a key attributed to an individual even if the process is automated.
>>
>> I believe other projects that release with high frequency also have
>> automated the signing process.
>>
>> This key is injected during the build process. A really ambitious release
>> manager could reverse engineer this in a way that reveals the private key,
>> however if someone is a release manager then they themselves can do quite a
>> bit of nefarious things anyways.
>>
>> It is true that we trust all previous release managers instead of only
>> one. We could probably rotate the jenkins credentials periodically in order
>> to compensate for this, if we think this is a nontrivial risk.
>>
>> - Patrick
>>
>> On Sun, Sep 17, 2017 at 7:04 PM, Holden Karau <holden@pigscanfly.ca>
>> wrote:
>>
>>> Would any of Patrick/Josh/Shane (or other PMC folks with
>>> understanding/opinions on this setup) care to comment? If this is a
>>> blocking issue I can cancel the current release vote thread while we
>>> discuss this some more.
>>>
>>> On Fri, Sep 15, 2017 at 5:18 PM Holden Karau <holden@pigscanfly.ca>
>>> wrote:
>>>
>>>> Oh yes and to keep people more informed I've been updating a PR for the
>>>> release documentation as I go to write down some of this unwritten
>>>> knowledge -- https://github.com/apache/spark-website/pull/66
>>>>
>>>>
>>>> On Fri, Sep 15, 2017 at 5:12 PM Holden Karau <holden@pigscanfly.ca>
>>>> wrote:
>>>>
>>>>> Also continuing the discussion from the vote threads, Shane probably
>>>>> has the best idea on the ACLs for Jenkins so I've CC'd him as well.
>>>>>
>>>>>
>>>>> On Fri, Sep 15, 2017 at 5:09 PM Holden Karau <holden@pigscanfly.ca>
>>>>> wrote:
>>>>>
>>>>>> Changing the release jobs, beyond the available parameters, right
now
>>>>>> depends on Josh arisen as there are some scripts which generate the
jobs
>>>>>> which aren't public. I've done temporary fixes in the past with the
Python
>>>>>> packaging but my understanding is that in the medium term it requires
>>>>>> access to the scripts.
>>>>>>
>>>>>> So +CC Josh.
>>>>>>
>>>>>> On Fri, Sep 15, 2017 at 4:38 PM Ryan Blue <rblue@netflix.com>
wrote:
>>>>>>
>>>>>>> I think this needs to be fixed. It's true that there are barriers
to
>>>>>>> publication, but the signature is what we use to authenticate
Apache
>>>>>>> releases.
>>>>>>>
>>>>>>> If Patrick's key is available on Jenkins for any Spark committer
to
>>>>>>> use, then the chance of a compromise are much higher than for
a normal RM
>>>>>>> key.
>>>>>>>
>>>>>>> rb
>>>>>>>
>>>>>>> On Fri, Sep 15, 2017 at 12:34 PM, Sean Owen <sowen@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yeah I had meant to ask about that in the past. While I presume
>>>>>>>> Patrick consents to this and all that, it does mean that
anyone with access
>>>>>>>> to said Jenkins scripts can create a signed Spark release,
regardless of
>>>>>>>> who they are.
>>>>>>>>
>>>>>>>> I haven't thought through whether that's a theoretical issue
we can
>>>>>>>> ignore or something we need to fix up. For example you can't
get a release
>>>>>>>> on the ASF mirrors without more authentication.
>>>>>>>>
>>>>>>>> How hard would it be to make the script take in a key? it
sort of
>>>>>>>> looks like the script already takes GPG_KEY, but don't know
how to modify
>>>>>>>> the jobs. I suppose it would be ideal, in any event, for
the actual release
>>>>>>>> manager to sign.
>>>>>>>>
>>>>>>>> On Fri, Sep 15, 2017 at 8:28 PM Holden Karau <holden@pigscanfly.ca>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> That's a good question, I built the release candidate
however the
>>>>>>>>> Jenkins scripts don't take a parameter for configuring
who signs them
>>>>>>>>> rather it always signs them with Patrick's key. You can
see this from
>>>>>>>>> previous releases which were managed by other folks but
still signed by
>>>>>>>>> Patrick.
>>>>>>>>>
>>>>>>>>> On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <rblue@netflix.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> The signature is valid, but why was the release signed
with
>>>>>>>>>> Patrick Wendell's private key? Did Patrick build
the release candidate?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>> --
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>
>>>>> --
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Mime
View raw message