Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of andrew.purtell@gmail.com
 designates 209.85.220.50 as permitted sender)
Subject: Re: On coprocessor API evolution
References: 
 <CA+RK=_C+m6CDfEye39LzjeBR-Hij+ymHCt-wqwU2Cn_btZtaKQ@mail.gmail.com>
 <BLU436-SMTP234571B80BA5CCF6991504B8F310@phx.gbl>
 <CA+RK=_Cdb2LFANj-+0qNE1iHLagfnRnfYA7zyfVsWAQG7AUkFg@mail.gmail.com>
 <BLU436-SMTP1138A3BC7BCA780D4E878A58F300@phx.gbl>
From: Andrew Purtell <andrew.purtell@gmail.com>
Content-Type: text/plain;
	charset=utf-8
In-Reply-To: <BLU436-SMTP1138A3BC7BCA780D4E878A58F300@phx.gbl>
Message-Id: <BF8C6F73-980A-43BA-B0BB-1DABFFC29737@gmail.com>
Date: Sat, 17 May 2014 18:29:28 +0800
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)

The idea that coprocessors are meant for running untrusted end user code is a=
 strawman of your own construction. You show up with this whenever coprocess=
ors comes up on the list and lecture on our supposed incompetence and inabil=
ity to see straight. It's not really that helpful. I understand you don't li=
ke the idea of running end user code in the server. That's great. Neither do=
es just about anyone else, I would imagine. We didn't create coprocessors so=
 end users could run code in the RegionServers, we don't sell them as such e=
ither. So what the hell are you talking about.=20


> On May 17, 2014, at 4:04 PM, Michael Segel <michael_segel@hotmail.com> wro=
te:
>=20
> Andrew,=20
>=20
> Is =E2=80=98magical fairy dust=E2=80=99 a reference to some new synthetic d=
rug you take at raves?=20
> But lets get back to reality.
>=20
>=20
> Lets try this again; simply put=E2=80=A6 the coprocessor runs on the same J=
VM as the RS, therefore you have an unacceptable level of risk.
> That inherent risk means that you cannot run HBase with end-user coprocess=
ors enabled when you want to have a stable and somewhat secure environment.=20=

>=20
> The simple truth is that you need to decouple the end-user code (coprocess=
or) from the RS.=20
> Its not a difficult concept to understand, and while reasonable, it would m=
ean a major rewrite and work done on co-processors.
>=20
> Will de-coupling the user-space from the RS remove all risk? No.  And no, I=
=E2=80=99m not suggesting that.=20
> But its a critical piece to the puzzle.=20
>=20
> Its not just security, but also reliability.=20
>=20
>=20
>> On May 17, 2014, at 4:43 AM, Andrew Purtell <apurtell@apache.org> wrote:
>>=20
>> Michael,
>>=20
>> As you know, we have implemented security features with coprocessors
>> precisely because they can be interposed on internal actions to make
>> authoritative decisions in-process. Coprocessors are a way to have
>> composable internal extensions. They don't have and probably never will
>> have magic fairy security dust. We do trust the security coprocessor code=

>> because it was developed by the project. That is not the same thing as
>> saying you can have 'security' and execute arbitrary user code in-process=

>> as a coprocessor. Just want to clear that up for you.
>>=20
>>> will want to allow system coprocessors but then write a coprocessor that=

>> reject user coprocessors.
>>=20
>> That's a reasonable point.
>>=20
>>=20
>>=20
>>=20
>> On Sat, May 17, 2014 at 12:13 AM, Michael Segel
>> <michael_segel@hotmail.com>wrote:
>>=20
>>> Until you move the coprocessor out of the RS space and into its own
>>> sandbox=E2=80=A6 saying security and coprocessor in the same sentence is=
 a joke.
>>> Oh wait=E2=80=A6 you were serious=E2=80=A6 :-(
>>>=20
>>> I=E2=80=99d say there=E2=80=99s a significant rethink on coprocessors th=
at=E2=80=99s required.
>>>=20
>>> Anyone running a secure (kerberos) cluster, will want to allow system
>>> coprocessors but then write a coprocessor that reject user coprocessors.=

>>>=20
>>> Just putting it out there=E2=80=A6
>>>=20
>>>> On May 15, 2014, at 2:13 AM, Andrew Purtell <apurtell@apache.org> wrote=
:
>>>>=20
>>>> Because coprocessor APIs are so tightly bound with internals, if we app=
ly
>>>> suggested rules like as mentioned on HBASE-11054:
>>>>=20
>>>>    I'd say policy should be no changes to method apis across minor
>>>> versions
>>>>=20
>>>> This will lock coprocessor based components to the limitations of the A=
PI
>>>> as we encounter them. Core code does not suffer this limitation, we are=

>>>> otherwise free to refactor and change internal methods. For example, if=

>>> we
>>>> apply this policy to the 0.98 branch, then we will have to abandon
>>> further
>>>> security feature development there and move to trunk only. This is
>>> because
>>>> we already are aware that coprocessor APIs as they stand are insufficie=
nt
>>>> still.
>>>>=20
>>>> Coprocessor APIs are a special class of internal method. We have had a
>>>> tension between allowing freedom of movement for developing them out an=
d
>>>> providing some measure of stability for implementors for a while.
>>>>=20
>>>> It is my belief that the way forward is something like HBASE-11125.
>>> Perhaps
>>>> we can take this discussion to that JIRA and have this long overdue
>>>> conversation.
>>>>=20
>>>> Regarding security features specifically, I would also like to call you=
r
>>>> attention to HBASE-11127. I think security has been an optional feature=

>>>> long enough, it is becoming a core requirement for the project, so shou=
ld
>>>> be moved into core. Sure, we can therefore sidestep any issues with
>>>> coprocessor API sufficiency for hosting security features. However, in m=
y
>>>> opinion we should pursue both HBASE-11125 and HBASE-11127; the first to=

>>>> provide the relative stability long asked for by coprocessor API users,=

>>> the
>>>> latter to cleanly solve emerging issues with concurrency and versioning=
.
>>>>=20
>>>>=20
>>>> --
>>>> Best regards,
>>>>=20
>>>> - Andy
>>>>=20
>>>> Problems worthy of attack prove their worth by hitting back. - Piet Hei=
n
>>>> (via Tom White)
>>=20
>>=20
>> --=20
>> Best regards,
>>=20
>>  - Andy
>>=20
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>=20