hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <michael_se...@hotmail.com>
Subject Re: On coprocessor API evolution
Date Sat, 17 May 2014 12:16:01 GMT

Since you focused on my term ‘end user’ let me try this one more time because you clearly,
don’t grok the problem.

Running third party code as a coprocessor in the same jvm at the RS creates the risk that
said code could cause the RS to fail and cause a cascading failure. 

That’s an unacceptable amount of risk in an enterprise environment. 

The fix would be to ’sandbox’ the external code, and there are techniques to do just that.
However, it would mean a lot of work. 
But hey! Intel’s paying your salary to do just that. ;-)  

On May 17, 2014, at 11:29 AM, Andrew Purtell <andrew.purtell@gmail.com> wrote:

> The idea that coprocessors are meant for running untrusted end user code is a strawman
of your own construction. You show up with this whenever coprocessors comes up on the list
and lecture on our supposed incompetence and inability to see straight. It's not really that
helpful. I understand you don't like the idea of running end user code in the server. That's
great. Neither does just about anyone else, I would imagine. We didn't create coprocessors
so end users could run code in the RegionServers, we don't sell them as such either. So what
the hell are you talking about. 
>> On May 17, 2014, at 4:04 PM, Michael Segel <michael_segel@hotmail.com> wrote:
>> Andrew, 
>> Is ‘magical fairy dust’ a reference to some new synthetic drug you take at raves?

>> But lets get back to reality.
>> Lets try this again; simply put… the coprocessor runs on the same JVM as the RS,
therefore you have an unacceptable level of risk.
>> That inherent risk means that you cannot run HBase with end-user coprocessors enabled
when you want to have a stable and somewhat secure environment. 
>> The simple truth is that you need to decouple the end-user code (coprocessor) from
the RS. 
>> Its not a difficult concept to understand, and while reasonable, it would mean a
major rewrite and work done on co-processors.
>> Will de-coupling the user-space from the RS remove all risk? No.  And no, I’m not
suggesting that. 
>> But its a critical piece to the puzzle. 
>> Its not just security, but also reliability. 
>>> On May 17, 2014, at 4:43 AM, Andrew Purtell <apurtell@apache.org> wrote:
>>> Michael,
>>> As you know, we have implemented security features with coprocessors
>>> precisely because they can be interposed on internal actions to make
>>> authoritative decisions in-process. Coprocessors are a way to have
>>> composable internal extensions. They don't have and probably never will
>>> have magic fairy security dust. We do trust the security coprocessor code
>>> because it was developed by the project. That is not the same thing as
>>> saying you can have 'security' and execute arbitrary user code in-process
>>> as a coprocessor. Just want to clear that up for you.
>>>> will want to allow system coprocessors but then write a coprocessor that
>>> reject user coprocessors.
>>> That's a reasonable point.
>>> On Sat, May 17, 2014 at 12:13 AM, Michael Segel
>>> <michael_segel@hotmail.com>wrote:
>>>> Until you move the coprocessor out of the RS space and into its own
>>>> sandbox… saying security and coprocessor in the same sentence is a joke.
>>>> Oh wait… you were serious… :-(
>>>> I’d say there’s a significant rethink on coprocessors that’s required.
>>>> Anyone running a secure (kerberos) cluster, will want to allow system
>>>> coprocessors but then write a coprocessor that reject user coprocessors.
>>>> Just putting it out there…
>>>>> On May 15, 2014, at 2:13 AM, Andrew Purtell <apurtell@apache.org>
>>>>> Because coprocessor APIs are so tightly bound with internals, if we apply
>>>>> suggested rules like as mentioned on HBASE-11054:
>>>>>   I'd say policy should be no changes to method apis across minor
>>>>> versions
>>>>> This will lock coprocessor based components to the limitations of the
>>>>> as we encounter them. Core code does not suffer this limitation, we are
>>>>> otherwise free to refactor and change internal methods. For example,
>>>> we
>>>>> apply this policy to the 0.98 branch, then we will have to abandon
>>>> further
>>>>> security feature development there and move to trunk only. This is
>>>> because
>>>>> we already are aware that coprocessor APIs as they stand are insufficient
>>>>> still.
>>>>> Coprocessor APIs are a special class of internal method. We have had
>>>>> tension between allowing freedom of movement for developing them out
>>>>> providing some measure of stability for implementors for a while.
>>>>> It is my belief that the way forward is something like HBASE-11125.
>>>> Perhaps
>>>>> we can take this discussion to that JIRA and have this long overdue
>>>>> conversation.
>>>>> Regarding security features specifically, I would also like to call your
>>>>> attention to HBASE-11127. I think security has been an optional feature
>>>>> long enough, it is becoming a core requirement for the project, so should
>>>>> be moved into core. Sure, we can therefore sidestep any issues with
>>>>> coprocessor API sufficiency for hosting security features. However, in
>>>>> opinion we should pursue both HBASE-11125 and HBASE-11127; the first
>>>>> provide the relative stability long asked for by coprocessor API users,
>>>> the
>>>>> latter to cleanly solve emerging issues with concurrency and versioning.
>>>>> --
>>>>> Best regards,
>>>>> - Andy
>>>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>>>> (via Tom White)
>>> -- 
>>> Best regards,
>>> - Andy
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)

View raw message