hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: On coprocessor API evolution
Date Mon, 19 May 2014 11:44:49 GMT
I would say the serious problem here is you can clearly see that we
understand the issues, as you point out, and then fail to even for a second
entertain a point of view that is not 100% yours. This can't go on. Feel
free to continue mailing the list but I won't be receiving email from you
further.


On Mon, May 19, 2014 at 4:18 AM, Michael Segel <michael_segel@hotmail.com>wrote:

> Andrew,
>
> I suggest that you go back and re-read Kevin O’Dell’s comment.
> Clearly there’s no straw man here.
>
> I thought your response was a joke, but apparently you’re serious about
> that.
>
> Again, when you allow user code to run in the same JVM as the RS, you add
> risk. Issues with stability and security need to be addressed.
> You post an earlier Jira where even you see this as an issue.  (That’s the
> biggest irony.)
>
> If you and the other committers can’t recognize this as an issue… then
> there is a more serious problem that needs to be addressed.
>
>
>
> On May 19, 2014, at 11:24 AM, Andrew Purtell <andrew.purtell@gmail.com>
> wrote:
>
> > Yes it's clear you have been laughing at many of the responses and have
> been having a merry time concern trolling your favorite straw man. You
> insist on fixating on analogies we have used to explain coprocessors to
> advanced users and spinning that into an absolutist (and ridiculous)
> position that doesn't match up with what everyone else is telling you. At
> some point you need to stop talking and start listening. The insults are
> not helping your case. Or maybe that's the point. Yes Michael you are the
> RDBMS king and we are all idiots. Happy? Now consider this: That means
> nothing as HBase isn't even in that space. Maybe you should try this over
> on the MySQL or Postgres mailing lists.
> >
> >
> >> On May 19, 2014, at 12:06 AM, Michael Segel <michael_segel@hotmail.com>
> wrote:
> >>
> >> Remind me how you do server side execution?
> >>
> >> I suggest you go through the emails on both the dev and user hbase list
> and you’ll see that when describing coprocessors, you’ll see the terms
> ‘trigger’ and ‘stored procedures’ which are terms most DBAs and Data
> modelers are familiar with.
> >>
> >> And if you read my email earlier in the thread, I said that the analogy
> fell a little short because of the extensibility aspect.
> >>
> >> I have to ask, how many of the committers and readers on this thread
> have actual RDBMs experience that goes beyond mySQL, and maybe Postgres.  I
> mentioned Sybase’s Adaptive Server as well as Informix’s IDS.  Anyone know
> what I’m talking about? How about Oracle? DB2?  I’ll even add Dick Pick’s
> Revelation or U2 (Universe) which are hierarchal systems.
> >>
> >> The fact that you said ‘it ain’t broken’ means that you just don’t
> understand the problem. Even after Kevin O’Dell admitted that his
> (Cloudera’s) customers who are using coprocessors are complaining of HBase
> not behaving in a stable fashion. It may be their code and lack of
> understanding, but Cloudera still has to address it and you can’t say using
> coprocessors voids the warranty. (Sorry Andrew, I had to laugh at that one…)
> >>
> >> I understand completely what coprocessors were designed for.  What you
> don’t seem to understand is that the design and implementation which falls
> short and open it up to potential reliability issues.
> >>
> >> To be clear… I haven’t even talked about APIs, I’m talking about the
> core design.
> >>
> >> To the best of my knowledge,  MapR’s M7 doesn’t have coprocessors. I’ll
> wager that when they do, it will work and not have these issues. I believe
> that they are writing their stuff in C/C++, if so, then they’d have an
> advantage of using shared memory.  Apache would have write C/C++ code and
> wrap it in JNI… which you may not want to do…
> >>
> >> But getting back to the issue at hand… if HBase is viewed as hard to
> tune and hard to keep up… people are going to look towards other solutions.
> >>
> >> I believe you can already disable the ability to create enduser
> coprocessors. When you load the last system coprocessor, you load one that
> looks to see who’s trying to add a coprocessor and you just deny it. If you
> wanted to make a formal change, then you would just have a database
> permission that you either GRANT or REVOKE the ability of a user the
> privilege  to add coprocessors. But that would mean more work for someone.
> >>
> >> -Mike
> >>
> >>> On May 18, 2014, at 9:58 PM, lars hofhansl <larsh@apache.org> wrote:
> >>>
> >>> Coprocessors are a means to extend HBase. Nothing more, nothing less.
> They are not stored procedures or triggers.
> >>> Not sure in how many other ways we can/need to phrase that.
> >>>
> >>>
> >>> I agree that there should be a simple way to disable user coprocessors
> (or at least disable loading from HDFS) for the security conscious. Let's
> do that, it's simple.
> >>>
> >>> There is nothing to "fix" since it ain't broken. It's only seems
> broken when you do not understand what it was designed for.
> >>>
> >>> You want a new API for less invasive things in a sandbox, more like
> stored procedures and triggers... Sure, let's do that too. But realize that
> is a *new* use case, and that we'll keep the old stuff.
> >>>
> >>> -- Lars
> >>>
> >>> ________________________________
> >>> From: Michael Segel <michael_segel@hotmail.com>
> >>> To: dev@hbase.apache.org; lars hofhansl <larsh@apache.org>
> >>> Sent: Sunday, May 18, 2014 10:21 AM
> >>> Subject: Re: On coprocessor API evolution
> >>>
> >>>
> >>> It doesn’t matter.
> >>>
> >>> Sure we can follow Vlad’s rules… but you still have to get to the root
> of the problem and that is making coprocessors safe.
> >>>
> >>> Its not an easy fix, and it would mean pretty much starting from
> scratch. Trying to kludge a fix is harder and will not be as good.
> >>>
> >>> Maybe you can salvage some code, but the issue is fixing coprocessors
> at the lowest level and work back up.
> >>>
> >>> You have to isolate the code to one or more separate jvms so you can
> not only stop, but reload the processes.
> >>> This is more than just simple triggers but also extensibility.
> >>>
> >>> If you could pick the brains of some of the folks still under Kevin
> Foster (@IBM) who work on IDS… you could get some ideas.
> >>>
> >>>
> >>>
> >>>> On May 18, 2014, at 7:01 AM, lars hofhansl <larsh@apache.org>
wrote:
> >>>>
> >>>> We've seen similar issues with Filters. Those are good rules to
> follow.
> >>>>
> >>>>
> >>>>
> >>>> ________________________________
> >>>> From: Vladimir Rodionov <vladrodionov@gmail.com>
> >>>> To: "dev@hbase.apache.org" <dev@hbase.apache.org>
> >>>> Sent: Friday, May 16, 2014 10:59 AM
> >>>> Subject: Re: On coprocessor API evolution
> >>>>
> >>>>
> >>>> 1) Have default implementations (abstract classes) for every
> interface from
> >>>> Coprocessor API.
> >>>> 2) Advise coprocessor users not to implement interface directly but
> sub
> >>>> class default impl.
> >>>> 3) Preserve backward compatibility by adding only new hooks/methods
> >>>> 4) DO NOT CHANGE existing API (no method renaming, method parameter
> type
> >>>> changes etc)
> >>>> 5) Have a regression tests to check backward compatibility.
> >>>>
> >>>> -Vladimir
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Fri, May 16, 2014 at 9:13 AM, Michael Segel <
> michael_segel@hotmail.com>wrote:
> >>>>
> >>>>> Until you move the coprocessor out of the RS space and into its
own
> >>>>> sandbox… saying security and coprocessor in the same sentence
is a
> joke.
> >>>>> Oh wait… you were serious… :-(
> >>>>>
> >>>>> I’d say there’s a significant rethink on coprocessors that’s
> required.
> >>>>>
> >>>>> Anyone running a secure (kerberos) cluster, will want to allow system
> >>>>> coprocessors but then write a coprocessor that reject user
> coprocessors.
> >>>>>
> >>>>> Just putting it out there…
> >>>>>
> >>>>>> On May 15, 2014, at 2:13 AM, Andrew Purtell <apurtell@apache.org>
> wrote:
> >>>>>>
> >>>>>> Because coprocessor APIs are so tightly bound with internals,
if we
> apply
> >>>>>> suggested rules like as mentioned on HBASE-11054:
> >>>>>>
> >>>>>>      I'd say policy should be no changes to method apis across
minor
> >>>>>> versions
> >>>>>>
> >>>>>> This will lock coprocessor based components to the limitations
of
> the API
> >>>>>> as we encounter them. Core code does not suffer this limitation,
we
> are
> >>>>>> otherwise free to refactor and change internal methods. For
> example, if
> >>>>> we
> >>>>>> apply this policy to the 0.98 branch, then we will have to abandon
> >>>>> further
> >>>>>> security feature development there and move to trunk only. This
is
> >>>>> because
> >>>>>> we already are aware that coprocessor APIs as they stand are
> insufficient
> >>>>>> still.
> >>>>>>
> >>>>>> Coprocessor APIs are a special class of internal method. We
have
> had a
> >>>>>> tension between allowing freedom of movement for developing
them
> out and
> >>>>>> providing some measure of stability for implementors for a while.
> >>>>>>
> >>>>>> It is my belief that the way forward is something like HBASE-11125.
> >>>>> Perhaps
> >>>>>> we can take this discussion to that JIRA and have this long
overdue
> >>>>>> conversation.
> >>>>>>
> >>>>>> Regarding security features specifically, I would also like
to call
> your
> >>>>>> attention to HBASE-11127. I think security has been an optional
> feature
> >>>>>> long enough, it is becoming a core requirement for the project,
so
> should
> >>>>>> be moved into core. Sure, we can therefore sidestep any issues
with
> >>>>>> coprocessor API sufficiency for hosting security features. However,
> in my
> >>>>>> opinion we should pursue both HBASE-11125 and HBASE-11127; the
> first to
> >>>>>> provide the relative stability long asked for by coprocessor
API
> users,
> >>>>> the
> >>>>>> latter to cleanly solve emerging issues with concurrency and
> versioning.
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best regards,
> >>>>>>
> >>>>>>   - Andy
> >>>>>>
> >>>>>> Problems worthy of attack prove their worth by hitting back.
- Piet
> Hein
> >>>>>> (via Tom White)
> >>
> >
>
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message