hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: On coprocessor API evolution
Date Mon, 19 May 2014 19:05:11 GMT
Michael S.
>> To the best of my knowledge,  MapR’s M7 doesn’t have coprocessors. I’ll
wager that when they do, it will work and not have these issues. I believe
that they are writing their stuff in C/C++, if so, then they’d have an
advantage of using shared memory.  Apache would have write C/C++ code and
wrap it in JNI… which you may not want to do…<<

MapR M7 does not support coprocessors and custom Filters as well. I
consider this to be a serious limitation of the product.
Shared memory communication can be done in Java w/o single line of C/C++
code, Michael by means of memory-mapped files.

-Vladimir




On Mon, May 19, 2014 at 11:52 AM, Michael Segel
<michael_segel@hotmail.com>wrote:

> Several of my friends have already walked away from using HBase in their
> solutions.
>
> Looks like I will have to do the same.
> I’ll limit my use of Hbase as a choice of last resort and even then limit
> how its to be used.
>
> But to be honest, Andrew… your quote… “Just tell them that it voids the
> warranty” made my day. Told it to a friend and we got a good laugh out of
> it.
>
> I’m going to have to capture that…
>
>
> On May 19, 2014, at 12:44 PM, Andrew Purtell <apurtell@apache.org> wrote:
>
> > I would say the serious problem here is you can clearly see that we
> > understand the issues, as you point out, and then fail to even for a
> second
> > entertain a point of view that is not 100% yours. This can't go on. Feel
> > free to continue mailing the list but I won't be receiving email from you
> > further.
> >
> >
> > On Mon, May 19, 2014 at 4:18 AM, Michael Segel <
> michael_segel@hotmail.com>wrote:
> >
> >> Andrew,
> >>
> >> I suggest that you go back and re-read Kevin O’Dell’s comment.
> >> Clearly there’s no straw man here.
> >>
> >> I thought your response was a joke, but apparently you’re serious about
> >> that.
> >>
> >> Again, when you allow user code to run in the same JVM as the RS, you
> add
> >> risk. Issues with stability and security need to be addressed.
> >> You post an earlier Jira where even you see this as an issue.  (That’s
> the
> >> biggest irony.)
> >>
> >> If you and the other committers can’t recognize this as an issue… then
> >> there is a more serious problem that needs to be addressed.
> >>
> >>
> >>
> >> On May 19, 2014, at 11:24 AM, Andrew Purtell <andrew.purtell@gmail.com>
> >> wrote:
> >>
> >>> Yes it's clear you have been laughing at many of the responses and have
> >> been having a merry time concern trolling your favorite straw man. You
> >> insist on fixating on analogies we have used to explain coprocessors to
> >> advanced users and spinning that into an absolutist (and ridiculous)
> >> position that doesn't match up with what everyone else is telling you.
> At
> >> some point you need to stop talking and start listening. The insults are
> >> not helping your case. Or maybe that's the point. Yes Michael you are
> the
> >> RDBMS king and we are all idiots. Happy? Now consider this: That means
> >> nothing as HBase isn't even in that space. Maybe you should try this
> over
> >> on the MySQL or Postgres mailing lists.
> >>>
> >>>
> >>>> On May 19, 2014, at 12:06 AM, Michael Segel <
> michael_segel@hotmail.com>
> >> wrote:
> >>>>
> >>>> Remind me how you do server side execution?
> >>>>
> >>>> I suggest you go through the emails on both the dev and user hbase
> list
> >> and you’ll see that when describing coprocessors, you’ll see the terms
> >> ‘trigger’ and ‘stored procedures’ which are terms most DBAs and Data
> >> modelers are familiar with.
> >>>>
> >>>> And if you read my email earlier in the thread, I said that the
> analogy
> >> fell a little short because of the extensibility aspect.
> >>>>
> >>>> I have to ask, how many of the committers and readers on this thread
> >> have actual RDBMs experience that goes beyond mySQL, and maybe
> Postgres.  I
> >> mentioned Sybase’s Adaptive Server as well as Informix’s IDS.  Anyone
> know
> >> what I’m talking about? How about Oracle? DB2?  I’ll even add Dick
> Pick’s
> >> Revelation or U2 (Universe) which are hierarchal systems.
> >>>>
> >>>> The fact that you said ‘it ain’t broken’ means that you just don’t
> >> understand the problem. Even after Kevin O’Dell admitted that his
> >> (Cloudera’s) customers who are using coprocessors are complaining of
> HBase
> >> not behaving in a stable fashion. It may be their code and lack of
> >> understanding, but Cloudera still has to address it and you can’t say
> using
> >> coprocessors voids the warranty. (Sorry Andrew, I had to laugh at that
> one…)
> >>>>
> >>>> I understand completely what coprocessors were designed for.  What you
> >> don’t seem to understand is that the design and implementation which
> falls
> >> short and open it up to potential reliability issues.
> >>>>
> >>>> To be clear… I haven’t even talked about APIs, I’m talking about
the
> >> core design.
> >>>>
> >>>> To the best of my knowledge,  MapR’s M7 doesn’t have coprocessors.
> I’ll
> >> wager that when they do, it will work and not have these issues. I
> believe
> >> that they are writing their stuff in C/C++, if so, then they’d have an
> >> advantage of using shared memory.  Apache would have write C/C++ code
> and
> >> wrap it in JNI… which you may not want to do…
> >>>>
> >>>> But getting back to the issue at hand… if HBase is viewed as hard
to
> >> tune and hard to keep up… people are going to look towards other
> solutions.
> >>>>
> >>>> I believe you can already disable the ability to create enduser
> >> coprocessors. When you load the last system coprocessor, you load one
> that
> >> looks to see who’s trying to add a coprocessor and you just deny it. If
> you
> >> wanted to make a formal change, then you would just have a database
> >> permission that you either GRANT or REVOKE the ability of a user the
> >> privilege  to add coprocessors. But that would mean more work for
> someone.
> >>>>
> >>>> -Mike
> >>>>
> >>>>> On May 18, 2014, at 9:58 PM, lars hofhansl <larsh@apache.org>
wrote:
> >>>>>
> >>>>> Coprocessors are a means to extend HBase. Nothing more, nothing
less.
> >> They are not stored procedures or triggers.
> >>>>> Not sure in how many other ways we can/need to phrase that.
> >>>>>
> >>>>>
> >>>>> I agree that there should be a simple way to disable user
> coprocessors
> >> (or at least disable loading from HDFS) for the security conscious.
> Let's
> >> do that, it's simple.
> >>>>>
> >>>>> There is nothing to "fix" since it ain't broken. It's only seems
> >> broken when you do not understand what it was designed for.
> >>>>>
> >>>>> You want a new API for less invasive things in a sandbox, more like
> >> stored procedures and triggers... Sure, let's do that too. But realize
> that
> >> is a *new* use case, and that we'll keep the old stuff.
> >>>>>
> >>>>> -- Lars
> >>>>>
> >>>>> ________________________________
> >>>>> From: Michael Segel <michael_segel@hotmail.com>
> >>>>> To: dev@hbase.apache.org; lars hofhansl <larsh@apache.org>
> >>>>> Sent: Sunday, May 18, 2014 10:21 AM
> >>>>> Subject: Re: On coprocessor API evolution
> >>>>>
> >>>>>
> >>>>> It doesn’t matter.
> >>>>>
> >>>>> Sure we can follow Vlad’s rules… but you still have to get to
the
> root
> >> of the problem and that is making coprocessors safe.
> >>>>>
> >>>>> Its not an easy fix, and it would mean pretty much starting from
> >> scratch. Trying to kludge a fix is harder and will not be as good.
> >>>>>
> >>>>> Maybe you can salvage some code, but the issue is fixing coprocessors
> >> at the lowest level and work back up.
> >>>>>
> >>>>> You have to isolate the code to one or more separate jvms so you
can
> >> not only stop, but reload the processes.
> >>>>> This is more than just simple triggers but also extensibility.
> >>>>>
> >>>>> If you could pick the brains of some of the folks still under Kevin
> >> Foster (@IBM) who work on IDS… you could get some ideas.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On May 18, 2014, at 7:01 AM, lars hofhansl <larsh@apache.org>
> wrote:
> >>>>>>
> >>>>>> We've seen similar issues with Filters. Those are good rules
to
> >> follow.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ________________________________
> >>>>>> From: Vladimir Rodionov <vladrodionov@gmail.com>
> >>>>>> To: "dev@hbase.apache.org" <dev@hbase.apache.org>
> >>>>>> Sent: Friday, May 16, 2014 10:59 AM
> >>>>>> Subject: Re: On coprocessor API evolution
> >>>>>>
> >>>>>>
> >>>>>> 1) Have default implementations (abstract classes) for every
> >> interface from
> >>>>>> Coprocessor API.
> >>>>>> 2) Advise coprocessor users not to implement interface directly
but
> >> sub
> >>>>>> class default impl.
> >>>>>> 3) Preserve backward compatibility by adding only new hooks/methods
> >>>>>> 4) DO NOT CHANGE existing API (no method renaming, method parameter
> >> type
> >>>>>> changes etc)
> >>>>>> 5) Have a regression tests to check backward compatibility.
> >>>>>>
> >>>>>> -Vladimir
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, May 16, 2014 at 9:13 AM, Michael Segel <
> >> michael_segel@hotmail.com>wrote:
> >>>>>>
> >>>>>>> Until you move the coprocessor out of the RS space and into
its own
> >>>>>>> sandbox… saying security and coprocessor in the same sentence
is a
> >> joke.
> >>>>>>> Oh wait… you were serious… :-(
> >>>>>>>
> >>>>>>> I’d say there’s a significant rethink on coprocessors
that’s
> >> required.
> >>>>>>>
> >>>>>>> Anyone running a secure (kerberos) cluster, will want to
allow
> system
> >>>>>>> coprocessors but then write a coprocessor that reject user
> >> coprocessors.
> >>>>>>>
> >>>>>>> Just putting it out there…
> >>>>>>>
> >>>>>>>> On May 15, 2014, at 2:13 AM, Andrew Purtell <apurtell@apache.org>
> >> wrote:
> >>>>>>>>
> >>>>>>>> Because coprocessor APIs are so tightly bound with internals,
if
> we
> >> apply
> >>>>>>>> suggested rules like as mentioned on HBASE-11054:
> >>>>>>>>
> >>>>>>>>     I'd say policy should be no changes to method apis
across
> minor
> >>>>>>>> versions
> >>>>>>>>
> >>>>>>>> This will lock coprocessor based components to the limitations
of
> >> the API
> >>>>>>>> as we encounter them. Core code does not suffer this
limitation,
> we
> >> are
> >>>>>>>> otherwise free to refactor and change internal methods.
For
> >> example, if
> >>>>>>> we
> >>>>>>>> apply this policy to the 0.98 branch, then we will have
to abandon
> >>>>>>> further
> >>>>>>>> security feature development there and move to trunk
only. This is
> >>>>>>> because
> >>>>>>>> we already are aware that coprocessor APIs as they stand
are
> >> insufficient
> >>>>>>>> still.
> >>>>>>>>
> >>>>>>>> Coprocessor APIs are a special class of internal method.
We have
> >> had a
> >>>>>>>> tension between allowing freedom of movement for developing
them
> >> out and
> >>>>>>>> providing some measure of stability for implementors
for a while.
> >>>>>>>>
> >>>>>>>> It is my belief that the way forward is something like
> HBASE-11125.
> >>>>>>> Perhaps
> >>>>>>>> we can take this discussion to that JIRA and have this
long
> overdue
> >>>>>>>> conversation.
> >>>>>>>>
> >>>>>>>> Regarding security features specifically, I would also
like to
> call
> >> your
> >>>>>>>> attention to HBASE-11127. I think security has been
an optional
> >> feature
> >>>>>>>> long enough, it is becoming a core requirement for the
project, so
> >> should
> >>>>>>>> be moved into core. Sure, we can therefore sidestep
any issues
> with
> >>>>>>>> coprocessor API sufficiency for hosting security features.
> However,
> >> in my
> >>>>>>>> opinion we should pursue both HBASE-11125 and HBASE-11127;
the
> >> first to
> >>>>>>>> provide the relative stability long asked for by coprocessor
API
> >> users,
> >>>>>>> the
> >>>>>>>> latter to cleanly solve emerging issues with concurrency
and
> >> versioning.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Best regards,
> >>>>>>>>
> >>>>>>>>  - Andy
> >>>>>>>>
> >>>>>>>> Problems worthy of attack prove their worth by hitting
back. -
> Piet
> >> Hein
> >>>>>>>> (via Tom White)
> >>>>
> >>>
> >>
> >>
> >
> >
> > --
> > Best regards,
> >
> >   - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message