Return-Path: X-Original-To: apmail-hbase-dev-archive@www.apache.org Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 012231139A for ; Mon, 19 May 2014 18:52:41 +0000 (UTC) Received: (qmail 79422 invoked by uid 500); 19 May 2014 18:52:40 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 79353 invoked by uid 500); 19 May 2014 18:52:40 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 79343 invoked by uid 99); 19 May 2014 18:52:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 May 2014 18:52:40 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.111.111 as permitted sender) Received: from [65.55.111.111] (HELO blu0-omc2-s36.blu0.hotmail.com) (65.55.111.111) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 May 2014 18:52:33 +0000 Received: from BLU436-SMTP136 ([65.55.111.73]) by blu0-omc2-s36.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Mon, 19 May 2014 11:52:11 -0700 X-TMN: [BZQDJ0379UKqFRqm+58dp2VXSnqw0xtY] X-Originating-Email: [michael_segel@hotmail.com] Message-ID: Received: from [172.17.112.39] ([91.214.5.86]) by BLU436-SMTP136.smtp.hotmail.com over TLS secured channel with Microsoft SMTPSVC(8.0.9200.16384); Mon, 19 May 2014 11:52:11 -0700 Content-Type: text/plain; charset="windows-1252" MIME-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: On coprocessor API evolution From: Michael Segel In-Reply-To: Date: Mon, 19 May 2014 19:52:06 +0100 Content-Transfer-Encoding: quoted-printable References: <1400392887.26951.YahooMailNeo@web140603.mail.bf1.yahoo.com> <1400446718.3009.YahooMailNeo@web140605.mail.bf1.yahoo.com> <74F84138-7CD7-4854-B4FB-1CD7E0ECDF41@gmail.com> To: dev@hbase.apache.org X-Mailer: Apple Mail (2.1874) X-OriginalArrivalTime: 19 May 2014 18:52:11.0321 (UTC) FILETIME=[712D3A90:01CF7393] X-Virus-Checked: Checked by ClamAV on apache.org Several of my friends have already walked away from using HBase in their = solutions.=20 Looks like I will have to do the same.=20 I=92ll limit my use of Hbase as a choice of last resort and even then = limit how its to be used. But to be honest, Andrew=85 your quote=85 =93Just tell them that it = voids the warranty=94 made my day. Told it to a friend and we got a good = laugh out of it.=20 I=92m going to have to capture that=85=20 On May 19, 2014, at 12:44 PM, Andrew Purtell = wrote: > I would say the serious problem here is you can clearly see that we > understand the issues, as you point out, and then fail to even for a = second > entertain a point of view that is not 100% yours. This can't go on. = Feel > free to continue mailing the list but I won't be receiving email from = you > further. >=20 >=20 > On Mon, May 19, 2014 at 4:18 AM, Michael Segel = wrote: >=20 >> Andrew, >>=20 >> I suggest that you go back and re-read Kevin O=92Dell=92s comment. >> Clearly there=92s no straw man here. >>=20 >> I thought your response was a joke, but apparently you=92re serious = about >> that. >>=20 >> Again, when you allow user code to run in the same JVM as the RS, you = add >> risk. Issues with stability and security need to be addressed. >> You post an earlier Jira where even you see this as an issue. = (That=92s the >> biggest irony.) >>=20 >> If you and the other committers can=92t recognize this as an issue=85 = then >> there is a more serious problem that needs to be addressed. >>=20 >>=20 >>=20 >> On May 19, 2014, at 11:24 AM, Andrew Purtell = >> wrote: >>=20 >>> Yes it's clear you have been laughing at many of the responses and = have >> been having a merry time concern trolling your favorite straw man. = You >> insist on fixating on analogies we have used to explain coprocessors = to >> advanced users and spinning that into an absolutist (and ridiculous) >> position that doesn't match up with what everyone else is telling = you. At >> some point you need to stop talking and start listening. The insults = are >> not helping your case. Or maybe that's the point. Yes Michael you are = the >> RDBMS king and we are all idiots. Happy? Now consider this: That = means >> nothing as HBase isn't even in that space. Maybe you should try this = over >> on the MySQL or Postgres mailing lists. >>>=20 >>>=20 >>>> On May 19, 2014, at 12:06 AM, Michael Segel = >> wrote: >>>>=20 >>>> Remind me how you do server side execution? >>>>=20 >>>> I suggest you go through the emails on both the dev and user hbase = list >> and you=92ll see that when describing coprocessors, you=92ll see the = terms >> =91trigger=92 and =91stored procedures=92 which are terms most DBAs = and Data >> modelers are familiar with. >>>>=20 >>>> And if you read my email earlier in the thread, I said that the = analogy >> fell a little short because of the extensibility aspect. >>>>=20 >>>> I have to ask, how many of the committers and readers on this = thread >> have actual RDBMs experience that goes beyond mySQL, and maybe = Postgres. I >> mentioned Sybase=92s Adaptive Server as well as Informix=92s IDS. = Anyone know >> what I=92m talking about? How about Oracle? DB2? I=92ll even add = Dick Pick=92s >> Revelation or U2 (Universe) which are hierarchal systems. >>>>=20 >>>> The fact that you said =91it ain=92t broken=92 means that you just = don=92t >> understand the problem. Even after Kevin O=92Dell admitted that his >> (Cloudera=92s) customers who are using coprocessors are complaining = of HBase >> not behaving in a stable fashion. It may be their code and lack of >> understanding, but Cloudera still has to address it and you can=92t = say using >> coprocessors voids the warranty. (Sorry Andrew, I had to laugh at = that one=85) >>>>=20 >>>> I understand completely what coprocessors were designed for. What = you >> don=92t seem to understand is that the design and implementation = which falls >> short and open it up to potential reliability issues. >>>>=20 >>>> To be clear=85 I haven=92t even talked about APIs, I=92m talking = about the >> core design. >>>>=20 >>>> To the best of my knowledge, MapR=92s M7 doesn=92t have = coprocessors. I=92ll >> wager that when they do, it will work and not have these issues. I = believe >> that they are writing their stuff in C/C++, if so, then they=92d have = an >> advantage of using shared memory. Apache would have write C/C++ code = and >> wrap it in JNI=85 which you may not want to do=85 >>>>=20 >>>> But getting back to the issue at hand=85 if HBase is viewed as hard = to >> tune and hard to keep up=85 people are going to look towards other = solutions. >>>>=20 >>>> I believe you can already disable the ability to create enduser >> coprocessors. When you load the last system coprocessor, you load one = that >> looks to see who=92s trying to add a coprocessor and you just deny = it. If you >> wanted to make a formal change, then you would just have a database >> permission that you either GRANT or REVOKE the ability of a user the >> privilege to add coprocessors. But that would mean more work for = someone. >>>>=20 >>>> -Mike >>>>=20 >>>>> On May 18, 2014, at 9:58 PM, lars hofhansl = wrote: >>>>>=20 >>>>> Coprocessors are a means to extend HBase. Nothing more, nothing = less. >> They are not stored procedures or triggers. >>>>> Not sure in how many other ways we can/need to phrase that. >>>>>=20 >>>>>=20 >>>>> I agree that there should be a simple way to disable user = coprocessors >> (or at least disable loading from HDFS) for the security conscious. = Let's >> do that, it's simple. >>>>>=20 >>>>> There is nothing to "fix" since it ain't broken. It's only seems >> broken when you do not understand what it was designed for. >>>>>=20 >>>>> You want a new API for less invasive things in a sandbox, more = like >> stored procedures and triggers... Sure, let's do that too. But = realize that >> is a *new* use case, and that we'll keep the old stuff. >>>>>=20 >>>>> -- Lars >>>>>=20 >>>>> ________________________________ >>>>> From: Michael Segel >>>>> To: dev@hbase.apache.org; lars hofhansl >>>>> Sent: Sunday, May 18, 2014 10:21 AM >>>>> Subject: Re: On coprocessor API evolution >>>>>=20 >>>>>=20 >>>>> It doesn=92t matter. >>>>>=20 >>>>> Sure we can follow Vlad=92s rules=85 but you still have to get to = the root >> of the problem and that is making coprocessors safe. >>>>>=20 >>>>> Its not an easy fix, and it would mean pretty much starting from >> scratch. Trying to kludge a fix is harder and will not be as good. >>>>>=20 >>>>> Maybe you can salvage some code, but the issue is fixing = coprocessors >> at the lowest level and work back up. >>>>>=20 >>>>> You have to isolate the code to one or more separate jvms so you = can >> not only stop, but reload the processes. >>>>> This is more than just simple triggers but also extensibility. >>>>>=20 >>>>> If you could pick the brains of some of the folks still under = Kevin >> Foster (@IBM) who work on IDS=85 you could get some ideas. >>>>>=20 >>>>>=20 >>>>>=20 >>>>>> On May 18, 2014, at 7:01 AM, lars hofhansl = wrote: >>>>>>=20 >>>>>> We've seen similar issues with Filters. Those are good rules to >> follow. >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> ________________________________ >>>>>> From: Vladimir Rodionov >>>>>> To: "dev@hbase.apache.org" >>>>>> Sent: Friday, May 16, 2014 10:59 AM >>>>>> Subject: Re: On coprocessor API evolution >>>>>>=20 >>>>>>=20 >>>>>> 1) Have default implementations (abstract classes) for every >> interface from >>>>>> Coprocessor API. >>>>>> 2) Advise coprocessor users not to implement interface directly = but >> sub >>>>>> class default impl. >>>>>> 3) Preserve backward compatibility by adding only new = hooks/methods >>>>>> 4) DO NOT CHANGE existing API (no method renaming, method = parameter >> type >>>>>> changes etc) >>>>>> 5) Have a regression tests to check backward compatibility. >>>>>>=20 >>>>>> -Vladimir >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> On Fri, May 16, 2014 at 9:13 AM, Michael Segel < >> michael_segel@hotmail.com>wrote: >>>>>>=20 >>>>>>> Until you move the coprocessor out of the RS space and into its = own >>>>>>> sandbox=85 saying security and coprocessor in the same sentence = is a >> joke. >>>>>>> Oh wait=85 you were serious=85 :-( >>>>>>>=20 >>>>>>> I=92d say there=92s a significant rethink on coprocessors that=92s= >> required. >>>>>>>=20 >>>>>>> Anyone running a secure (kerberos) cluster, will want to allow = system >>>>>>> coprocessors but then write a coprocessor that reject user >> coprocessors. >>>>>>>=20 >>>>>>> Just putting it out there=85 >>>>>>>=20 >>>>>>>> On May 15, 2014, at 2:13 AM, Andrew Purtell = >> wrote: >>>>>>>>=20 >>>>>>>> Because coprocessor APIs are so tightly bound with internals, = if we >> apply >>>>>>>> suggested rules like as mentioned on HBASE-11054: >>>>>>>>=20 >>>>>>>> I'd say policy should be no changes to method apis across = minor >>>>>>>> versions >>>>>>>>=20 >>>>>>>> This will lock coprocessor based components to the limitations = of >> the API >>>>>>>> as we encounter them. Core code does not suffer this = limitation, we >> are >>>>>>>> otherwise free to refactor and change internal methods. For >> example, if >>>>>>> we >>>>>>>> apply this policy to the 0.98 branch, then we will have to = abandon >>>>>>> further >>>>>>>> security feature development there and move to trunk only. This = is >>>>>>> because >>>>>>>> we already are aware that coprocessor APIs as they stand are >> insufficient >>>>>>>> still. >>>>>>>>=20 >>>>>>>> Coprocessor APIs are a special class of internal method. We = have >> had a >>>>>>>> tension between allowing freedom of movement for developing = them >> out and >>>>>>>> providing some measure of stability for implementors for a = while. >>>>>>>>=20 >>>>>>>> It is my belief that the way forward is something like = HBASE-11125. >>>>>>> Perhaps >>>>>>>> we can take this discussion to that JIRA and have this long = overdue >>>>>>>> conversation. >>>>>>>>=20 >>>>>>>> Regarding security features specifically, I would also like to = call >> your >>>>>>>> attention to HBASE-11127. I think security has been an optional >> feature >>>>>>>> long enough, it is becoming a core requirement for the project, = so >> should >>>>>>>> be moved into core. Sure, we can therefore sidestep any issues = with >>>>>>>> coprocessor API sufficiency for hosting security features. = However, >> in my >>>>>>>> opinion we should pursue both HBASE-11125 and HBASE-11127; the >> first to >>>>>>>> provide the relative stability long asked for by coprocessor = API >> users, >>>>>>> the >>>>>>>> latter to cleanly solve emerging issues with concurrency and >> versioning. >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> -- >>>>>>>> Best regards, >>>>>>>>=20 >>>>>>>> - Andy >>>>>>>>=20 >>>>>>>> Problems worthy of attack prove their worth by hitting back. - = Piet >> Hein >>>>>>>> (via Tom White) >>>>=20 >>>=20 >>=20 >>=20 >=20 >=20 > --=20 > Best regards, >=20 > - Andy >=20 > Problems worthy of attack prove their worth by hitting back. - Piet = Hein > (via Tom White)