Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A0692200B4F for ; Tue, 26 Jul 2016 19:06:48 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 9EED8160A75; Tue, 26 Jul 2016 17:06:48 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6D9FA160A69 for ; Tue, 26 Jul 2016 19:06:47 +0200 (CEST) Received: (qmail 5930 invoked by uid 500); 26 Jul 2016 17:06:46 -0000 Mailing-List: contact dev-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list dev@drill.apache.org Received: (qmail 5918 invoked by uid 99); 26 Jul 2016 17:06:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jul 2016 17:06:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 72503CB9B5 for ; Tue, 26 Jul 2016 17:06:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.198 X-Spam-Level: * X-Spam-Status: No, score=1.198 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=maprtech.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id pHzL0nc5ar73 for ; Tue, 26 Jul 2016 17:06:41 +0000 (UTC) Received: from mail-pf0-f180.google.com (mail-pf0-f180.google.com [209.85.192.180]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id 71E9A5F245 for ; Tue, 26 Jul 2016 17:06:40 +0000 (UTC) Received: by mail-pf0-f180.google.com with SMTP id y134so1638143pfg.0 for ; Tue, 26 Jul 2016 10:06:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=maprtech.com; s=google; h=from:message-id:mime-version:subject:date:references:to:in-reply-to; bh=06Ku4b3DFXZ3hJBHpsOBJPnOHu6RNVfAvOKp6tPkDYc=; b=UzAmTjDjetRWPgxdsQN4dhcyJR8SedlV6uZJU6CUfALJB1WGvDPDJQoDCetONOoxf+ SNYIk+bqByGu9MVMDraAfSEwzzOabXcaJt+QNT19KQHdk7lAEkBE8YZcgXIsi53XxiKM vcczDYlgDqd7d0d9+uS8GOyDCM4U5cU89YKZI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:message-id:mime-version:subject:date :references:to:in-reply-to; bh=06Ku4b3DFXZ3hJBHpsOBJPnOHu6RNVfAvOKp6tPkDYc=; b=R79Jh9ItJGmTG6mLGs4VgD4zD3jO1vN8z+2XWu4n7arroZYCrSToGcRhd4sdD4Dy/a +74734zkjbdUxXDia9kiUMYHqXEziTQ72RvL5jkxlAbUDR85qUgYw9pHb11tTlxkWOnK JB+i5dFikDI4Wdon2a2TqEdHdOeC8ObFvOM4ZZ/HNdtHjJ4kx3nj2CRp+S88PeHS5y2w WR6AEnaJIwasrpCdvEOZrsCbQSU6h+/2H97eaCyjymKgetCge7vsWFldoMz5Sg0fQMqM oKRnUI+VNTJuDrE8VEBX+Zl7jcRKqZKqQT5AT7/BGmTarHtNcvPfZlRT92z4fKN+djQt 1/lA== X-Gm-Message-State: AEkoouv4mDVwH/Fz12UWLUZXOFruyQj52MoftHaWZLKGb0PAE7BqHtsvwbvtANsvCFI1mc+k X-Received: by 10.98.208.1 with SMTP id p1mr41366390pfg.10.1469552798871; Tue, 26 Jul 2016 10:06:38 -0700 (PDT) Received: from [10.131.193.192] ([192.12.13.9]) by smtp.gmail.com with ESMTPSA id xx7sm2983426pac.3.2016.07.26.10.06.36 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 26 Jul 2016 10:06:37 -0700 (PDT) From: Keys Botzum Content-Type: multipart/alternative; boundary="Apple-Mail=_C80EE410-A268-4C89-8B54-913B9BB5C9D8" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: Dynamic UDFs support Date: Tue, 26 Jul 2016 13:06:35 -0400 References: <9346FF31-737E-4D20-A9A2-52910FBC9107@gmail.com> <589CC834-6617-4F2A-8A6D-8C4B5BA26AED@maprtech.com> <7DC68202-6186-4BC4-82DF-A01F5FB5D5A7@maprtech.com> <9F93A3B3-164F-461C-B1C2-C35D2CFCCD13@maprtech.com> <826331634.1895282.1466525700374.JavaMail.yahoo@mail.yahoo.com> <706D913F-08B8-4EAD-9651-75 8EACDBB163@maprtech.com> <87DE4551-6E26-456B-AC56-5F9C893C64C7@maprtech.com> <16F8B8DD-067C-46CC-8AB1-A9BE7BB18358@maprtech.com> <5BD440FE-8A97-4AAC-A990-735498C5F831@maprtech.com> <319439126.4842740.1469552716259.JavaMail.yahoo@mail.yahoo.com> To: dev@drill.apache.org, yuliya Feldman In-Reply-To: <319439126.4842740.1469552716259.JavaMail.yahoo@mail.yahoo.com> X-Mailer: Apple Mail (2.3112) archived-at: Tue, 26 Jul 2016 17:06:48 -0000 --Apple-Mail=_C80EE410-A268-4C89-8B54-913B9BB5C9D8 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 +1 Keys _______________________________ Keys Botzum=20 Senior Principal Technologist kbotzum@maprtech.com 443-718-0098 MapR Technologies=20 http://www.mapr.com > On Jul 26, 2016, at 1:05 PM, yuliya Feldman = wrote: >=20 > I want to make sure (also will make a note in the design doc) that we = have an option to disable dynamic loading/unloading of UDFs until we = will be able to have an ability to do proper authentication AND = authorization of the user(s). >=20 > From: Arina Yelchiyeva > > To: dev@drill.apache.org =20 > Sent: Monday, July 25, 2016 9:09 AM > Subject: Re: Dynamic UDFs support >=20 > My fault, agree, DROP is more appropriate. > Thanks Julian! >=20 > On Mon, Jul 25, 2016 at 7:07 PM Julian Hyde > wrote: >=20 >> But don't call it DELETE. In SQL the opposite of CREATE is DROP. >>=20 >> Julian >>=20 >>> On Jul 25, 2016, at 8:48 AM, Keys Botzum > wrote: >>>=20 >>> I like the approach to handling DELETE. This is very useful. I think = an >> implementation that does not guarantee consistent behavior is = perfectly >> fine for use that is targeted at developers that are working on UDFs. = As >> long as the docs make the intent clear this makes me very happy. >>>=20 >>> I'll defer to others more expert than I on the remainder of the = design. >>>=20 >>> Keys >>> _______________________________ >>> Keys Botzum >>> Senior Principal Technologist >>> kbotzum@maprtech.com = > >>> 443-718-0098 >>> MapR Technologies >>> http://www.mapr.com > >>>> On Jul 25, 2016, at 9:55 AM, Arina Yelchiyeva < >> arina.yelchiyeva@gmail.com > = wrote: >>>>=20 >>>> Taking into account all previous comments and discussion we had = with >> Parth >>>> and Paul, please find below my design notes (I am going to prepare >> proper >>>> design document, just want to see if all agree with raw version). >>>> I propose will use lazy-init to dynamically loaded UDFs, in such = case >> when >>>> user issues CREATE UDF command, foreman will only validate jar and >> update >>>> ZK function registry, and only if function is needed it will be = loaded >> to >>>> appropriate drillbit (during planning stage or fragment execution). = We >>>> might add listeners (as Paul proposed) to pre-load UDFs but I = didn't >>>> include it to current release to simplify solution but we might >> re-consider >>>> this. >>>> I have looked at issue with class loading and unloading and if we = ship >> each >>>> jar with its own classloader, DELETE functionality can be = introduced in >>>> current release, at least marked as experimental or for developers = use >>>> only, to ease UDF development process. >>>>=20 >>>> Any comments are welcomed. >>>>=20 >>>> *Invariants* >>>>=20 >>>> 1. DFS staging area where user copies jar to be loaded >>>>=20 >>>> 2. DFS udf area (former registration area) where all validated jars = are >>>> present >>>>=20 >>>> 3. ZK function registry - contains list of all dynamically loaded = UDFs >> and >>>> their jars. UDF name will be represented as combination of name and >> input >>>> parameters. >>>>=20 >>>> 4. Lazy-init - all dynamically loaded UDFs will be loaded to = drillbit >> upon >>>> request, i.e. if drillbits receives query or fragment that contains >> such UDF >>>>=20 >>>> 5. Currently only CREATE and DELETE statements are supported >>>>=20 >>>>=20 >>>> *Adding UDFs* >>>>=20 >>>> 1. User copies source and binary (hereinafter jar) to DFS staging = area >>>> 2. User issues CREATE UDF command >>>> 3. Foreman receives request to create UDF: >>>> a) checks if jar is present in staging area >>>> b) copies jar to temporary DFS location >>>> c) validates UDFs present in jar locally: >>>> 1) copies jar to temporary local fs >>>> 2) scans jar using temporary classloader >>>> 3) checks if there are any duplicates in local function registry >>>> 4) returns list of UDFs to be registered >>>> d) validates UDFs present in jar in ZK: >>>> 1) takes list of dynamically loaded UDFs from ZK >>>> 2) checks if there are no duplicates either by jar name or among = UDFs >>>> 3) moves jar from DFS temporary area to DFS udf area >>>> 4) updates ZK with list of new dynamic UDFs >>>> 5) removes jar from staging area >>>> 6) returns confirmation to user that UDFs were registered >>>>=20 >>>>=20 >>>> *Lazy-init* >>>>=20 >>>> 1. User issues query with dynamically loaded UDF. >>>>=20 >>>> 2. During planning stage or fragment execution, if UDF is not = present in >>>> local function registry, drillbit: >>>>=20 >>>> a) checks if such UDF is present in ZK function registry >>>>=20 >>>> b) if present, loads UDF using jar name, otherwise return an error >>>>=20 >>>> c) proceeds planning stage or fragment execution >>>>=20 >>>>=20 >>>> *New drillbit registration / Drillbit re-start* >>>>=20 >>>> Local udf directory is re-created, to clean up previously loaded = jars >> if any >>>>=20 >>>>=20 >>>> *Delete UDF* >>>>=20 >>>> Each jar that going to be loaded dynamically will have its own >> classloader >>>> which will solve problem with loading and unloading classes with = the >> same >>>> name. >>>>=20 >>>>=20 >>>> 1. User issues DELETE command (delete will operate on jar name = level) >>>>=20 >>>> 2. Foreman receives DELETE request: >>>>=20 >>>> a) checks if such jar is present in ZK function registry >>>>=20 >>>> b) creates ephemeral znode /udf/delete/jar_name >>>>=20 >>>> c) removes record in ZK function registry >>>>=20 >>>> d) removes jar from DFS udf area >>>>=20 >>>> e) removes ephemeral znode from /udf/delete/jar_name >>>>=20 >>>> f) returns confirmation to user that UDFs were deleted >>>>=20 >>>> 3. Drillbits are subscribed to /udf/delete znode, when new znode = with >> jar >>>> name appears, drillbit: >>>>=20 >>>> a) removes all UDFs associated with jar name from local function >> registry >>>>=20 >>>> b) removes jar from local udf directory >>>>=20 >>>>=20 >>>> *Limitations* >>>>=20 >>>> 1. When user runs DELETE command, some queries that are using = deleted >> UDFs >>>> may fail during fragment execution if by that time UDF has been = deleted >>>> from local registry. Ideally, before submitting DELETE command, = user >> needs >>>> to make sure, no one is running queries using UDFs from that = particular >> jar. >>>>=20 >>>>=20 >>>> 2. We encourage users not to delete any jars from DFS udf area >> manually, as >>>> it may lead to inconsistency between ZK function registry and DFS = udf >> area. >>>>=20 >>>>=20 >>>> 3. CREATE statement is not atomic in part when we copy validated = jar to >> DFS >>>> udf area and updating ZK function registry with list of new UDFs. = In >> case >>>> of failure between these two steps, some unused jars may be left in = DFS >> udf >>>> area but they won=E2=80=99t harm current process. LIST JARS command = can be >>>> introduced to show used jars. >>>>=20 >>>>=20 >>>> Kind regards >>>> Arina >>>>=20 >>>>> On Fri, Jul 22, 2016 at 7:15 PM Keys Botzum > >> wrote: >>>>>=20 >>>>> No disagreement on deferral but I raised my initial concern = precisely >>>>> because I'm concerned about the practicality of the "restart the >> cluster" >>>>> option. I sighted my concerns about laptops and development >> clusters. I >>>>> was wondering if there might be some small things Drill could do = to >> help. >>>>> If there is nothing that can be done to make this easier, so be = it, >> but I >>>>> think that's going to be a big impedance. >>>>>=20 >>>>> Keys >>>>> _______________________________ >>>>> Keys Botzum >>>>> Senior Principal Technologist >>>>> kbotzum@maprtech.com = > >>>>> 443-718-0098 >>>>> MapR Technologies >>>>> http://www.mapr.com > >>>>>>> On Jul 22, 2016, at 1:37 AM, Neeraja Rentachintala < >>>>>> nrentachintala@maprtech.com > = wrote: >>>>>>=20 >>>>>> It seems like we are reaching a conclusion here in terms of = starting >>>>> with a >>>>>> simpler implementation i.e being able to deploy UDFs dynamically >> without >>>>>> Drillbit restarts based off a jars in DFS location. Dropping >> functions >>>>>> dynamically is out of scope for version 1 of this feature (we = assume >>>>>> development of UDFs is happening on user laptop or a dev cluster = where >>>>> its >>>>>> ok to have restart). >>>>>>=20 >>>>>> -Neeraja >>>>>>=20 >>>>>>> On Thu, Jul 21, 2016 at 11:56 AM, Keys Botzum = > >>>>>> wrote: >>>>>>=20 >>>>>>> Recognize the difficulty. Not suggesting this be addressed in = first >>>>>>> version. Just suggesting some thought about how a real user will >>>>>>> workaround. Maybe some doc and/or small changes can make this = easier. >>>>>>>=20 >>>>>>> Keys >>>>>>> _______________________________ >>>>>>> Keys Botzum >>>>>>> Senior Principal Technologist >>>>>>> kbotzum@maprtech.com >>>>>>> 443-718-0098 >>>>>>> MapR Technologies >>>>>>> http://www.mapr.com >>>>>>>> On Jul 21, 2016 1:45 PM, "Paul Rogers" >> wrote: >>>>>>>>=20 >>>>>>>> Hi All, >>>>>>>>=20 >>>>>>>> Adding a dynamic DROP would, of course, be a great addition! = The >> reason >>>>>>>> for suggesting we skip that was to control project scope. >>>>>>>>=20 >>>>>>>> Dynamic DROP requires a synchronization step. Here=E2=80=99s = the scenario: >>>>>>>>=20 >>>>>>>> * Foreman A starts a query using UDF U. >>>>>>>> * Foreman B receives a request to drop UDF U, followed by a = request >> to >>>>>>> add >>>>>>>> a new version of U, U=E2=80=99. >>>>>>>>=20 >>>>>>>> How do we drop a function that may be in use? There are some = tricky >>>>> bits >>>>>>>> to work out, which seemed too overwhelming to consider all in = one >> go. >>>>>>>>=20 >>>>>>>> Clearly just dropping U and adding a new version of U with the = same >>>>> name >>>>>>>> leads to issues if not synchronized. If a Drillbit D is running = a >> query >>>>>>>> with U when it receives notice to drop U, should D complete the >> query >>>>> or >>>>>>>> fail it? If the query completes, then how does D deal with the >> request >>>>> to >>>>>>>> register U=E2=80=99, which has the same name? >>>>>>>>=20 >>>>>>>> Do we globally synchronize function deletion? (The foreman B = that >>>>>>> receives >>>>>>>> the drop request waits for all queries using U to finish.) But, = how >> do >>>>> we >>>>>>>> know which queries use U? >>>>>>>>=20 >>>>>>>> An eventually consistent approach is to track the age of the = oldest >>>>>>>> running query. Suppose B drops U at time T. Any query received >> after T >>>>>>> that >>>>>>>> uses U will fail in planning. A new U=E2=80=99 can=E2=80=99t be = registered until all >>>>>>>> queries that started before T complete. >>>>>>>>=20 >>>>>>>> The primary challenge we face in both the CREATE and DROP cases = is >> that >>>>>>>> Drill is distributed with little central coordination. That=E2=80= =99s great >> for >>>>>>>> scale, but makes it hard to design features that require >> coordination. >>>>>>> Some >>>>>>>> other tools solve this problem with a data dictionary (or >> =E2=80=9Cmetastore"). >>>>>>>> Alas, Drill does not have such a concept. So a seemingly simple >> feature >>>>>>>> like dynamic UDF becomes a major design challenge to get right. >>>>>>>>=20 >>>>>>>> Thanks, >>>>>>>>=20 >>>>>>>> - Paul >>>>>>>>=20 >>>>>>>>>> On Jul 21, 2016, at 7:21 AM, Neeraja Rentachintala < >>>>>>>>> nrentachintala@maprtech.com> wrote: >>>>>>>>>=20 >>>>>>>>> The whole point of this feature is to avoid Drill cluster = restarts >> as >>>>>>> the >>>>>>>>> name indicates 'Dynamic' UDFs. >>>>>>>>> So any design that requires restarts I would think would beat = the >>>>>>>> purpose. >>>>>>>>>=20 >>>>>>>>> I also think this is an example of a feature we start with a = simple >>>>>>>> design >>>>>>>>> to serve the purpose, take feedback on how it is being >> deployed/used >>>>> in >>>>>>>>> real user situations and improve it in subsequent releases. >>>>>>>>>=20 >>>>>>>>> -thanks >>>>>>>>> Neeraja >>>>>>>>>=20 >>>>>>>>>> On Thu, Jul 21, 2016 at 6:32 AM, Keys Botzum < >> kbotzum@maprtech.com> >>>>>>>>> wrote: >>>>>>>>>=20 >>>>>>>>>> I think there are a lot of great ideas here. My one concern = is the >>>>>>> lack >>>>>>>> of >>>>>>>>>> unload and thus presumably replace functionality. I'm just >> thinking >>>>>>>> about >>>>>>>>>> typical actual usage. >>>>>>>>>>=20 >>>>>>>>>> In a typical development cycle someone writes something, = tries it, >>>>>>>> learns, >>>>>>>>>> changes it, and tries again. Assuming I understand the design = that >>>>>>>> change >>>>>>>>>> step requires a full Drill cluster restart. That is going to = be >> very >>>>>>>>>> disruptive and will make UDF work nearly impossible without a >>>>>>> dedicated >>>>>>>>>> "private" cluster for Drill. I realize that people should = have >> access >>>>>>> to >>>>>>>>>> the data they need and Drill in a development cluster but = even >> then >>>>>>>>>> restarts can be hard since development clusters are often = shared - >>>>> and >>>>>>>>>> that's assuming such a cluster exists. I realize of course = Drill >> can >>>>>>> be >>>>>>>> run >>>>>>>>>> as a standalone Drillbit but I'm not convinced that desktops = will >>>>> have >>>>>>>>>> adequate access to the needed data. >>>>>>>>>>=20 >>>>>>>>>> Having dealt with Java classloading over the years, I'm not >> claiming >>>>>>>> class >>>>>>>>>> replacement is an easy thing so I'll defer to others on the >> priority >>>>>>> of >>>>>>>>>> that, but I'm wondering if there isn't some way to make UDF >>>>>>>> experimentation >>>>>>>>>> a bit easier/practical. >>>>>>>>>>=20 >>>>>>>>>> Given the above, let me toss out some possibly naive ideas = that >> maybe >>>>>>>> are >>>>>>>>>> workable: >>>>>>>>>> * can I easily run a standalone Drillbit on a Hadoop cluster = node >>>>> that >>>>>>>> is >>>>>>>>>> already running Drill servers? I'm sure this can be done, but = is >> it >>>>>>>> easy? >>>>>>>>>> Could we perhaps make this clearer as an explicit kind of = thing? >>>>>>>>>> * is there a way that when I deploy a UDF I can constrain the = # of >>>>>>> bits >>>>>>>> it >>>>>>>>>> is loaded into and perhaps even specify the bits? >>>>>>>>>> * Obvious correlarary is I'd want my query to run on those = bits >> and a >>>>>>>>>> not too disruptive way to restart just those bits >>>>>>>>>>=20 >>>>>>>>>> The above may be obvious to Drill experts. If it is then = perhaps >> the >>>>>>> UDF >>>>>>>>>> docs could just point out how to easily develop UDFs in an >> iterative >>>>>>>>>> fashion. >>>>>>>>>>=20 >>>>>>>>>> Keys >>>>>>>>>> _______________________________ >>>>>>>>>> Keys Botzum >>>>>>>>>> Senior Principal Technologist >>>>>>>>>> kbotzum@maprtech.com >>>>>>>>>> 443-718-0098 >>>>>>>>>> MapR Technologies >>>>>>>>>> http://www.mapr.com --Apple-Mail=_C80EE410-A268-4C89-8B54-913B9BB5C9D8--