Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
MIME-Version: 1.0
References: <CAC1ju53iTkoxkqtd01y5S4971y3ye3KUsvPRmr2wT2y6xySVDA@mail.gmail.com>
 <9346FF31-737E-4D20-A9A2-52910FBC9107@gmail.com> <CAC1ju51EwtiRYsXBUwQ=5RR9LGHXicE3F2zNSSXsX2-1VvhKbQ@mail.gmail.com>
 <CAJwFCa0Egsf78hScGvep-Xr+_42vKOM7zeZ49D2c_RzhMtEcUA@mail.gmail.com>
 <CA+oS81+CuV-a+07iAGVU3acyhXdUpOi-UpRBDxFQu_kB73AQNQ@mail.gmail.com>
 <CAKOFcwo9yLUNytQjoM65ommTkno=i9zzLkn1w5c49undXjWNjQ@mail.gmail.com>
 <CAJrw0ORvaAUueKVPg_LfftpgbM+aZch2n3NWVvK54G1Ziybj=w@mail.gmail.com>
 <589CC834-6617-4F2A-8A6D-8C4B5BA26AED@maprtech.com> <CAAOiHjEGsDFhOP+h8=uUyQ+tewC3Ucy0eARzLKXe+XappdVuYw@mail.gmail.com>
 <7DC68202-6186-4BC4-82DF-A01F5FB5D5A7@maprtech.com> <CAH-_v4+4XBsRzUMb2-p+6sOv3kQJC=6odpctwkFzGukFyExZPw@mail.gmail.com>
 <9F93A3B3-164F-461C-B1C2-C35D2CFCCD13@maprtech.com> <826331634.1895282.1466525700374.JavaMail.yahoo@mail.yahoo.com>
 <CAC1ju52boucFGfo3ftarSedQKr2+t=+KNti-Bn_UikqCokdmyQ@mail.gmail.com>
 <CAC1ju50ipsPKWRk78ctDPJ6N=u=0f=mn9+HyNiiX=KvLrX+fhQ@mail.gmail.com>
 <706D913F-08B8-4EAD-9651-758EACDBB163@maprtech.com> <CAC1ju52KKf-VNjG6xnYGfvruUjtqTpfyF9M5-TBaC08rQ_UvRA@mail.gmail.com>
 <CAAOiHjFAOwMj6cnYe_sLrzgX2-FEKcpunsgfvEw2N8giwGVx1w@mail.gmail.com>
 <E89A88C1-6D11-46AB-B5AD-822238B83F6E@maprtech.com> <CAC1ju5352ZacEbJeL02XSy5TGi8+q7nQtvY-BerTxG5t3JUQag@mail.gmail.com>
 <A75BAEE0-411D-4783-9F68-8EC8948F4631@maprtech.com> <CAAOiHjHfrBtCNwYCCq1cXkHGxCUPEJwH9TcgYo_EqupbcXenXg@mail.gmail.com>
 <CAAOiHjHm7n1GSyfPbyk61hmF6iOxbmyrNiBJMfb2Dp1g+47vFw@mail.gmail.com>
 <87DE4551-6E26-456B-AC56-5F9C893C64C7@maprtech.com> <EDEF0EA6-3699-44ED-962A-947F90A6861D@maprtech.com>
 <CAH-_v4KLkSNW2Jok8QZzGQLOYZ=6=nPNBve97aJWctxDrOKZSA@mail.gmail.com>
 <16F8B8DD-067C-46CC-8AB1-A9BE7BB18358@maprtech.com> <CAFt_Wg896NVduy8aupMAoFcC3MrjE9LJ2Cz2syDMz2umsxAYJQ@mail.gmail.com>
 <CAH-_v4+vhbxkPuLZLQ2Tx39WDUSCDqHfXEuth5aT+9F-NbZPOQ@mail.gmail.com>
 <5BD440FE-8A97-4AAC-A990-735498C5F831@maprtech.com> <CAC1ju50uDwZuOR5jOSWBhPATE2mjU+TFp-33d=ubNwTQbhD9DA@mail.gmail.com>
 <D5D33FF5-7E02-4794-81E9-6DB4E84C58E7@maprtech.com> <C031D9BC-ACD2-4435-BE45-0E17E6DC56D2@gmail.com>
In-Reply-To: <C031D9BC-ACD2-4435-BE45-0E17E6DC56D2@gmail.com>
From: Arina Yelchiyeva <arina.yelchiyeva@gmail.com>
Date: Mon, 25 Jul 2016 16:09:37 +0000
Message-ID: <CAC1ju50jJ=amcGSNwzxsD36emHMkr6Y+hLiEf=Kk7Q-cECSyGA@mail.gmail.com>
Subject: Re: Dynamic UDFs support
To: dev@drill.apache.org
Content-Type: multipart/alternative; boundary=089e013d17c46219cd0538780382
archived-at: Mon, 25 Jul 2016 16:10:03 -0000

--089e013d17c46219cd0538780382
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

My fault, agree, DROP is more appropriate.
Thanks Julian!

On Mon, Jul 25, 2016 at 7:07 PM Julian Hyde <jhyde.apache@gmail.com> wrote:

> But don't call it DELETE. In SQL the opposite of CREATE is DROP.
>
> Julian
>
> > On Jul 25, 2016, at 8:48 AM, Keys Botzum <kbotzum@maprtech.com> wrote:
> >
> > I like the approach to handling DELETE. This is very useful. I think an
> implementation that does not guarantee consistent behavior is perfectly
> fine for use that is targeted at developers that are working on UDFs. As
> long as the docs make the intent clear this makes me very happy.
> >
> > I'll defer to others more expert than I on the remainder of the design.
> >
> > Keys
> > _______________________________
> > Keys Botzum
> > Senior Principal Technologist
> > kbotzum@maprtech.com <mailto:kbotzum@maprtech.com>
> > 443-718-0098
> > MapR Technologies
> > http://www.mapr.com <http://www.mapr.com/>
> >> On Jul 25, 2016, at 9:55 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
> >>
> >> Taking into account all previous comments and discussion we had with
> Parth
> >> and Paul, please find below my design notes (I am going to prepare
> proper
> >> design document, just want to see if all agree with raw version).
> >> I propose will use lazy-init to dynamically loaded UDFs, in such case
> when
> >> user issues CREATE UDF command, foreman will only validate jar and
> update
> >> ZK function registry, and only if function is needed it will be loaded
> to
> >> appropriate drillbit (during planning stage or fragment execution). We
> >> might add listeners (as Paul proposed) to pre-load UDFs but I didn't
> >> include it to current release to simplify solution but we might
> re-consider
> >> this.
> >> I have looked at issue with class loading and unloading and if we ship
> each
> >> jar with its own classloader, DELETE functionality can be introduced i=
n
> >> current release, at least marked as experimental or for developers use
> >> only, to ease UDF development process.
> >>
> >> Any comments are welcomed.
> >>
> >> *Invariants*
> >>
> >> 1. DFS staging area where user copies jar to be loaded
> >>
> >> 2. DFS udf area (former registration area) where all validated jars ar=
e
> >> present
> >>
> >> 3. ZK function registry - contains list of all dynamically loaded UDFs
> and
> >> their jars. UDF name will be represented as combination of name and
> input
> >> parameters.
> >>
> >> 4. Lazy-init - all dynamically loaded UDFs will be loaded to drillbit
> upon
> >> request, i.e. if drillbits receives query or fragment that contains
> such UDF
> >>
> >> 5. Currently only CREATE and DELETE statements are supported
> >>
> >>
> >> *Adding UDFs*
> >>
> >> 1. User copies source and binary (hereinafter jar) to DFS staging area
> >> 2. User issues CREATE UDF command
> >> 3. Foreman receives request to create UDF:
> >> a) checks if jar is present in staging area
> >> b) copies jar to temporary DFS location
> >> c) validates UDFs present in jar locally:
> >> 1) copies jar to temporary local fs
> >> 2) scans jar using temporary classloader
> >> 3) checks if there are any duplicates in local function registry
> >> 4) returns list of UDFs to be registered
> >> d) validates UDFs present in jar in ZK:
> >> 1) takes list of dynamically loaded UDFs from ZK
> >> 2) checks if there are no duplicates either by jar name or among UDFs
> >> 3) moves jar from DFS temporary area to DFS udf area
> >> 4) updates ZK with list of new dynamic UDFs
> >> 5) removes jar from staging area
> >> 6) returns confirmation to user that UDFs were registered
> >>
> >>
> >> *Lazy-init*
> >>
> >> 1. User issues query with dynamically loaded UDF.
> >>
> >> 2. During planning stage or fragment execution, if UDF is not present =
in
> >> local function registry,  drillbit:
> >>
> >> a) checks if such UDF is present in ZK function registry
> >>
> >> b) if present, loads UDF using jar name, otherwise return an error
> >>
> >> c) proceeds planning stage or fragment execution
> >>
> >>
> >> *New drillbit registration / Drillbit re-start*
> >>
> >> Local udf directory is re-created, to clean up previously loaded jars
> if any
> >>
> >>
> >> *Delete UDF*
> >>
> >> Each jar that going to be loaded dynamically will have its own
> classloader
> >> which will solve problem with loading and unloading classes with the
> same
> >> name.
> >>
> >>
> >> 1. User issues DELETE command (delete will operate on jar name level)
> >>
> >> 2. Foreman receives DELETE request:
> >>
> >> a) checks if such jar is present in ZK function registry
> >>
> >> b) creates ephemeral znode /udf/delete/jar_name
> >>
> >> c) removes record in ZK function registry
> >>
> >> d) removes jar from DFS udf area
> >>
> >> e) removes ephemeral znode from /udf/delete/jar_name
> >>
> >> f) returns confirmation to user that UDFs were deleted
> >>
> >> 3. Drillbits are subscribed to /udf/delete znode, when new znode with
> jar
> >> name appears, drillbit:
> >>
> >> a) removes all UDFs associated with jar name from local function
> registry
> >>
> >> b) removes jar from local udf directory
> >>
> >>
> >> *Limitations*
> >>
> >> 1. When user runs DELETE command, some queries that are using deleted
> UDFs
> >> may fail during fragment execution if by that time UDF has been delete=
d
> >> from local registry. Ideally, before submitting DELETE command, user
> needs
> >> to make sure, no one is running queries using UDFs from that particula=
r
> jar.
> >>
> >>
> >> 2. We encourage users not to delete any jars from DFS udf area
> manually, as
> >> it may lead to inconsistency between ZK function registry and DFS udf
> area.
> >>
> >>
> >> 3. CREATE statement is not atomic in part when we copy validated jar t=
o
> DFS
> >> udf area and updating ZK function registry with list of new UDFs. In
> case
> >> of failure between these two steps, some unused jars may be left in DF=
S
> udf
> >> area but they won=E2=80=99t harm current process. LIST JARS command ca=
n be
> >> introduced to show used jars.
> >>
> >>
> >> Kind regards
> >> Arina
> >>
> >>> On Fri, Jul 22, 2016 at 7:15 PM Keys Botzum <kbotzum@maprtech.com>
> wrote:
> >>>
> >>> No disagreement on deferral but I raised my initial concern precisely
> >>> because I'm concerned about the practicality of the "restart the
> cluster"
> >>> option. I  sighted my concerns about laptops and development
> clusters.  I
> >>> was wondering if there might be some small things Drill could do to
> help.
> >>> If there is nothing that can be done to make this easier, so be it,
> but I
> >>> think that's going to be a big impedance.
> >>>
> >>> Keys
> >>> _______________________________
> >>> Keys Botzum
> >>> Senior Principal Technologist
> >>> kbotzum@maprtech.com <mailto:kbotzum@maprtech.com>
> >>> 443-718-0098
> >>> MapR Technologies
> >>> http://www.mapr.com <http://www.mapr.com/>
> >>>>> On Jul 22, 2016, at 1:37 AM, Neeraja Rentachintala <
> >>>> nrentachintala@maprtech.com> wrote:
> >>>>
> >>>> It seems like we are reaching a conclusion here in terms of starting
> >>> with a
> >>>> simpler implementation i.e being able to deploy UDFs dynamically
> without
> >>>> Drillbit restarts based off a jars in DFS location.  Dropping
> functions
> >>>> dynamically is out of scope for version 1 of this feature (we assume
> >>>> development of UDFs is happening on user laptop or a dev cluster whe=
re
> >>> its
> >>>> ok to have restart).
> >>>>
> >>>> -Neeraja
> >>>>
> >>>>> On Thu, Jul 21, 2016 at 11:56 AM, Keys Botzum <kbotzum@maprtech.com=
>
> >>>> wrote:
> >>>>
> >>>>> Recognize the difficulty. Not suggesting this be addressed in first
> >>>>> version. Just suggesting some thought about how a real user will
> >>>>> workaround. Maybe some doc and/or small changes can make this easie=
r.
> >>>>>
> >>>>> Keys
> >>>>> _______________________________
> >>>>> Keys Botzum
> >>>>> Senior Principal Technologist
> >>>>> kbotzum@maprtech.com
> >>>>> 443-718-0098
> >>>>> MapR Technologies
> >>>>> http://www.mapr.com
> >>>>>> On Jul 21, 2016 1:45 PM, "Paul Rogers" <progers@maprtech.com>
> wrote:
> >>>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> Adding a dynamic DROP would, of course, be a great addition! The
> reason
> >>>>>> for suggesting we skip that was to control project scope.
> >>>>>>
> >>>>>> Dynamic DROP requires a synchronization step. Here=E2=80=99s the s=
cenario:
> >>>>>>
> >>>>>> * Foreman A starts a query using UDF U.
> >>>>>> * Foreman B receives a request to drop UDF U, followed by a reques=
t
> to
> >>>>> add
> >>>>>> a new version of U, U=E2=80=99.
> >>>>>>
> >>>>>> How do we drop a function that may be in use? There are some trick=
y
> >>> bits
> >>>>>> to work out, which seemed too overwhelming to consider all in one
> go.
> >>>>>>
> >>>>>> Clearly just dropping U and adding a new version of U with the sam=
e
> >>> name
> >>>>>> leads to issues if not synchronized. If a Drillbit D is running a
> query
> >>>>>> with U when it receives notice to drop U, should D complete the
> query
> >>> or
> >>>>>> fail it? If the query completes, then how does D deal with the
> request
> >>> to
> >>>>>> register U=E2=80=99, which has the same name?
> >>>>>>
> >>>>>> Do we globally synchronize function deletion? (The foreman B that
> >>>>> receives
> >>>>>> the drop request waits for all queries using U to finish.) But, ho=
w
> do
> >>> we
> >>>>>> know which queries use U?
> >>>>>>
> >>>>>> An eventually consistent approach is to track the age of the oldes=
t
> >>>>>> running query. Suppose B drops U at time T. Any query received
> after T
> >>>>> that
> >>>>>> uses U will fail in planning. A new U=E2=80=99 can=E2=80=99t be re=
gistered until all
> >>>>>> queries that started before T complete.
> >>>>>>
> >>>>>> The primary challenge we face in both the CREATE and DROP cases is
> that
> >>>>>> Drill is distributed with little central coordination. That=E2=80=
=99s great
> for
> >>>>>> scale, but makes it hard to design features that require
> coordination.
> >>>>> Some
> >>>>>> other tools solve this problem with a data dictionary (or
> =E2=80=9Cmetastore").
> >>>>>> Alas, Drill does not have such a concept. So a seemingly simple
> feature
> >>>>>> like dynamic UDF becomes a major design challenge to get right.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> - Paul
> >>>>>>
> >>>>>>>> On Jul 21, 2016, at 7:21 AM, Neeraja Rentachintala <
> >>>>>>> nrentachintala@maprtech.com> wrote:
> >>>>>>>
> >>>>>>> The whole point of this feature is to avoid Drill cluster restart=
s
> as
> >>>>> the
> >>>>>>> name indicates 'Dynamic' UDFs.
> >>>>>>> So any design that requires restarts I would think would beat the
> >>>>>> purpose.
> >>>>>>>
> >>>>>>> I also think this is an example of a feature we start with a simp=
le
> >>>>>> design
> >>>>>>> to serve the purpose, take feedback on how it is being
> deployed/used
> >>> in
> >>>>>>> real user situations and improve it in subsequent releases.
> >>>>>>>
> >>>>>>> -thanks
> >>>>>>> Neeraja
> >>>>>>>
> >>>>>>>> On Thu, Jul 21, 2016 at 6:32 AM, Keys Botzum <
> kbotzum@maprtech.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> I think there are a lot of great ideas here. My one concern is t=
he
> >>>>> lack
> >>>>>> of
> >>>>>>>> unload and thus presumably replace functionality. I'm just
> thinking
> >>>>>> about
> >>>>>>>> typical actual usage.
> >>>>>>>>
> >>>>>>>> In a typical development cycle someone writes something, tries i=
t,
> >>>>>> learns,
> >>>>>>>> changes it, and tries again. Assuming I understand the design th=
at
> >>>>>> change
> >>>>>>>> step requires a full Drill cluster restart. That is going to be
> very
> >>>>>>>> disruptive and will make UDF work nearly impossible without a
> >>>>> dedicated
> >>>>>>>> "private" cluster for Drill. I realize that people should have
> access
> >>>>> to
> >>>>>>>> the data they need and Drill in a development cluster but even
> then
> >>>>>>>> restarts can be hard since development clusters are often shared=
 -
> >>> and
> >>>>>>>> that's assuming such a cluster exists. I realize of course Drill
> can
> >>>>> be
> >>>>>> run
> >>>>>>>> as a standalone Drillbit but I'm not convinced that desktops wil=
l
> >>> have
> >>>>>>>> adequate access to the needed data.
> >>>>>>>>
> >>>>>>>> Having dealt with Java classloading over the years, I'm not
> claiming
> >>>>>> class
> >>>>>>>> replacement is an easy thing so I'll defer to others on the
> priority
> >>>>> of
> >>>>>>>> that, but I'm wondering if there isn't some way to make UDF
> >>>>>> experimentation
> >>>>>>>> a bit easier/practical.
> >>>>>>>>
> >>>>>>>> Given the above, let me toss out some possibly naive ideas that
> maybe
> >>>>>> are
> >>>>>>>> workable:
> >>>>>>>> * can I easily run a standalone Drillbit on a Hadoop cluster nod=
e
> >>> that
> >>>>>> is
> >>>>>>>> already running Drill servers? I'm sure this can be done, but is
> it
> >>>>>> easy?
> >>>>>>>> Could we perhaps make this clearer as an explicit kind of thing?
> >>>>>>>> * is there a way that when I deploy a UDF I can constrain the # =
of
> >>>>> bits
> >>>>>> it
> >>>>>>>> is loaded into and perhaps even specify the bits?
> >>>>>>>> * Obvious correlarary is I'd want my query to run on those bits
> and a
> >>>>>>>> not too disruptive way to restart just those bits
> >>>>>>>>
> >>>>>>>> The above may be obvious to Drill experts. If it is then perhaps
> the
> >>>>> UDF
> >>>>>>>> docs could just point out how to easily develop UDFs in an
> iterative
> >>>>>>>> fashion.
> >>>>>>>>
> >>>>>>>> Keys
> >>>>>>>> _______________________________
> >>>>>>>> Keys Botzum
> >>>>>>>> Senior Principal Technologist
> >>>>>>>> kbotzum@maprtech.com <mailto:kbotzum@maprtech.com>
> >>>>>>>> 443-718-0098
> >>>>>>>> MapR Technologies
> >>>>>>>> http://www.mapr.com <http://www.mapr.com/>
> >
>

--089e013d17c46219cd0538780382--