Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Tue, 21 Jun 2016 16:15:00 +0000 (UTC)
From: yuliya Feldman <yufeldman@yahoo.com.INVALID>
Reply-To: yuliya Feldman <yufeldman@yahoo.com>
To: "dev@drill.apache.org" <dev@drill.apache.org>
Message-ID: <826331634.1895282.1466525700374.JavaMail.yahoo@mail.yahoo.com>
In-Reply-To: <9F93A3B3-164F-461C-B1C2-C35D2CFCCD13@maprtech.com>
References: <CAC1ju53iTkoxkqtd01y5S4971y3ye3KUsvPRmr2wT2y6xySVDA@mail.gmail.com> <9346FF31-737E-4D20-A9A2-52910FBC9107@gmail.com> <CAC1ju51EwtiRYsXBUwQ=5RR9LGHXicE3F2zNSSXsX2-1VvhKbQ@mail.gmail.com> <CAJwFCa0Egsf78hScGvep-Xr+_42vKOM7zeZ49D2c_RzhMtEcUA@mail.gmail.com> <CA+oS81+CuV-a+07iAGVU3acyhXdUpOi-UpRBDxFQu_kB73AQNQ@mail.gmail.com> <CAKOFcwo9yLUNytQjoM65ommTkno=i9zzLkn1w5c49undXjWNjQ@mail.gmail.com> <CAJrw0ORvaAUueKVPg_LfftpgbM+aZch2n3NWVvK54G1Ziybj=w@mail.gmail.com> <589CC834-6617-4F2A-8A6D-8C4B5BA26AED@maprtech.com> <CAAOiHjEGsDFhOP+h8=uUyQ+tewC3Ucy0eARzLKXe+XappdVuYw@mail.gmail.com> <7DC68202-6186-4BC4-82DF-A01F5FB5D5A7@maprtech.com> <CAH-_v4+4XBsRzUMb2-p+6sOv3kQJC=6odpctwkFzGukFyExZPw@mail.gmail.com> <9F93A3B3-164F-461C-B1C2-C35D2CFCCD13@maprtech.com>
Subject: Re: Dynamic UDFs support
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_1895281_1885777437.1466525700362"
archived-at: Tue, 21 Jun 2016 16:15:17 -0000

------=_Part_1895281_1885777437.1466525700362
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Just thoughts:
You can try to reuse distributed cache=C2=A0Let Drill AM do the needful in =
terms of orchestrating UDF jars distribution.
But
I would be inclined to have a common path that is independent of the fact t=
hat it is Drill on YARN or not, as maintaining two separate ways of dealing=
 with loading/unloading UDFs will be painful and error prone.
One more note (I left a comment in the doc) - not sure about authorization =
model here - we need to have some.
Just my 2cThanks

      From: Paul Rogers <progers@maprtech.com>
 To: "dev@drill.apache.org" <dev@drill.apache.org>=20
 Sent: Monday, June 20, 2016 7:32 PM
 Subject: Re: Dynamic UDFs support
  =20
Hi Neeraja,

The proposal calls for the user to copy the jar file to each Drillbit node.=
 The jar would go into a new $DRILL_HOME/jars/3rdparty/udf directory.

In Drill-on-YARN (DoY), YARN is responsible for copying Drill code to each =
node (which is good.) YARN puts that code in a location known only to YARN.=
 Since the location is private to YARN, the user can=E2=80=99t easily hunt =
down the location in order to add the udf jar. Even if the user did find th=
e location, the next Drillbit to start would create a new copy of the Drill=
 software, without the udf jar.

Second, in DoY we have separated user files from Drill software. This makes=
 it much easier to distribute the software to each node: we give the Drill =
distribution tar archive to YARN, and YARN copies it to each node and untar=
s the Drill files. We make a separate copy of the (far smaller) set of user=
 config files.

If the udf jar goes into a Drill folder ($DRILL_HOME/jars/3rdparty/udf), th=
en the user would have to rebuild the Drill tar file each time they add a u=
df jar. When I tried this myself when building DoY, I found it to be slow a=
nd error-prone.

So, the solution is to place the udf code in the new =E2=80=9Csite=E2=80=9D=
 directory: $DRILL_SITE/jars. That=E2=80=99s what that is for. Then, let Do=
Y automatically distribute the code to every node. Perfect! Except that it =
does not work to dynamically distribute code after Drill starts.

For DoY, the solution requirements are:

1. Distribute code using Drill itself, rather than manually copying jars to=
 (unknown) Drill directories.
2. Ensure the solution works even if another Drillbit is spun up later, and=
 uses the original Drill tar file.

I=E2=80=99m thinking we want to leverage DFS: place udf files into a well-k=
nown DFS directory. Register the udf into, say, ZK. When a new Drillbit sta=
rts, it looks for new udf jars in ZK, copies the file to a temporary locati=
on, and launches. An existing Drill is notified of the change and does the =
same download process. Clean-up is needed at some point to remove ZK entrie=
s if the udf jar becomes statically available on the next launch. That need=
s more thought.

We=E2=80=99d still need the phases mentioned earlier to ensure consistency.

Suggestions anyone as to how to do this super simply & still get it to work=
 with DoY?

Thanks,

- Paul
=20
> On Jun 20, 2016, at 7:18 PM, Neeraja Rentachintala <nrentachintala@maprte=
ch.com> wrote:
>=20
> This will need to work with YARN (Once Drill is YARN enabled, I would
> expect a lot of users using it in conjunction with YARN).
> Paul, I am not clear why this wouldn't work with YARN. Can you elaborate.
>=20
> -Neeraja
>=20
> On Mon, Jun 20, 2016 at 7:01 PM, Paul Rogers <progers@maprtech.com> wrote=
:
>=20
>> Good enough, as long as we document the limitation that this feature can=
=E2=80=99t
>> work with YARN deployment as users generally do not have access to the
>> temporary =E2=80=9Clocalization=E2=80=9D directories where the Drill cod=
e is placed by YARN.
>>=20
>> Note that the jar distribution race condition issue occurs with the
>> proposed design: I believe I sketched out a scenario in one of the earli=
er
>> comments. Drillbit A receives the CREATE FUNCTION command. It tells
>> Drillbit B. While informing the other Drillbits, Drillbit B plans and
>> launches a query that uses the function. Drillbit Z starts execution of =
the
>> query before it learns from A about the new function. This will be rare =
=E2=80=94
>> just rare enough to create very hard to reproduce bugs.
>>=20
>> The only reliable solution is to do the work in multiple passes:
>>=20
>> Pass 1: Ask each node to load the function, but not make it available to
>> the planner. (it would be available to the execution engine.)
>> Pass 2: Await confirmation from each node that this is done.
>> Pass 3: Alert every node that it is now free to plan queries with the
>> function.
>>=20
>> Finally, I wonder if we should design the SQL syntax based on a long-ter=
m
>> design, even if the feature itself is a short-term work-around. Changing
>> the syntax later might break scripts that users might write.
>>=20
>> So, the question for the group is this: is the value of semi-complete
>> feature sufficient to justify the potential problems?
>>=20
>> - Paul
>>=20
>>> On Jun 20, 2016, at 6:15 PM, Parth Chandra <pchandra@maprtech.com>
>> wrote:
>>>=20
>>> Moving discussion to dev.
>>>=20
>>> I believe the aim is to do a simple implementation without the complexi=
ty
>>> of distributing the UDF. I think the document should make this limitati=
on
>>> clear.
>>>=20
>>> Per Paul's point on there being a simpler solution of just having each
>>> drillbit detect the if a UDF is present, I think the problem is if a UD=
F
>>> get's deployed to some but not all drillbits. A query can then start
>>> executing but not run successfully. The intent of the create commands
>> would
>>> be to ensure that all drillbits have the UDF or none would.
>>>=20
>>> I think Jacques' point about ownership conflicts is not addressed
>> clearly.
>>> Also, the unloading is not clear. The delete command should probably
>> remove
>>> the UDF and unload it.
>>>=20
>>>=20
>>> On Fri, Jun 17, 2016 at 11:19 AM, Paul Rogers <progers@maprtech.com>
>> wrote:
>>>=20
>>>> Reviewed the spec; many comments posted. Three primary comments for th=
e
>>>> community to consider.
>>>>=20
>>>> 1. The design conflicts with the Drill-on-YARN project. Is this a
>> specific
>>>> fix for one unique problem, or is it worth expanding the solution to
>> work
>>>> with Drill-on-YARN deployments? Might be hard to make the two work
>> together
>>>> later. See comments in docs for details.
>>>>=20
>>>> 2. Have we, by chance, looked at how other projects handle code
>>>> distribution? Spark, Storm and others automatically deploy code across
>> the
>>>> cluster; no manual distribution to each node. The key difference betwe=
en
>>>> Drill and others is that, for Storm, say, code is associated with a jo=
b
>>>> (=E2=80=9Ctopology=E2=80=9D in Storm terms.) But, in Drill, functions =
are global and
>> have
>>>> no obvious life cycle that suggests when the code can be unloaded.
>>>>=20
>>>> 3. Have considered the class loader, dependency and name space isolati=
on
>>>> issues addressed by such products as Tomcat (web apps) or Eclipse
>>>> (plugins)? Putting user code in the same namespace as Drill code=C2=A0=
 is
>> quick
>>>> & dirty. It turns out, however, that doing so leads to problems that
>>>> require long, frustrating debugging sessions to resolve.
>>>>=20
>>>> Addressing item 1 might expand scope a bit. Addressing items 2 and 3
>> are a
>>>> big increase in scope, so I won=E2=80=99t be surprised if we leave tho=
se issues
>> for
>>>> later. (Though, addressing item 2 might be the best way to address ite=
m
>> 1.)
>>>>=20
>>>> If we want a very simple solution that requires minimal change, perhap=
s
>> we
>>>> can use an even simpler solution. In the proposed design, the user sti=
ll
>>>> must distribute code to all the nodes. The primary change is to tell
>> Drill
>>>> to load (or unload) that code. Can accomplish the same result easier
>> simply
>>>> by having Drill periodically scan certain directories looking for new
>> (or
>>>> removed) jars? Still won=E2=80=99t work with YARN, or solve the name s=
pace
>> issues,
>>>> but will work for existing non-YARN Drill users without new SQL syntax=
.
>>>>=20
>>>> Thanks,
>>>>=20
>>>> - Paul
>>>>=20
>>>>> On Jun 16, 2016, at 2:07 PM, Jacques Nadeau <jacques@dremio.com>
>> wrote:
>>>>>=20
>>>>> Two quick thoughts:
>>>>>=20
>>>>> - (user) In the design document I didn't see any discussion of
>>>>> ownership/conflicts or unloading. Would be helpful to see the thinkin=
g
>>>> there
>>>>> - (dev) There is a row oriented facade via the
>>>>> FieldReader/FieldWriter/ComplexWriter classes. That would be a good
>> place
>>>>> to start when trying to implement an alternative interface.
>>>>>=20
>>>>>=20
>>>>> --
>>>>> Jacques Nadeau
>>>>> CTO and Co-Founder, Dremio
>>>>>=20
>>>>> On Thu, Jun 16, 2016 at 11:32 AM, John Omernik <john@omernik.com>
>> wrote:
>>>>>=20
>>>>>> Honestly, I don't see it as a priority issue. I think some of the
>> ideas
>>>>>> around community java UDFs could be a better approach. I'd hate to
>> take
>>>>>> away from other work to hack in something like this.
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> On Thu, Jun 16, 2016 at 1:19 PM, Paul Rogers <progers@maprtech.com>
>>>> wrote:
>>>>>>=20
>>>>>>> Ted refers to source code transformation. Drill gains its speed fro=
m
>>>>>> value
>>>>>>> vectors. However, VVs are a far cry from the row-based interface th=
at
>>>>>> most
>>>>>>> mere mortals are accustomed to using. Since VVs are very type
>> specific,
>>>>>>> code is typically generated to handle the specifics of each type.
>>>>>> Accessing
>>>>>>> VVs in Jython may be a bit of a challenge because of the "impedence
>>>>>>> mismatch" between how VVs work and the row-and-column view expected
>> by
>>>>>> most
>>>>>>> (non-Drill) developers.
>>>>>>>=20
>>>>>>> I wonder if we've considered providing a row-oriented "facade" that
>> can
>>>>>> be
>>>>>>> used by roll-your own data sources and user-defined row transforms?
>>>> Might
>>>>>>> be a hiccup in the fast VV pipeline, but might be handy for users
>>>> willing
>>>>>>> to trade a bit of speed for convenience. With such a facade, the
>> Jython
>>>>>> row
>>>>>>> transforms that John mentions could be quite simple.
>>>>>>>=20
>>>>>>> On Thu, Jun 16, 2016 at 10:36 AM, Ted Dunning <ted.dunning@gmail.co=
m
>>>=20
>>>>>>> wrote:
>>>>>>>=20
>>>>>>>> Since UDF's use source code transformation, using Jython would be
>>>>>>>> difficult.
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> On Thu, Jun 16, 2016 at 9:42 AM, Arina Yelchiyeva <
>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>>>>>=20
>>>>>>>>> Hi Charles,
>>>>>>>>>=20
>>>>>>>>> not that I am aware of. Proposed solution doesn't invent anything
>>>>>> new,
>>>>>>>> just
>>>>>>>>> adds possibility to add UDFs without drillbit restart. But
>>>>>>> contributions
>>>>>>>>> are welcomed.
>>>>>>>>>=20
>>>>>>>>> On Thu, Jun 16, 2016 at 4:52 PM Charles Givre <cgivre@gmail.com>
>>>>>>> wrote:
>>>>>>>>>=20
>>>>>>>>>> Arina,
>>>>>>>>>> Has there been any discussion about making it possible via Jytho=
n
>>>>>> or
>>>>>>>>>> something for users to write simple UDFs in Python?
>>>>>>>>>> My ideal would be to have this capability integrated in the web
>> GUI
>>>>>>>> such
>>>>>>>>>> that a user could write their UDF (in Python) right there, submi=
t
>>>>>> it
>>>>>>>> and
>>>>>>>>> it
>>>>>>>>>> would be deployed to Drill if it passes validation tests.
>>>>>>>>>> =E2=80=94C
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>> On Jun 16, 2016, at 09:34, Arina Yelchiyeva <
>>>>>>>>> arina.yelchiyeva@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>> Hi all!
>>>>>>>>>>>=20
>>>>>>>>>>> I have created Jira to allow dynamic UDFs support in Drill (
>>>>>>>>>>> https://issues.apache.org/jira/browse/DRILL-4726). There is a
>>>>>> link
>>>>>>>> to
>>>>>>>>>>> design document in Jira description.
>>>>>>>>>>> Comments or suggestions are welcomed.
>>>>>>>>>>>=20
>>>>>>>>>>> Kind regards
>>>>>>>>>>> Arina
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>=20
>>>>>>=20
>>>>=20
>>>>=20
>>=20
>>=20


------=_Part_1895281_1885777437.1466525700362--