Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
From: Julian Feinauer <julian.feinauer@web.de>
Content-Type: multipart/alternative; boundary="Apple-Mail=_9506906B-0AC0-4B47-A464-F6AC22AAFBE3"
Message-Id: <1E0C640E-CFB6-4478-ACA8-C1BEF9056B03@web.de>
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
Subject: Re: UDF is not recognized by drill / Validation Error
Date: Fri, 20 May 2016 20:28:38 +0200
References: <A88F6650-7C60-4917-ACC6-AC22FBB6A81A@web.de> <CAKTYAC8s4as9=j6bh5xufMQhyy9zrY3+0wiDBoZtZNwjdyXKPA@mail.gmail.com> <CAKTYAC854e1OV-dUH4xw=XmaQMtmxgg-z7-YDEqgRFgLB_KYiA@mail.gmail.com> <CA+HHcWucTwq_MnBiQe76zry-XKofNCP3cP7c91nqYjPNjjaurQ@mail.gmail.com> <7A9EB36A-6E87-4256-8030-AE7B11026B18@web.de> <CAJrw0OQpb9gjKzdwh8fRPQTmeo7erP+7QafSHOi9207ncuPv2g@mail.gmail.com> <D371480A-C973-4F57-85C6-1DD3F83F01A7@web.de> <CAKTYAC9VtpnUKi+6GK2hJ8foRqdoyupvV_y=iS7rk5FfZ_ZEew@mail.gmail.com>
To: dev@drill.apache.org
In-Reply-To: <CAKTYAC9VtpnUKi+6GK2hJ8foRqdoyupvV_y=iS7rk5FfZ_ZEew@mail.gmail.com>
archived-at: Fri, 20 May 2016 18:28:55 -0000

--Apple-Mail=_9506906B-0AC0-4B47-A464-F6AC22AAFBE3
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hey Abdel,

I renamed it due to your previous hint with the static subclass.
But the SQL Commands still returns:
0: jdbc:drill:> SELECT myintadd(CAST(position_id AS int),CAST(store_id =
AS int)) FROM cp.`employee.json`;
Error: VALIDATION ERROR: =46rom line 1, column 8 to line 1, column 63: =
No match found for function signature myintadd(<NUMERIC>, <NUMERIC>)

So this has to be something wrong with my drill setup?

All I do is to start my drillbit with drillbit.sh start (or restart) and =
then start drill-conf where the conf/drill-override.conf contains:
drill.exec: {
  cluster-id: "drillbits1",
  zk.connect: "localhost:2181"
}

There=E2=80=99s one other thing that seems strange to me with my setup:
Although I set the loglevel in conf/logback.xml to trace the =
drillbit.log file does not show any trace outputs.

Am I doing something wrong with my setup?

Thanks
Julian

> Am 20.05.2016 um 20:13 schrieb Abdel Hakim Deneche =
<adeneche@maprtech.com>:
>=20
> You defined your function as myintadd:
>=20
>=20
> @FunctionTemplate(name =3D "*myintadd*", scope ...
>=20
>=20
> The following query worked fine for me:
>=20
> SELECT *myintadd*(CAST(position_id AS int),CAST(store_id AS int)) FROM
>> cp.`employee.json`;
>=20
>=20
> Thanks
>=20
>=20
>=20
>=20
>=20
> On Fri, May 20, 2016 at 10:17 AM, Julian Feinauer =
<julian.feinauer@web.de>
> wrote:
>=20
>> Dear Jacques,
>> Dear Abdel,
>>=20
>> thanks for your kind help.
>>=20
>> jar tf drillUDF-1.0-SNAPSHOT.jar returns:
>>=20
>> META-INF/
>> META-INF/MANIFEST.MF
>> org/
>> org/apache/
>> org/apache/drill/
>> org/apache/drill/contrib/
>> org/apache/drill/contrib/function/
>> org/julian/
>> drill-module.conf
>> IntIntAdd.class
>> org/apache/drill/contrib/function/IntIntAdd.class
>> org/julian/IntIntAdd$myintadd.class
>> org/julian/IntIntAdd.class
>> org/julian/IntIntAdd.java
>> META-INF/maven/
>> META-INF/maven/julian.test/
>> META-INF/maven/julian.test/drillUDF/
>> META-INF/maven/julian.test/drillUDF/pom.xml
>> META-INF/maven/julian.test/drillUDF/pom.properties
>>=20
>> And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns
>> META-INF/
>> META-INF/MANIFEST.MF
>> org/
>> org/apache/
>> org/apache/drill/
>> org/apache/drill/contrib/
>> org/apache/drill/contrib/function/
>> org/julian/
>> org/julian/IntIntAdd.java
>> drill-module.conf
>>=20
>> so the drill-module.conf is in both roots.
>>=20
>> @Abdel:
>> How or where can I share the jar files with you?
>>=20
>> Best regards
>> Julian
>>=20
>>> Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <jacques@dremio.com>:
>>>=20
>>> Can you run  jar tf myudf.jar against your jar files? Since Drill is =
not
>>> detecting the jar file, we need to resolve that first. The
>>> drill-module.conf must be in the root of each jar file that should =
be
>>> included. Lets start by verifying that.
>>>=20
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>=20
>>> On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer =
<julian.feinauer@web.de
>>>=20
>>> wrote:
>>>=20
>>>> Dear all,
>>>>=20
>>>> thank you very much for all your replies.
>>>> I tried everything but it is still not working.
>>>>=20
>>>> 1. I copy both files (classes and sources) in the /jars/3rdparty
>> directory
>>>> 2. I restarted the drillbit after this (I use only one drillbit and
>>>> drill-conf both running on my local machine)
>>>> 3. I changed the class to a static subclass
>>>> 4. I have the drill-module.conf in my ressources
>>>> 4. The error appears in the drillbit.log as soon as i call the udf
>> because
>>>> it is not recognized by drill.
>>>> On startup drillbit.log states all the packages and jars that are
>> scanned
>>>> and my custom jar is not listed there.
>>>> Therefore I think it is a problem with the class loader or =
something
>>>> related?
>>>>=20
>>>> Could this be possible?
>>>>=20
>>>> Greetings
>>>> Julian
>>>>=20
>>>>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tugdual@gmail.com>:
>>>>>=20
>>>>> Hi
>>>>>=20
>>>>> Be sure you deploy on each nodes, 2 jars:
>>>>> - the jar containing the classes
>>>>> - the jar contains the sources
>>>>>=20
>>>>> The POM.xml in the simple function examples contains the maven
>>>>> configuration to generate these 2 files, be sure you have the same =
in
>>>> your
>>>>> project:
>>>>>=20
>>>>=20
>> =
https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L=
24-L37
>>>>>=20
>>>>> and you have restarted the drillbit
>>>>>=20
>>>>> Regards
>>>>> Tug
>>>>> @tgrall
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
>>>> adeneche@maprtech.com
>>>>>> wrote:
>>>>>=20
>>>>>> the example I gave you was incomplete, here is what I meant to =
send:
>>>>>>=20
>>>>>> public class MyUDF {
>>>>>>=20
>>>>>> @FunctionTemplate(name =3D =E2=80=9Emyaddints", scope =3D =
FunctionTemplate.
>>>>>> FunctionScope.SIMPLE, nulls =3D
>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>> public *static *class IntIntAdd implements DrillSimpleFunc {
>>>>>>    ...
>>>>>> }
>>>>>>=20
>>>>>> }
>>>>>>=20
>>>>>>=20
>>>>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
>>>>>> adeneche@maprtech.com>
>>>>>> wrote:
>>>>>>=20
>>>>>>> Hey Julian,
>>>>>>>=20
>>>>>>> one more thing you could try out: declare the UDF as a static =
class
>>>>>> inside
>>>>>>> another class:
>>>>>>>=20
>>>>>>> public class MyUDF {
>>>>>>>=20
>>>>>>> @FunctionTemplate(name =3D =E2=80=9Emyaddints", scope =3D =
FunctionTemplate.
>>>>>>> FunctionScope.SIMPLE, nulls =3D FunctionTemplate.NullHandling.
>>>>>>> NULL_IF_NULL)
>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>>>    ...
>>>>>>> }
>>>>>>>=20
>>>>>>> }
>>>>>>>=20
>>>>>>> Take a look at the following page to see an examples of UDFs:
>>>>>>> http://drill.apache.org/docs/custom-function-interfaces/
>>>>>>>=20
>>>>>>> If this doesn't work check the drillbit log, it should print an =
error
>>>>>>> message when it's starting up if something's wrong with your =
UDF.
>>>>>>>=20
>>>>>>> Thanks
>>>>>>>=20
>>>>>>>=20
>>>>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
>>>> julian.feinauer@web.de
>>>>>>>=20
>>>>>>> wrote:
>>>>>>>=20
>>>>>>>> Dear folks,
>>>>>>>>=20
>>>>>>>> I=E2=80=99m currently experimenting with user defined functions =
in drill but
>>>> I=E2=80=99m
>>>>>>>> not able to get them to work on my drillbits.
>>>>>>>> I always get the error: Error: VALIDATION ERROR: =46rom line 1,
>> column 8
>>>>>> to
>>>>>>>> line 1, column 41: No match found for function signature
>>>>>> myaddints(<ANY>,
>>>>>>>> <ANY>).
>>>>>>>>=20
>>>>>>>> I already went through all the tips I found in the mailing =
list.
>>>>>>>> The jar contains a drill-module.conf with the content:
>>>>>>>> drill.classpath.scanning.packages +=3D "org.julian"
>>>>>>>> And the UDF is defined as:
>>>>>>>> package org.julian;
>>>>>>>>=20
>>>>>>>> import ...
>>>>>>>>=20
>>>>>>>> @FunctionTemplate(name =3D =E2=80=9Emyaddints", scope =3D
>>>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =3D
>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>>>>=20
>>>>>>>>  @Param
>>>>>>>>  IntHolder in1;
>>>>>>>>=20
>>>>>>>>  @Param
>>>>>>>>  IntHolder in2;
>>>>>>>>=20
>>>>>>>>  @Output
>>>>>>>>  IntHolder out;
>>>>>>>>=20
>>>>>>>>  public void setup() {};
>>>>>>>>=20
>>>>>>>>  public void eval() {
>>>>>>>>      out.value =3D in1.value + in2.value;
>>>>>>>>  }
>>>>>>>>=20
>>>>>>>> }
>>>>>>>> I tried to find the bug using the logs and encountered the =
following
>>>>>>>> messages from the drillbit.log:
>>>>>>>> Base Configuration:
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.=
6.0.jar!/drill-default.conf
>>>>>>>>=20
>>>>>>>> Intermediate Configuration and Plugin files, in order of =
precedence:
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0=
.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1=
.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-stor=
age-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-=
memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec=
-shaded-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-ba=
se-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-sto=
rage-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec=
-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.=
6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-h=
base-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-stor=
age-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>>=20
>>>>>>=20
>>>>=20
>> =
jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-h=
ive-core-1.6.0.jar!/drill-module.conf
>>>>>>>>=20
>>>>>>>> which let=E2=80=99s me think that my jar file =
(drillUDF-1.0-SNAPSHOT.jar) is
>>>> not
>>>>>>>> scanned altough it is located in jars/3rdparty.
>>>>>>>>=20
>>>>>>>> Am I doing something wrong or am I missing something or has =
anyone a
>>>>>> good
>>>>>>>> hint for me?
>>>>>>>>=20
>>>>>>>> Thank you already!
>>>>>>>> Julian
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>> --
>>>>>>>=20
>>>>>>> Abdelhakim Deneche
>>>>>>>=20
>>>>>>> Software Engineer
>>>>>>>=20
>>>>>>> <http://www.mapr.com/>
>>>>>>>=20
>>>>>>>=20
>>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>>> <
>>>>>>=20
>>>>=20
>> =
http://www.mapr.com/training?utm_source=3DEmail&utm_medium=3DSignature&utm=
_campaign=3DFree%20available
>>>>>>>=20
>>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> --
>>>>>>=20
>>>>>> Abdelhakim Deneche
>>>>>>=20
>>>>>> Software Engineer
>>>>>>=20
>>>>>> <http://www.mapr.com/>
>>>>>>=20
>>>>>>=20
>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>> <
>>>>>>=20
>>>>=20
>> =
http://www.mapr.com/training?utm_source=3DEmail&utm_medium=3DSignature&utm=
_campaign=3DFree%20available
>>>>>>>=20
>>>>>>=20
>>>>=20
>>>>=20
>>=20
>>=20
>=20
>=20
> --=20
>=20
> Abdelhakim Deneche
>=20
> Software Engineer
>=20
>  <http://www.mapr.com/>
>=20
>=20
> Now Available - Free Hadoop On-Demand Training
> =
<http://www.mapr.com/training?utm_source=3DEmail&utm_medium=3DSignature&ut=
m_campaign=3DFree%20available>


--Apple-Mail=_9506906B-0AC0-4B47-A464-F6AC22AAFBE3--