drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Feinauer <julian.feina...@web.de>
Subject Re: UDF is not recognized by drill / Validation Error
Date Sat, 21 May 2016 06:25:12 GMT
Hey Guys,

I finally found the solution of my weird problem.
I’m using OS X and besides the drill installation according to the documentation I also
had drill-embedded installed via homebrew (package apache-drill).
The homebrew package seemed to interact in a weird way with my „regular“ installation
because all the problems (correct log level from logback appeared in the drillbit.log) as
well as the UDF problem suddenly disappeared.

Nonetheless, thank you all for your kind help and your efforts!
Julian

> Am 20.05.2016 um 20:38 schrieb Julian Feinauer <julian.feinauer@web.de>:
> 
> Hey,
> 
> I’m using Drill 1.6.0.
> 
>> Am 20.05.2016 um 20:37 schrieb Abdel Hakim Deneche <adeneche@maprtech.com>:
>> 
>> What version of Drill are you using ?
>> 
>> On Fri, May 20, 2016 at 11:28 AM, Julian Feinauer <julian.feinauer@web.de>
>> wrote:
>> 
>>> Hey Abdel,
>>> 
>>> I renamed it due to your previous hint with the static subclass.
>>> But the SQL Commands still returns:
>>> 0: jdbc:drill:> SELECT myintadd(CAST(position_id AS int),CAST(store_id AS
>>> int)) FROM cp.`employee.json`;
>>> Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 63: No
>>> match found for function signature myintadd(<NUMERIC>, <NUMERIC>)
>>> 
>>> So this has to be something wrong with my drill setup?
>>> 
>>> All I do is to start my drillbit with drillbit.sh start (or restart) and
>>> then start drill-conf where the conf/drill-override.conf contains:
>>> drill.exec: {
>>> cluster-id: "drillbits1",
>>> zk.connect: "localhost:2181"
>>> }
>>> 
>>> There’s one other thing that seems strange to me with my setup:
>>> Although I set the loglevel in conf/logback.xml to trace the drillbit.log
>>> file does not show any trace outputs.
>>> 
>>> Am I doing something wrong with my setup?
>>> 
>>> Thanks
>>> Julian
>>> 
>>>> Am 20.05.2016 um 20:13 schrieb Abdel Hakim Deneche <
>>> adeneche@maprtech.com>:
>>>> 
>>>> You defined your function as myintadd:
>>>> 
>>>> 
>>>> @FunctionTemplate(name = "*myintadd*", scope ...
>>>> 
>>>> 
>>>> The following query worked fine for me:
>>>> 
>>>> SELECT *myintadd*(CAST(position_id AS int),CAST(store_id AS int)) FROM
>>>>> cp.`employee.json`;
>>>> 
>>>> 
>>>> Thanks
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Fri, May 20, 2016 at 10:17 AM, Julian Feinauer <
>>> julian.feinauer@web.de>
>>>> wrote:
>>>> 
>>>>> Dear Jacques,
>>>>> Dear Abdel,
>>>>> 
>>>>> thanks for your kind help.
>>>>> 
>>>>> jar tf drillUDF-1.0-SNAPSHOT.jar returns:
>>>>> 
>>>>> META-INF/
>>>>> META-INF/MANIFEST.MF
>>>>> org/
>>>>> org/apache/
>>>>> org/apache/drill/
>>>>> org/apache/drill/contrib/
>>>>> org/apache/drill/contrib/function/
>>>>> org/julian/
>>>>> drill-module.conf
>>>>> IntIntAdd.class
>>>>> org/apache/drill/contrib/function/IntIntAdd.class
>>>>> org/julian/IntIntAdd$myintadd.class
>>>>> org/julian/IntIntAdd.class
>>>>> org/julian/IntIntAdd.java
>>>>> META-INF/maven/
>>>>> META-INF/maven/julian.test/
>>>>> META-INF/maven/julian.test/drillUDF/
>>>>> META-INF/maven/julian.test/drillUDF/pom.xml
>>>>> META-INF/maven/julian.test/drillUDF/pom.properties
>>>>> 
>>>>> And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns
>>>>> META-INF/
>>>>> META-INF/MANIFEST.MF
>>>>> org/
>>>>> org/apache/
>>>>> org/apache/drill/
>>>>> org/apache/drill/contrib/
>>>>> org/apache/drill/contrib/function/
>>>>> org/julian/
>>>>> org/julian/IntIntAdd.java
>>>>> drill-module.conf
>>>>> 
>>>>> so the drill-module.conf is in both roots.
>>>>> 
>>>>> @Abdel:
>>>>> How or where can I share the jar files with you?
>>>>> 
>>>>> Best regards
>>>>> Julian
>>>>> 
>>>>>> Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <jacques@dremio.com>:
>>>>>> 
>>>>>> Can you run  jar tf myudf.jar against your jar files? Since Drill
is
>>> not
>>>>>> detecting the jar file, we need to resolve that first. The
>>>>>> drill-module.conf must be in the root of each jar file that should
be
>>>>>> included. Lets start by verifying that.
>>>>>> 
>>>>>> --
>>>>>> Jacques Nadeau
>>>>>> CTO and Co-Founder, Dremio
>>>>>> 
>>>>>> On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <
>>> julian.feinauer@web.de
>>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> Dear all,
>>>>>>> 
>>>>>>> thank you very much for all your replies.
>>>>>>> I tried everything but it is still not working.
>>>>>>> 
>>>>>>> 1. I copy both files (classes and sources) in the /jars/3rdparty
>>>>> directory
>>>>>>> 2. I restarted the drillbit after this (I use only one drillbit
and
>>>>>>> drill-conf both running on my local machine)
>>>>>>> 3. I changed the class to a static subclass
>>>>>>> 4. I have the drill-module.conf in my ressources
>>>>>>> 4. The error appears in the drillbit.log as soon as i call the
udf
>>>>> because
>>>>>>> it is not recognized by drill.
>>>>>>> On startup drillbit.log states all the packages and jars that
are
>>>>> scanned
>>>>>>> and my custom jar is not listed there.
>>>>>>> Therefore I think it is a problem with the class loader or something
>>>>>>> related?
>>>>>>> 
>>>>>>> Could this be possible?
>>>>>>> 
>>>>>>> Greetings
>>>>>>> Julian
>>>>>>> 
>>>>>>>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tugdual@gmail.com>:
>>>>>>>> 
>>>>>>>> Hi
>>>>>>>> 
>>>>>>>> Be sure you deploy on each nodes, 2 jars:
>>>>>>>> - the jar containing the classes
>>>>>>>> - the jar contains the sources
>>>>>>>> 
>>>>>>>> The POM.xml in the simple function examples contains the
maven
>>>>>>>> configuration to generate these 2 files, be sure you have
the same in
>>>>>>> your
>>>>>>>> project:
>>>>>>>> 
>>>>>>> 
>>>>> 
>>> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
>>>>>>>> 
>>>>>>>> and you have restarted the drillbit
>>>>>>>> 
>>>>>>>> Regards
>>>>>>>> Tug
>>>>>>>> @tgrall
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
>>>>>>> adeneche@maprtech.com
>>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> the example I gave you was incomplete, here is what I
meant to send:
>>>>>>>>> 
>>>>>>>>> public class MyUDF {
>>>>>>>>> 
>>>>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>>>>> FunctionScope.SIMPLE, nulls =
>>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>>>>> public *static *class IntIntAdd implements DrillSimpleFunc
{
>>>>>>>>>  ...
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche
<
>>>>>>>>> adeneche@maprtech.com>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hey Julian,
>>>>>>>>>> 
>>>>>>>>>> one more thing you could try out: declare the UDF
as a static class
>>>>>>>>> inside
>>>>>>>>>> another class:
>>>>>>>>>> 
>>>>>>>>>> public class MyUDF {
>>>>>>>>>> 
>>>>>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>>>>>> FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
>>>>>>>>>> NULL_IF_NULL)
>>>>>>>>>> public class IntIntAdd implements DrillSimpleFunc
{
>>>>>>>>>>  ...
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> Take a look at the following page to see an examples
of UDFs:
>>>>>>>>>> http://drill.apache.org/docs/custom-function-interfaces/
>>>>>>>>>> 
>>>>>>>>>> If this doesn't work check the drillbit log, it should
print an
>>> error
>>>>>>>>>> message when it's starting up if something's wrong
with your UDF.
>>>>>>>>>> 
>>>>>>>>>> Thanks
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer
<
>>>>>>> julian.feinauer@web.de
>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Dear folks,
>>>>>>>>>>> 
>>>>>>>>>>> I’m currently experimenting with user defined
functions in drill
>>> but
>>>>>>> I’m
>>>>>>>>>>> not able to get them to work on my drillbits.
>>>>>>>>>>> I always get the error: Error: VALIDATION ERROR:
From line 1,
>>>>> column 8
>>>>>>>>> to
>>>>>>>>>>> line 1, column 41: No match found for function
signature
>>>>>>>>> myaddints(<ANY>,
>>>>>>>>>>> <ANY>).
>>>>>>>>>>> 
>>>>>>>>>>> I already went through all the tips I found in
the mailing list.
>>>>>>>>>>> The jar contains a drill-module.conf with the
content:
>>>>>>>>>>> drill.classpath.scanning.packages += "org.julian"
>>>>>>>>>>> And the UDF is defined as:
>>>>>>>>>>> package org.julian;
>>>>>>>>>>> 
>>>>>>>>>>> import ...
>>>>>>>>>>> 
>>>>>>>>>>> @FunctionTemplate(name = „myaddints", scope
=
>>>>>>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls
=
>>>>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>>>>>>> public class IntIntAdd implements DrillSimpleFunc
{
>>>>>>>>>>> 
>>>>>>>>>>> @Param
>>>>>>>>>>> IntHolder in1;
>>>>>>>>>>> 
>>>>>>>>>>> @Param
>>>>>>>>>>> IntHolder in2;
>>>>>>>>>>> 
>>>>>>>>>>> @Output
>>>>>>>>>>> IntHolder out;
>>>>>>>>>>> 
>>>>>>>>>>> public void setup() {};
>>>>>>>>>>> 
>>>>>>>>>>> public void eval() {
>>>>>>>>>>>    out.value = in1.value + in2.value;
>>>>>>>>>>> }
>>>>>>>>>>> 
>>>>>>>>>>> }
>>>>>>>>>>> I tried to find the bug using the logs and encountered
the
>>> following
>>>>>>>>>>> messages from the drillbit.log:
>>>>>>>>>>> Base Configuration:
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
>>>>>>>>>>> 
>>>>>>>>>>> Intermediate Configuration and Plugin files,
in order of
>>> precedence:
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
>>>>>>>>>>> 
>>>>>>>>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar)
>>> is
>>>>>>> not
>>>>>>>>>>> scanned altough it is located in jars/3rdparty.
>>>>>>>>>>> 
>>>>>>>>>>> Am I doing something wrong or am I missing something
or has
>>> anyone a
>>>>>>>>> good
>>>>>>>>>>> hint for me?
>>>>>>>>>>> 
>>>>>>>>>>> Thank you already!
>>>>>>>>>>> Julian
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> 
>>>>>>>>>> Abdelhakim Deneche
>>>>>>>>>> 
>>>>>>>>>> Software Engineer
>>>>>>>>>> 
>>>>>>>>>> <http://www.mapr.com/>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>>>>>> <
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> Abdelhakim Deneche
>>>>>>>>> 
>>>>>>>>> Software Engineer
>>>>>>>>> 
>>>>>>>>> <http://www.mapr.com/>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>>>>> <
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> 
>>>> Abdelhakim Deneche
>>>> 
>>>> Software Engineer
>>>> 
>>>> <http://www.mapr.com/>
>>>> 
>>>> 
>>>> Now Available - Free Hadoop On-Demand Training
>>>> <
>>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>> 
>>> 
>>> 
>> 
>> 
>> -- 
>> 
>> Abdelhakim Deneche
>> 
>> Software Engineer
>> 
>> <http://www.mapr.com/>
>> 
>> 
>> Now Available - Free Hadoop On-Demand Training
>> <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
> 


Mime
View raw message