drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Feinauer <julian.feina...@web.de>
Subject Re: UDF is not recognized by drill / Validation Error
Date Fri, 20 May 2016 18:28:38 GMT
Hey Abdel,

I renamed it due to your previous hint with the static subclass.
But the SQL Commands still returns:
0: jdbc:drill:> SELECT myintadd(CAST(position_id AS int),CAST(store_id AS int)) FROM cp.`employee.json`;
Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 63: No match found for function
signature myintadd(<NUMERIC>, <NUMERIC>)

So this has to be something wrong with my drill setup?

All I do is to start my drillbit with drillbit.sh start (or restart) and then start drill-conf
where the conf/drill-override.conf contains:
drill.exec: {
  cluster-id: "drillbits1",
  zk.connect: "localhost:2181"
}

There’s one other thing that seems strange to me with my setup:
Although I set the loglevel in conf/logback.xml to trace the drillbit.log file does not show
any trace outputs.

Am I doing something wrong with my setup?

Thanks
Julian

> Am 20.05.2016 um 20:13 schrieb Abdel Hakim Deneche <adeneche@maprtech.com>:
> 
> You defined your function as myintadd:
> 
> 
> @FunctionTemplate(name = "*myintadd*", scope ...
> 
> 
> The following query worked fine for me:
> 
> SELECT *myintadd*(CAST(position_id AS int),CAST(store_id AS int)) FROM
>> cp.`employee.json`;
> 
> 
> Thanks
> 
> 
> 
> 
> 
> On Fri, May 20, 2016 at 10:17 AM, Julian Feinauer <julian.feinauer@web.de>
> wrote:
> 
>> Dear Jacques,
>> Dear Abdel,
>> 
>> thanks for your kind help.
>> 
>> jar tf drillUDF-1.0-SNAPSHOT.jar returns:
>> 
>> META-INF/
>> META-INF/MANIFEST.MF
>> org/
>> org/apache/
>> org/apache/drill/
>> org/apache/drill/contrib/
>> org/apache/drill/contrib/function/
>> org/julian/
>> drill-module.conf
>> IntIntAdd.class
>> org/apache/drill/contrib/function/IntIntAdd.class
>> org/julian/IntIntAdd$myintadd.class
>> org/julian/IntIntAdd.class
>> org/julian/IntIntAdd.java
>> META-INF/maven/
>> META-INF/maven/julian.test/
>> META-INF/maven/julian.test/drillUDF/
>> META-INF/maven/julian.test/drillUDF/pom.xml
>> META-INF/maven/julian.test/drillUDF/pom.properties
>> 
>> And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns
>> META-INF/
>> META-INF/MANIFEST.MF
>> org/
>> org/apache/
>> org/apache/drill/
>> org/apache/drill/contrib/
>> org/apache/drill/contrib/function/
>> org/julian/
>> org/julian/IntIntAdd.java
>> drill-module.conf
>> 
>> so the drill-module.conf is in both roots.
>> 
>> @Abdel:
>> How or where can I share the jar files with you?
>> 
>> Best regards
>> Julian
>> 
>>> Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <jacques@dremio.com>:
>>> 
>>> Can you run  jar tf myudf.jar against your jar files? Since Drill is not
>>> detecting the jar file, we need to resolve that first. The
>>> drill-module.conf must be in the root of each jar file that should be
>>> included. Lets start by verifying that.
>>> 
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>> 
>>> On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <julian.feinauer@web.de
>>> 
>>> wrote:
>>> 
>>>> Dear all,
>>>> 
>>>> thank you very much for all your replies.
>>>> I tried everything but it is still not working.
>>>> 
>>>> 1. I copy both files (classes and sources) in the /jars/3rdparty
>> directory
>>>> 2. I restarted the drillbit after this (I use only one drillbit and
>>>> drill-conf both running on my local machine)
>>>> 3. I changed the class to a static subclass
>>>> 4. I have the drill-module.conf in my ressources
>>>> 4. The error appears in the drillbit.log as soon as i call the udf
>> because
>>>> it is not recognized by drill.
>>>> On startup drillbit.log states all the packages and jars that are
>> scanned
>>>> and my custom jar is not listed there.
>>>> Therefore I think it is a problem with the class loader or something
>>>> related?
>>>> 
>>>> Could this be possible?
>>>> 
>>>> Greetings
>>>> Julian
>>>> 
>>>>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tugdual@gmail.com>:
>>>>> 
>>>>> Hi
>>>>> 
>>>>> Be sure you deploy on each nodes, 2 jars:
>>>>> - the jar containing the classes
>>>>> - the jar contains the sources
>>>>> 
>>>>> The POM.xml in the simple function examples contains the maven
>>>>> configuration to generate these 2 files, be sure you have the same in
>>>> your
>>>>> project:
>>>>> 
>>>> 
>> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
>>>>> 
>>>>> and you have restarted the drillbit
>>>>> 
>>>>> Regards
>>>>> Tug
>>>>> @tgrall
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
>>>> adeneche@maprtech.com
>>>>>> wrote:
>>>>> 
>>>>>> the example I gave you was incomplete, here is what I meant to send:
>>>>>> 
>>>>>> public class MyUDF {
>>>>>> 
>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>> FunctionScope.SIMPLE, nulls =
>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>> public *static *class IntIntAdd implements DrillSimpleFunc {
>>>>>>    ...
>>>>>> }
>>>>>> 
>>>>>> }
>>>>>> 
>>>>>> 
>>>>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
>>>>>> adeneche@maprtech.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hey Julian,
>>>>>>> 
>>>>>>> one more thing you could try out: declare the UDF as a static
class
>>>>>> inside
>>>>>>> another class:
>>>>>>> 
>>>>>>> public class MyUDF {
>>>>>>> 
>>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>>> FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
>>>>>>> NULL_IF_NULL)
>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>>>    ...
>>>>>>> }
>>>>>>> 
>>>>>>> }
>>>>>>> 
>>>>>>> Take a look at the following page to see an examples of UDFs:
>>>>>>> http://drill.apache.org/docs/custom-function-interfaces/
>>>>>>> 
>>>>>>> If this doesn't work check the drillbit log, it should print
an error
>>>>>>> message when it's starting up if something's wrong with your
UDF.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
>>>> julian.feinauer@web.de
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Dear folks,
>>>>>>>> 
>>>>>>>> I’m currently experimenting with user defined functions
in drill but
>>>> I’m
>>>>>>>> not able to get them to work on my drillbits.
>>>>>>>> I always get the error: Error: VALIDATION ERROR: From line
1,
>> column 8
>>>>>> to
>>>>>>>> line 1, column 41: No match found for function signature
>>>>>> myaddints(<ANY>,
>>>>>>>> <ANY>).
>>>>>>>> 
>>>>>>>> I already went through all the tips I found in the mailing
list.
>>>>>>>> The jar contains a drill-module.conf with the content:
>>>>>>>> drill.classpath.scanning.packages += "org.julian"
>>>>>>>> And the UDF is defined as:
>>>>>>>> package org.julian;
>>>>>>>> 
>>>>>>>> import ...
>>>>>>>> 
>>>>>>>> @FunctionTemplate(name = „myaddints", scope =
>>>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>>>> 
>>>>>>>>  @Param
>>>>>>>>  IntHolder in1;
>>>>>>>> 
>>>>>>>>  @Param
>>>>>>>>  IntHolder in2;
>>>>>>>> 
>>>>>>>>  @Output
>>>>>>>>  IntHolder out;
>>>>>>>> 
>>>>>>>>  public void setup() {};
>>>>>>>> 
>>>>>>>>  public void eval() {
>>>>>>>>      out.value = in1.value + in2.value;
>>>>>>>>  }
>>>>>>>> 
>>>>>>>> }
>>>>>>>> I tried to find the bug using the logs and encountered the
following
>>>>>>>> messages from the drillbit.log:
>>>>>>>> Base Configuration:
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
>>>>>>>> 
>>>>>>>> Intermediate Configuration and Plugin files, in order of
precedence:
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
>>>>>>>> 
>>>>>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar)
is
>>>> not
>>>>>>>> scanned altough it is located in jars/3rdparty.
>>>>>>>> 
>>>>>>>> Am I doing something wrong or am I missing something or has
anyone a
>>>>>> good
>>>>>>>> hint for me?
>>>>>>>> 
>>>>>>>> Thank you already!
>>>>>>>> Julian
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> 
>>>>>>> Abdelhakim Deneche
>>>>>>> 
>>>>>>> Software Engineer
>>>>>>> 
>>>>>>> <http://www.mapr.com/>
>>>>>>> 
>>>>>>> 
>>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>>> <
>>>>>> 
>>>> 
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> 
>>>>>> Abdelhakim Deneche
>>>>>> 
>>>>>> Software Engineer
>>>>>> 
>>>>>> <http://www.mapr.com/>
>>>>>> 
>>>>>> 
>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>> <
>>>>>> 
>>>> 
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 
> 
> 
> -- 
> 
> Abdelhakim Deneche
> 
> Software Engineer
> 
>  <http://www.mapr.com/>
> 
> 
> Now Available - Free Hadoop On-Demand Training
> <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message