drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdel Hakim Deneche <adene...@maprtech.com>
Subject Re: UDF is not recognized by drill / Validation Error
Date Fri, 20 May 2016 18:13:59 GMT
You defined your function as myintadd:


@FunctionTemplate(name = "*myintadd*", scope ...


The following query worked fine for me:

SELECT *myintadd*(CAST(position_id AS int),CAST(store_id AS int)) FROM
> cp.`employee.json`;


Thanks





On Fri, May 20, 2016 at 10:17 AM, Julian Feinauer <julian.feinauer@web.de>
wrote:

> Dear Jacques,
> Dear Abdel,
>
> thanks for your kind help.
>
> jar tf drillUDF-1.0-SNAPSHOT.jar returns:
>
> META-INF/
> META-INF/MANIFEST.MF
> org/
> org/apache/
> org/apache/drill/
> org/apache/drill/contrib/
> org/apache/drill/contrib/function/
> org/julian/
> drill-module.conf
> IntIntAdd.class
> org/apache/drill/contrib/function/IntIntAdd.class
> org/julian/IntIntAdd$myintadd.class
> org/julian/IntIntAdd.class
> org/julian/IntIntAdd.java
> META-INF/maven/
> META-INF/maven/julian.test/
> META-INF/maven/julian.test/drillUDF/
> META-INF/maven/julian.test/drillUDF/pom.xml
> META-INF/maven/julian.test/drillUDF/pom.properties
>
> And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns
> META-INF/
> META-INF/MANIFEST.MF
> org/
> org/apache/
> org/apache/drill/
> org/apache/drill/contrib/
> org/apache/drill/contrib/function/
> org/julian/
> org/julian/IntIntAdd.java
> drill-module.conf
>
> so the drill-module.conf is in both roots.
>
> @Abdel:
> How or where can I share the jar files with you?
>
> Best regards
> Julian
>
> > Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <jacques@dremio.com>:
> >
> > Can you run  jar tf myudf.jar against your jar files? Since Drill is not
> > detecting the jar file, we need to resolve that first. The
> > drill-module.conf must be in the root of each jar file that should be
> > included. Lets start by verifying that.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <julian.feinauer@web.de
> >
> > wrote:
> >
> >> Dear all,
> >>
> >> thank you very much for all your replies.
> >> I tried everything but it is still not working.
> >>
> >> 1. I copy both files (classes and sources) in the /jars/3rdparty
> directory
> >> 2. I restarted the drillbit after this (I use only one drillbit and
> >> drill-conf both running on my local machine)
> >> 3. I changed the class to a static subclass
> >> 4. I have the drill-module.conf in my ressources
> >> 4. The error appears in the drillbit.log as soon as i call the udf
> because
> >> it is not recognized by drill.
> >> On startup drillbit.log states all the packages and jars that are
> scanned
> >> and my custom jar is not listed there.
> >> Therefore I think it is a problem with the class loader or something
> >> related?
> >>
> >> Could this be possible?
> >>
> >> Greetings
> >> Julian
> >>
> >>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tugdual@gmail.com>:
> >>>
> >>> Hi
> >>>
> >>> Be sure you deploy on each nodes, 2 jars:
> >>> - the jar containing the classes
> >>> - the jar contains the sources
> >>>
> >>> The POM.xml in the simple function examples contains the maven
> >>> configuration to generate these 2 files, be sure you have the same in
> >> your
> >>> project:
> >>>
> >>
> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
> >>>
> >>> and you have restarted the drillbit
> >>>
> >>> Regards
> >>> Tug
> >>> @tgrall
> >>>
> >>>
> >>>
> >>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
> >> adeneche@maprtech.com
> >>>> wrote:
> >>>
> >>>> the example I gave you was incomplete, here is what I meant to send:
> >>>>
> >>>> public class MyUDF {
> >>>>
> >>>>  @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >>>>  FunctionScope.SIMPLE, nulls =
> >>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >>>>  public *static *class IntIntAdd implements DrillSimpleFunc {
> >>>>     ...
> >>>>  }
> >>>>
> >>>> }
> >>>>
> >>>>
> >>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
> >>>> adeneche@maprtech.com>
> >>>> wrote:
> >>>>
> >>>>> Hey Julian,
> >>>>>
> >>>>> one more thing you could try out: declare the UDF as a static class
> >>>> inside
> >>>>> another class:
> >>>>>
> >>>>> public class MyUDF {
> >>>>>
> >>>>>  @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >>>>>  FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
> >>>>> NULL_IF_NULL)
> >>>>>  public class IntIntAdd implements DrillSimpleFunc {
> >>>>>     ...
> >>>>>  }
> >>>>>
> >>>>> }
> >>>>>
> >>>>> Take a look at the following page to see an examples of UDFs:
> >>>>> http://drill.apache.org/docs/custom-function-interfaces/
> >>>>>
> >>>>> If this doesn't work check the drillbit log, it should print an
error
> >>>>> message when it's starting up if something's wrong with your UDF.
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>>
> >>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
> >> julian.feinauer@web.de
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Dear folks,
> >>>>>>
> >>>>>> I’m currently experimenting with user defined functions in
drill but
> >> I’m
> >>>>>> not able to get them to work on my drillbits.
> >>>>>> I always get the error: Error: VALIDATION ERROR: From line 1,
> column 8
> >>>> to
> >>>>>> line 1, column 41: No match found for function signature
> >>>> myaddints(<ANY>,
> >>>>>> <ANY>).
> >>>>>>
> >>>>>> I already went through all the tips I found in the mailing list.
> >>>>>> The jar contains a drill-module.conf with the content:
> >>>>>> drill.classpath.scanning.packages += "org.julian"
> >>>>>> And the UDF is defined as:
> >>>>>> package org.julian;
> >>>>>>
> >>>>>> import ...
> >>>>>>
> >>>>>> @FunctionTemplate(name = „myaddints", scope =
> >>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
> >>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >>>>>> public class IntIntAdd implements DrillSimpleFunc {
> >>>>>>
> >>>>>>   @Param
> >>>>>>   IntHolder in1;
> >>>>>>
> >>>>>>   @Param
> >>>>>>   IntHolder in2;
> >>>>>>
> >>>>>>   @Output
> >>>>>>   IntHolder out;
> >>>>>>
> >>>>>>   public void setup() {};
> >>>>>>
> >>>>>>   public void eval() {
> >>>>>>       out.value = in1.value + in2.value;
> >>>>>>   }
> >>>>>>
> >>>>>> }
> >>>>>> I tried to find the bug using the logs and encountered the following
> >>>>>> messages from the drillbit.log:
> >>>>>> Base Configuration:
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
> >>>>>>
> >>>>>> Intermediate Configuration and Plugin files, in order of precedence:
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
> >>>>>>
> >>>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar)
is
> >> not
> >>>>>> scanned altough it is located in jars/3rdparty.
> >>>>>>
> >>>>>> Am I doing something wrong or am I missing something or has
anyone a
> >>>> good
> >>>>>> hint for me?
> >>>>>>
> >>>>>> Thank you already!
> >>>>>> Julian
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Abdelhakim Deneche
> >>>>>
> >>>>> Software Engineer
> >>>>>
> >>>>> <http://www.mapr.com/>
> >>>>>
> >>>>>
> >>>>> Now Available - Free Hadoop On-Demand Training
> >>>>> <
> >>>>
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Abdelhakim Deneche
> >>>>
> >>>> Software Engineer
> >>>>
> >>>> <http://www.mapr.com/>
> >>>>
> >>>>
> >>>> Now Available - Free Hadoop On-Demand Training
> >>>> <
> >>>>
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>>>
> >>>>
> >>
> >>
>
>


-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message