drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julien Le Dem <jul...@dremio.com>
Subject Re: Classpath scanning & udfs
Date Mon, 11 Jan 2016 18:26:34 GMT
You are correct:
The jar containing the UDFs should have a drill-module.conf at the root
adding your package to the property drill.classpath.scanning.packages for
scanning
drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
        my.package.containing.my.udfs
    ]
Jars that don't contain a drill-module.conf will not get scanned.



On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Thanks for your reply Jason.
>
> If we cannot override the global configuration file, then for existing
> UDF's we have to re-compile them by modifying the drill-module.conf file.
> If so our UDF's are not backward compatible. Appreciate it if someone can
> confirm this.
>
> - Rahul
>
> On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <altekrusejason@gmail.com
> >
> wrote:
>
> > Rahul,
> >
> > The error message you are seeing is in reading a storage plugin
> > configuration file. I am planning to fix these kinds of messages to
> > actually direct users at the file that is failing parsing. I have seen
> this
> > in the past when the classpath was incorrect and one of the plugins (like
> > Hbase) was not included.
> >
> > Julien can confirm, but I think this might be intentional to have the
> paths
> > read out of the modules configuration rather than the global one to save
> > time when scanning the path (rather than scanning all of the jars for all
> > paths given in the override file).
> >
> > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Before 1.2, my udfs project contained an empty drill-override.conf file
> > and
> > > I used to update the drill-override.conf on all the drillbits to
> specify
> > > the package of my UDF. This is no longer working for me. I tried a few
> > > things and below is how my drill-override.conf file looks now
> > >
> > > drill.classpath.scanning.packages :
> > ${?drill.classpath.scanning.packages} [
> > > org.apache.drill.udfs ]
> > > drill.exec: {
> > >   cluster-id: "rahul_cluster_com-drillbits",
> > >   zk.connect: "localhost:5181"
> > > }
> > >
> > > When I restart the drillbits, I get this strange error " Caused by:
> > > com.fasterxml.jackson.databind.JsonMappingException: Could not resolve
> > type
> > > id 'hbase' into a subtype of [simple type, class
> > > org.apache.drill.common.logical.StoragePluginConfig]"
> > >
> > > If I moved the package information to the drill-module.conf in my udf's
> > > project, then things are working fine. However this requires
> re-compiling
> > > the udfs which is not desirable. Is there any other way around this ?
> > >
> > > - Rahul
> > >
> >
>



-- 
Julien

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message