drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rahul challapalli <challapallira...@gmail.com>
Subject Re: Classpath scanning & udfs
Date Tue, 12 Jan 2016 17:49:21 GMT
Adding the drill-module.conf file in the drill-conf directory with the
custom package names worked for me. Thanks for the assistance Jason, Julien
and Jacques.

- Rahul

On Tue, Jan 12, 2016 at 8:09 AM, Jason Altekruse <altekrusejason@gmail.com>
wrote:

> Copying info over from Slack.
>
> For anyone who finds this thread, empty drill-module.conf files do not
> cause problems. The issue was a misunderstanding about the function of
> drill-override.conf. Values cannot be added to existing property lists
> using this file, it is designed for overriding the default values.
>
> If you put your UDFs in a package that is already being scanned as part of
> the default list then there is no need to add a new package to the list.
> The default list is generated by merging the drill-module.conf files in all
> of the default Drill packages. Here is one example from the drill exec
> module [1]
>
> To augment the list, add a drill-module.conf file to the classpath, the
> easiest way to do this is just include it in your jar.
>
> [1] -
>
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf
> <
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf
> >
>
> - Jason
>
> On Mon, Jan 11, 2016 at 11:24 AM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Sure!
> >
> > On Mon, Jan 11, 2016 at 11:06 AM, Jason Altekruse <
> > altekrusejason@gmail.com>
> > wrote:
> >
> > > The error you posted originally is from deserializing a storage plugin
> > > config, not the drill-override or drill-module files. That being said,
> we
> > > find the list of available storage plugins using classpath scanning.
> > >
> > > I don't know why the inclusion or exclusion of contents inside of a
> > > module.conf file would impact our scanning for storage plugins, but I
> > > believe this is the issue you are seeing. Somehow the package that
> > contains
> > > the hbase plugin is being removed from the list of packages to scan
> when
> > > you add the empty file.
> > >
> > > Do you want to jump on slack to chat about this?
> > >
> > > - Jason
> > >
> > > On Mon, Jan 11, 2016 at 10:52 AM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Julien,
> > > >
> > > > I have an empty drill-module.conf file in the udf jar file. Below are
> > the
> > > > contents of my global drill-override.conf file
> > > >
> > > > drill.classpath.scanning.packages :
> > > ${?drill.classpath.scanning.packages} [
> > > > org.apache.drill.udfs ]
> > > > drill.exec: {
> > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > >   zk.connect: "localhost:5181"
> > > > }
> > > >
> > > > With this, the drillbit fails to start with the error I posted in the
> > > first
> > > > email. May be I am getting the syntax wrong?
> > > >
> > > > The reason I am insisting an empty drill-module.conf file is because,
> > > udf's
> > > > developed prior to the "classpath scanning change" had an empty
> > > > drill-module.conf file and we used to override the udf package name
> in
> > > the
> > > > global conf file.
> > > >
> > > > - Rahul
> > > >
> > > > On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <julien@dremio.com>
> > > wrote:
> > > >
> > > > > Yes I believe that should work:
> > > > > -  add an empty drill-module.conf in the root of the udf jar
> > > > > -  add the package to drill.classpath.scanning.packages in the
> drill
> > > conf
> > > > > (possibly using drill-override.conf)
> > > > >
> > > > > However if you are adding the drill-module.conf file to the jar,
> you
> > > > might
> > > > > as well add the package in it. (unless there's some other reason)
> > > > >
> > > > > On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> > > > > challapallirahul@gmail.com> wrote:
> > > > >
> > > > > > Just to be sure, If I have an empty drill-module.conf in the
root
> > of
> > > my
> > > > > udf
> > > > > > jar, then there is no way to add the package information to
the
> > > global
> > > > > > drill-override.conf file?
> > > > > >
> > > > > > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <
> julien@dremio.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > You are correct:
> > > > > > > The jar containing the UDFs should have a drill-module.conf
at
> > the
> > > > root
> > > > > > > adding your package to the property
> > > drill.classpath.scanning.packages
> > > > > for
> > > > > > > scanning
> > > > > > > drill.classpath.scanning.packages :
> > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > >         my.package.containing.my.udfs
> > > > > > >     ]
> > > > > > > Jars that don't contain a drill-module.conf will not get
> scanned.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > > > > > challapallirahul@gmail.com> wrote:
> > > > > > >
> > > > > > > > Thanks for your reply Jason.
> > > > > > > >
> > > > > > > > If we cannot override the global configuration file,
then for
> > > > > existing
> > > > > > > > UDF's we have to re-compile them by modifying the
> > > drill-module.conf
> > > > > > file.
> > > > > > > > If so our UDF's are not backward compatible. Appreciate
it if
> > > > someone
> > > > > > can
> > > > > > > > confirm this.
> > > > > > > >
> > > > > > > > - Rahul
> > > > > > > >
> > > > > > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > > > > > altekrusejason@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Rahul,
> > > > > > > > >
> > > > > > > > > The error message you are seeing is in reading
a storage
> > plugin
> > > > > > > > > configuration file. I am planning to fix these
kinds of
> > > messages
> > > > to
> > > > > > > > > actually direct users at the file that is failing
parsing.
> I
> > > have
> > > > > > seen
> > > > > > > > this
> > > > > > > > > in the past when the classpath was incorrect
and one of the
> > > > plugins
> > > > > > > (like
> > > > > > > > > Hbase) was not included.
> > > > > > > > >
> > > > > > > > > Julien can confirm, but I think this might be
intentional
> to
> > > have
> > > > > the
> > > > > > > > paths
> > > > > > > > > read out of the modules configuration rather
than the
> global
> > > one
> > > > to
> > > > > > > save
> > > > > > > > > time when scanning the path (rather than scanning
all of
> the
> > > jars
> > > > > for
> > > > > > > all
> > > > > > > > > paths given in the override file).
> > > > > > > > >
> > > > > > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli
<
> > > > > > > > > challapallirahul@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Before 1.2, my udfs project contained an
empty
> > > > > drill-override.conf
> > > > > > > file
> > > > > > > > > and
> > > > > > > > > > I used to update the drill-override.conf
on all the
> > drillbits
> > > > to
> > > > > > > > specify
> > > > > > > > > > the package of my UDF. This is no longer
working for me.
> I
> > > > tried
> > > > > a
> > > > > > > few
> > > > > > > > > > things and below is how my drill-override.conf
file looks
> > now
> > > > > > > > > >
> > > > > > > > > > drill.classpath.scanning.packages :
> > > > > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > > > > > org.apache.drill.udfs ]
> > > > > > > > > > drill.exec: {
> > > > > > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > > > > > >   zk.connect: "localhost:5181"
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > When I restart the drillbits, I get this
strange error "
> > > Caused
> > > > > by:
> > > > > > > > > > com.fasterxml.jackson.databind.JsonMappingException:
> Could
> > > not
> > > > > > > resolve
> > > > > > > > > type
> > > > > > > > > > id 'hbase' into a subtype of [simple type,
class
> > > > > > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > > > > > >
> > > > > > > > > > If I moved the package information to the
> drill-module.conf
> > > in
> > > > my
> > > > > > > udf's
> > > > > > > > > > project, then things are working fine. However
this
> > requires
> > > > > > > > re-compiling
> > > > > > > > > > the udfs which is not desirable. Is there
any other way
> > > around
> > > > > > this ?
> > > > > > > > > >
> > > > > > > > > > - Rahul
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Julien
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Julien
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message