drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Muhammad Gelbana <m.gelb...@gmail.com>
Subject Re: A tutorial on how to write a custom storage plugin
Date Thu, 16 Mar 2017 11:10:04 GMT
That's very helpful Paul ! But I can't access the wiki page you referenced.
May be it's set to private ?

*---------------------*
*Muhammad Gelbana*
http://www.linkedin.com/in/mgelbana

On Thu, Mar 16, 2017 at 1:50 AM, Paul Rogers <progers@mapr.com> wrote:

> Hi Muhammad,
>
> I know of no tutorial. I recently updated the “mock” storage plugin and so
> have some experience with this interface. You can find my notes at [1].
>
> Unfortunately, creating a storage plugin seems to require a significant
> commitment of time and effort because you must understand:
>
> * The storage plugin structure. Some bits are a bit unusual (such as
> bindings between the various bits and pieces.)
> * Enough about Calcite to provide it with the required plan-time
> information.
> * The Jackson-serialization structure for various components.
> * The rather complex process by which you work with ScanBatch and value
> vectors to get data from your data source into value vectors.
> * Drill as a whole so you can build Drill and debug it. The only way to
> test a plugin is by running it inside Drill.
>
> Your best approach is to carefully study existing plugins. You can start
> simple, say with the mock plugin. Replace the bits of the mock
> implementation with your own. Try to get it to work for a single table.
> Then, add other functionality gradually.
>
> By the time you are done you will be well on your way to being an expert
> in some of Drill’s internals.
>
> Thanks,
>
> - Paul
>
> [1] https://github.com/paul-rogers/drill/wiki/Storage-Plugin-Model
>
> > On Mar 15, 2017, at 10:35 AM, Muhammad Gelbana <m.gelbana@gmail.com>
> wrote:
> >
> > Everyone,
> >
> > Is there a tutorial on how to write a custom storage plugin to support
> some
> > sort of a proprietary data source ?
> >
> > I understand I can configure a storage plugin based on pre-shipped
> storage
> > plugins such as the one for MongoDB, MySQL\JDBC, HBase, Hadoop HDFS..etc,
> > but that's not what I need.
> >
> > I need to write a new plugin to support a storage that is not publicly
> > available.
> >
> > The best that I've got so far is the information on these pages:
> >
> >   -
> >   https://github.com/apache/drill/blob/master/exec/java-
> exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java
> >   -
> >   https://github.com/apache/drill/tree/master/contrib/
> storage-kudu/src/main/java/org/apache/drill/exec/store/kudu
> >
> >
> > But this isn't enough to understand what needs to be done or troubleshoot
> > errors while developing the plugin.
> >
> > *---------------------*
> > *Muhammad Gelbana*
> > http://www.linkedin.com/in/mgelbana
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message