drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Rogers <prog...@mapr.com>
Subject Re: A tutorial on how to write a custom storage plugin
Date Wed, 15 Mar 2017 23:50:20 GMT
Hi Muhammad,

I know of no tutorial. I recently updated the “mock” storage plugin and so have some experience
with this interface. You can find my notes at [1]. 

Unfortunately, creating a storage plugin seems to require a significant commitment of time
and effort because you must understand:

* The storage plugin structure. Some bits are a bit unusual (such as bindings between the
various bits and pieces.)
* Enough about Calcite to provide it with the required plan-time information.
* The Jackson-serialization structure for various components.
* The rather complex process by which you work with ScanBatch and value vectors to get data
from your data source into value vectors.
* Drill as a whole so you can build Drill and debug it. The only way to test a plugin is by
running it inside Drill.

Your best approach is to carefully study existing plugins. You can start simple, say with
the mock plugin. Replace the bits of the mock implementation with your own. Try to get it
to work for a single table. Then, add other functionality gradually.

By the time you are done you will be well on your way to being an expert in some of Drill’s


- Paul

[1] https://github.com/paul-rogers/drill/wiki/Storage-Plugin-Model

> On Mar 15, 2017, at 10:35 AM, Muhammad Gelbana <m.gelbana@gmail.com> wrote:
> Everyone,
> Is there a tutorial on how to write a custom storage plugin to support some
> sort of a proprietary data source ?
> I understand I can configure a storage plugin based on pre-shipped storage
> plugins such as the one for MongoDB, MySQL\JDBC, HBase, Hadoop HDFS..etc,
> but that's not what I need.
> I need to write a new plugin to support a storage that is not publicly
> available.
> The best that I've got so far is the information on these pages:
>   -
>   https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java
>   -
>   https://github.com/apache/drill/tree/master/contrib/storage-kudu/src/main/java/org/apache/drill/exec/store/kudu
> But this isn't enough to understand what needs to be done or troubleshoot
> errors while developing the plugin.
> *---------------------*
> *Muhammad Gelbana*
> http://www.linkedin.com/in/mgelbana

View raw message