drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Altekruse <altekruseja...@gmail.com>
Subject Re: drill support mysql?
Date Thu, 15 Jan 2015 19:27:13 GMT
That's what we were thinking. This would make the only work that needs to
be done is to implement a mapping between Enumerable and value vectors for
getting the data into Drill.

Linfeng, does this sound like something you would be interested in trying
to contribute to Drill?

-Jason

On Thu, Jan 15, 2015 at 10:59 AM, Julian Hyde <julianhyde@gmail.com> wrote:

> Calcite has a JDBC adapter, so it can convert parts of the query plan to
> SQL and execute it in the JDBC source. Generally you want to push down as
> much as possible. It works with MySQL and pretty much any JDBC data source.
>
> Drill uses Calcite internally, so if Drill exposed Calcite adapters as
> data sources this problem would be solved.
>
> Julian
>
> > On Jan 15, 2015, at 10:30 AM, Jason Altekruse <altekrusejason@gmail.com>
> wrote:
> >
> > Hello Linfeng!
> >
> > Welcome to the Drill community!
> >
> > Currently drill does not support querying traditional databases like
> MySQL,
> > but it is a feature we have discussed adding for some time. If you are
> > interested in trying to add your own, you can start by taking a look at
> > some of the existing storage plugins. For a basic overview of the process
> > of populating the in memory data structure for records in Drill, I would
> > recommend reading through the Json reader implementation. While the in
> > memory format for Drill is columnar, to allow for various execution
> > optimizations, a simple row-by-row interface is provided for writing into
> > the data structure.
> >
> >
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java
> >
> > Another important aspect of creating a storage plugin is leveraging the
> > underlying systems capabilities to avoid reading an entire dataset for
> each
> > query. We have a system in place for pushing down filters and selected
> > columns to the underlying storage layer. This is not necessary to be able
> > to run queries in Drill,  as you can simply read all of the data out of
> the
> > storage system and let Drill filter it out as necessary, but for many
> > workloads this will obviously be sub-optimal in many cases ( just about
> > anything but a select *). For an example of some of this filter rewriting
> > you can see the work done on the Mongo plugin.
> >
> >
> https://github.com/apache/drill/tree/master/contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo
> >
> > An important thing to note here is that the filters have to be rewritten
> > from Drill convention to Mongo convention. In the case of a MySQL plugin,
> > the storage system really could perform any sql operations that require
> > only the tables within your MySQL instance and likely further optimize
> > those subsets of the queries. To implement this functionality, we could
> go
> > far beyond what we have implemented with other storage plugins in terms
> of
> > pushdown, which requires more complex rewriting. Thankfully a dependency
> > that Drill is already leveraging for its planning, calcite, has the
> > capabilities to do these types of more complex pushdowns already. It uses
> > an Enumerable interface (a fancy iterator) to expose data from one of its
> > storage engines, such as a relational database, calcite includes a
> simpler
> > single node execution engine that can actually act much like Drill for
> > smaller workloads. Likely the best first step for getting MySQL working
> > would be to write a Drill storage plugin that takes an Enumerable as
> input,
> > then we should be able to leverage all of calcite's existing
> functionality
> > for doing the query rewrites and pushdowns relatively easily.
> >
> > -Jason Altekruse
> >
> > On Wed, Jan 7, 2015 at 7:39 PM, lifeng.wang@baifendian.com <
> > lifeng.wang@baifendian.com> wrote:
> >
> >> 你好:
> >> 请问drill目前是否支持mysql
> >> 如果我自己想开发一套连接mysql的插件我该怎么做?
> >>
> >> Hello:
> >> Does the drill is currently supports mysql
> >> If I want to develop a set of connection mysql plugin what should I do?
> >>
> >>
> >>
> >>
> >> lifeng.wang@baifendian.com
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message