drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@dremio.com>
Subject Re: WIP for prepared statement/metadata querying in C++ native client
Date Wed, 24 Aug 2016 18:50:49 GMT
I'm not sure what unused code you are worried about. The backend already
implements these APIs and these changes would simply expose that interface
to the C++ consumers. I'd expect the ODBC driver to be updated to take
advantage of these APIs and thus all the code would be exercised all the
time. The main difference is: once we have the new client in user's hands,
we can iterate on performance improvements without having to upgrade the
client again in the future. Since this is an additive change, the existing
clients will continue to work without issue.

I think we already have some jiras about improving metadata performance and
BI tool experience. I'll try to dig them up. The thought was to start as
simple as possible, which is what the backend does now: do what the client
is doing but on the server. The next clear step for metadata preparation is
to do single parsing & planning (without the limit 0) so that in known
schema cases, we can avoid double parsing/planning (especially expensive in
partition pruning and metadata cache cases). Note, I'd expect that this
next version will still maintain the same behavior if the RowType is not
known on the initial DrillTable (where we don't have the limit 0 short
circuit today)

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Aug 23, 2016 at 6:30 PM, Parth Chandra <parthc@apache.org> wrote:

> I would think that the Java client would be sufficient for experimentation.
> But what I'm looking for is an actual proposal for the backend changes. If
> we don't have one (not sure if there is a JIRA open for it), then we should
> start that now.
> Otherwise I'm afraid we will end up with a bunch of large amount of code
> that is not used. In particular, for the C++ client, I would like to avoid
> that.
>
>
>
>
> On Tue, Aug 23, 2016 at 6:09 PM, Jacques Nadeau <jacques@dremio.com>
> wrote:
>
> > The clear quick win would be caching parsing/planning/pruning on the
> server
> > and reusing (if executed within time t or until statement/connection are
> > closed).
> >
> > My thinking is to get an implementation of the client that is opaque to
> the
> > server implementation so that we can iterate on preparation without
> having
> > to constantly update the client. From there, developers can easily
> > experiment with different mechanisms to find what works best for BI
> tools.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Aug 23, 2016 at 1:54 PM, Laurent Goujon <laurent@dremio.com>
> > wrote:
> >
> > > I'm  currently focusing on the client work, and making sure the C++
> > client
> > > is not lagging behind the Java one. I personally haven't worked on
> > backend
> > > changes for prepared statements.
> > >
> > > Laurent
> > >
> > > On Mon, Aug 22, 2016 at 7:32 PM, Parth Chandra <parthc@apache.org>
> > wrote:
> > >
> > > > Hi Laurent,
> > > >
> > > >   I'll take a look at this in the next few days.
> > > >
> > > >   On a related note, do you or Venki have a proposal for the backend
> > > > changes (i.e actual implementation of prepare)? It would be a good
> idea
> > > to
> > > > start a discussion on that.
> > > >
> > > > Parth
> > > >
> > > > On Mon, Aug 22, 2016 at 3:24 PM, Laurent Goujon <laurent@dremio.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I just started working on adding support for prepared statements
> and
> > > > > metadata querying in the C++ Drill client. Hopefully, nobody else
> has
> > > > > started working on this (The Drill jiras don't mention any activity
> > on
> > > > > this), but if it is not the case, let me know.
> > > > >
> > > > > My working branch is
> > > > > https://github.com/laurentgo/drill/tree/laurent/improve-
> > native-client.
> > > > >
> > > > > For now, I just have a basic interface API (
> > > > > https://github.com/laurentgo/drill/commit/
> > > 1f55a3e631cd97016b113b9d4bca07
> > > > > b5e016a25e),
> > > > > and it would be nice if people knowledgeable about the C++ client
> > could
> > > > > review it and give me some feedback. I'm also adding an actual
> > initial
> > > > > implementation in the coming days.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Laurent
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message