drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parth Chandra <pchan...@maprtech.com>
Subject Re: WIP for prepared statement/metadata querying in C++ native client
Date Thu, 25 Aug 2016 17:30:51 GMT
That's along the lines of what I was asking. What I'm hoping for, though,
is a concrete proposal.
 As you say, we could be doing a bunch of improvements, but before we start
talking about changing the ODBC driver, I would recommend we have at least
one incremental improvement that end users will benefit from.


On Wed, Aug 24, 2016 at 11:50 AM, Jacques Nadeau <jacques@dremio.com> wrote:

> I'm not sure what unused code you are worried about. The backend already
> implements these APIs and these changes would simply expose that interface
> to the C++ consumers. I'd expect the ODBC driver to be updated to take
> advantage of these APIs and thus all the code would be exercised all the
> time. The main difference is: once we have the new client in user's hands,
> we can iterate on performance improvements without having to upgrade the
> client again in the future. Since this is an additive change, the existing
> clients will continue to work without issue.
>
> I think we already have some jiras about improving metadata performance and
> BI tool experience. I'll try to dig them up. The thought was to start as
> simple as possible, which is what the backend does now: do what the client
> is doing but on the server. The next clear step for metadata preparation is
> to do single parsing & planning (without the limit 0) so that in known
> schema cases, we can avoid double parsing/planning (especially expensive in
> partition pruning and metadata cache cases). Note, I'd expect that this
> next version will still maintain the same behavior if the RowType is not
> known on the initial DrillTable (where we don't have the limit 0 short
> circuit today)
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Aug 23, 2016 at 6:30 PM, Parth Chandra <parthc@apache.org> wrote:
>
> > I would think that the Java client would be sufficient for
> experimentation.
> > But what I'm looking for is an actual proposal for the backend changes.
> If
> > we don't have one (not sure if there is a JIRA open for it), then we
> should
> > start that now.
> > Otherwise I'm afraid we will end up with a bunch of large amount of code
> > that is not used. In particular, for the C++ client, I would like to
> avoid
> > that.
> >
> >
> >
> >
> > On Tue, Aug 23, 2016 at 6:09 PM, Jacques Nadeau <jacques@dremio.com>
> > wrote:
> >
> > > The clear quick win would be caching parsing/planning/pruning on the
> > server
> > > and reusing (if executed within time t or until statement/connection
> are
> > > closed).
> > >
> > > My thinking is to get an implementation of the client that is opaque to
> > the
> > > server implementation so that we can iterate on preparation without
> > having
> > > to constantly update the client. From there, developers can easily
> > > experiment with different mechanisms to find what works best for BI
> > tools.
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Tue, Aug 23, 2016 at 1:54 PM, Laurent Goujon <laurent@dremio.com>
> > > wrote:
> > >
> > > > I'm  currently focusing on the client work, and making sure the C++
> > > client
> > > > is not lagging behind the Java one. I personally haven't worked on
> > > backend
> > > > changes for prepared statements.
> > > >
> > > > Laurent
> > > >
> > > > On Mon, Aug 22, 2016 at 7:32 PM, Parth Chandra <parthc@apache.org>
> > > wrote:
> > > >
> > > > > Hi Laurent,
> > > > >
> > > > >   I'll take a look at this in the next few days.
> > > > >
> > > > >   On a related note, do you or Venki have a proposal for the
> backend
> > > > > changes (i.e actual implementation of prepare)? It would be a good
> > idea
> > > > to
> > > > > start a discussion on that.
> > > > >
> > > > > Parth
> > > > >
> > > > > On Mon, Aug 22, 2016 at 3:24 PM, Laurent Goujon <
> laurent@dremio.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I just started working on adding support for prepared statements
> > and
> > > > > > metadata querying in the C++ Drill client. Hopefully, nobody
else
> > has
> > > > > > started working on this (The Drill jiras don't mention any
> activity
> > > on
> > > > > > this), but if it is not the case, let me know.
> > > > > >
> > > > > > My working branch is
> > > > > > https://github.com/laurentgo/drill/tree/laurent/improve-
> > > native-client.
> > > > > >
> > > > > > For now, I just have a basic interface API (
> > > > > > https://github.com/laurentgo/drill/commit/
> > > > 1f55a3e631cd97016b113b9d4bca07
> > > > > > b5e016a25e),
> > > > > > and it would be nice if people knowledgeable about the C++ client
> > > could
> > > > > > review it and give me some feedback. I'm also adding an actual
> > > initial
> > > > > > implementation in the coming days.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Laurent
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message