drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parth Chandra <pchan...@maprtech.com>
Subject Re: Partition pruning inconsistency
Date Tue, 01 Sep 2015 06:17:27 GMT
Yes we would need to enhance the protocol a bit. Depending on whether the
warning is issued before results are sent (as in planning/optimization) or
whether the warning is issued after the results are completed (for example
we can issue a warning if, in future, Drill decides to drop bad rows and
continue), we would send a QUERY_RUNNING_WITH_INFO or a
QUERY_COMPLETED_WITH_INFO status message or something similar. The status
message can already carry a DrillPBError message with it.
Of course, JDBC/C++ client/ODBC will all have to be updated.
We could potentially add additional information to send back to the client
in the status message including DDL status etc.


On Mon, Aug 31, 2015 at 10:57 PM, Jacques Nadeau <jacques@dremio.com> wrote:

> I've been thinking that we need to add support for returning warnings.
> Have you looked how to add to JDBC or ODBC?  We'll need to update the RPC
> protocol since I believe we don't currently have an accommodation for
> warnings. Maybe add along with DDL queries?
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Aug 31, 2015 at 10:52 PM, Parth Chandra <pchandra@maprtech.com>
> wrote:
>
> > A better idea would be to return a warning in the results and let
> jdbc/odbc
> > show a warning with the result data.
> >
> > On Wed, Aug 26, 2015 at 8:31 AM, Aman Sinha <asinha@maprtech.com> wrote:
> >
> > > We have had some issues where the same query run at different times
> > > (possibly with other queries running concurrently...not sure about the
> > > concurrency level)  either performed partition pruning or did not.  The
> > > times where it failed happened due to couple of reasons :
> > >   (a) allocateNew() in the PruneScanRule failed with an out of memory
> > > condition
> > >   (b) the interpreter evaluator encountered an error with a particular
> > > expression type evaluation
> > >
> > > The PruneScanRule currently logs a warning message and does not fail
> the
> > > query since this is a performance optimization.  While we will address
> > the
> > > root cause of (a) and (b) (there's a JIRA open for (b) )  an important
> > > issue is the inconsistent behavior of a query.
> > >
> > > Should we provide a system setting that allows the query to fail in
> this
> > > situation ?
> > > Note that other rules in the optimizer could also fail and some rules
> > log
> > > warnings but those failures are very rare, while PruneScan rule is
> doing
> > > more complex operations - creating value vectors, doing interpreter
> > > evaluation - so the chances of something failing increases.
> > >
> > > Aman
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message