drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5083) IteratorValidator does not handle RecordIterator cleanup call to next( )
Date Tue, 29 Nov 2016 03:21:58 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15704033#comment-15704033

Paul Rogers commented on DRILL-5083:

More info. The query in question had, in essence:

- Merge Join
  - Merge Join
    - Sort
    - Sort
 - Sort

Each Merge Join input seems to be wrapped in a RecordIterator and IteratorValidatorBatchIterator.
If the first sort above fails, its RecordIterator learns about the error. But, the others
do not, especially not the third sort. When closing, we try to clean out batches by calling
next(). This sort of works (with the exception explained above) for an the active iterator
above the failed sort. But, the other record iterators never saw the error, so they are in
their prime start-up state. When they get the batch-cleaning next(), they think it is a real
one and proceed to do real work.

So, the clean-up code needs to handle the case where it is cleaning up operators down a path
in the tree that did not fail. But, because of the way sort and joins work, those operators
may never even have started. Thus, they can't tell the clean-up next from a "time to get started"

> IteratorValidator does not handle RecordIterator cleanup call to next( )
> ------------------------------------------------------------------------
>                 Key: DRILL-5083
>                 URL: https://issues.apache.org/jira/browse/DRILL-5083
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Priority: Minor
> This one is very confusing...
> In a test with a MergeJoin and external sort, operators are stacked something like this:
> {code}
> Screen
> - MergeJoin
> - - External Sort
> ...
> {code}
> Using the injector to force a OOM in spill, the external sort threw a UserException up
the stack. This was handed by:
> {code}
> IteratorValidatorBatchIterator.next( )
> RecordIterator.clearInflightBatches( )
> RecordIterator.close( )
> MergeJoinBatch.close( )
> {code}
> Which does the following:
> {code}
>       // Check whether next() should even have been called in current state.
>       if (null != exceptionState) {
>         throw new IllegalStateException(
> {code}
> But, the exceptionState is set, so we end up throwing an IllegalStateException during
> Seems the code should agree: if {{next( )}} will be called during cleanup, then {{next(
)}} should gracefully handle that case.

This message was sent by Atlassian JIRA

View raw message