drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3845) PartitionSender doesn't send last batch for receivers that already terminated
Date Thu, 14 Jan 2016 20:28:40 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098789#comment-15098789

ASF GitHub Bot commented on DRILL-3845:

Github user amansinha100 commented on a diff in the pull request:

    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
    @@ -286,7 +286,7 @@ public void flush(boolean schemaChanged) throws IOException {
           //      sender has acknowledged the terminate request. After sending the last batch,
all further batches are
           //      dropped.
           //   3. Partitioner thread is interrupted due to cancellation of fragment.
    -      final boolean isLastBatch = isLast || terminated || Thread.currentThread().isInterrupted();
    +      final boolean isLastBatch = isLast || Thread.currentThread().isInterrupted();
    --- End diff --
    If the sender is not sending the "last batch" message, could we be sure that no receiver
is waiting for it ? Like we discussed it seems the MergingReceiver does not behave the same
as UnorderedReceiver in the sense that the MR waits for the last batch. 

> PartitionSender doesn't send last batch for receivers that already terminated
> -----------------------------------------------------------------------------
>                 Key: DRILL-3845
>                 URL: https://issues.apache.org/jira/browse/DRILL-3845
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>            Reporter: Deneche A. Hakim
>            Assignee: Jacques Nadeau
>             Fix For: 1.5.0
>         Attachments: 29c45a5b-e2b9-72d6-89f2-d49ba88e2939.sys.drill
> Even if a receiver has finished and informed the corresponding partition sender, the
sender will still try to send a "last batch" to the receiver when it's done. In most cases
this is fine as those batches will be silently dropped by the receiving DataServer, but if
a receiver has finished +10 minutes ago, DataServer will throw an exception as it couldn't
find the corresponding FragmentManager (WorkEventBus has a 10 minutes recentlyFinished cache).
> DRILL-2274 is a reproduction for this case (after the corresponding fix is applied).

This message was sent by Atlassian JIRA

View raw message