Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
From: adeneche <git@git.apache.org>
To: dev@drill.apache.org
Reply-To: dev@drill.apache.org
References: <git-pr-319-drill@git.apache.org>
In-Reply-To: <git-pr-319-drill@git.apache.org>
Subject: [GitHub] drill pull request: DRILL-3845: PartitionSender doesn't send
 last ...
Content-Type: text/plain
Message-Id: <20160114203801.53BC3E3889@git1-us-west.apache.org>
Date: Thu, 14 Jan 2016 20:38:01 +0000 (UTC)

Github user adeneche commented on a diff in the pull request:

    https://github.com/apache/drill/pull/319#discussion_r49781604
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java ---
    @@ -286,7 +286,7 @@ public void flush(boolean schemaChanged) throws IOException {
           //      sender has acknowledged the terminate request. After sending the last batch, all further batches are
           //      dropped.
           //   3. Partitioner thread is interrupted due to cancellation of fragment.
    -      final boolean isLastBatch = isLast || terminated || Thread.currentThread().isInterrupted();
    +      final boolean isLastBatch = isLast || Thread.currentThread().isInterrupted();
    --- End diff --
    
    That is true. We have 2 possible solutions to fix this problem:
    - either we change all receivers so they no longer wait for the last batch when it's an early termination
    - or we make sure the partition sender sends the last batch as soon as possible to avoid the case where it's sent too late.
    
    Like we discussed it, it's not easy to enforce the 1st solution as even the senders don't respect this , e.g. a single sender that receives an early termination message will close the fragment without letting it's receivers wait for the last batch.
    
    I will update the PR to implement the 2nd solution 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---