Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Content-Type: multipart/alternative;
 boundary="===============1684396001016115980=="
MIME-Version: 1.0
Subject: Re: Review Request 33115: DRILL-2762: Update Fragment state reporting
 and error collection
From: "Aman Sinha" <asinha@maprtech.com>
To: "Chris Westin" <chriswestin42@gmail.com>,
 "abdelhakim deneche" <adeneche@gmail.com>,
 "Steven Phillips" <sphillips@maprtech.com>
Cc: "Jacques Nadeau" <jacques.drill@gmail.com>,
 "Aman Sinha" <asinha@maprtech.com>, "drill" <drill-dev@incubator.apache.org>
Date: Wed, 15 Apr 2015 17:28:43 -0000
Message-ID: <20150415172843.1421.25737@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Aman Sinha" <noreply@reviews.apache.org>
References: <20150415052243.14114.91223@reviews.apache.org>
In-Reply-To: <20150415052243.14114.91223@reviews.apache.org>
Reply-To: "Aman Sinha" <asinha@maprtech.com>

--===============1684396001016115980==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33115/#review80216
-----------------------------------------------------------


I am running with this patch on the performance cluster and noticed that the query profile display is all messed up - probably because of new fields added to the profile ?  Some operators are missing on the display and the ones that are shown have incorrect fields.

- Aman Sinha


On April 15, 2015, 5:22 a.m., Jacques Nadeau wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33115/
> -----------------------------------------------------------
> 
> (Updated April 15, 2015, 5:22 a.m.)
> 
> 
> Review request for drill, abdelhakim deneche, Chris Westin, and Steven Phillips.
> 
> 
> Repository: drill-git
> 
> 
> Description
> -------
> 
> DRILL-2762: Update Fragment state reporting and error collection
> 
> DeferredException
> - Add new throwAndClear operation on to allow checking for exceptions preClose in FragmentContext
> - Add new getAndClear operation
> 
> BufferManager
> - Ensure close() can be called multiple times by clearing managed buffer list on close().
> 
> FragmentContext/FragmentExecutor
> - Update FragmentContext to have a preClose so that we can check closure state before doing final close.
> - Update so that there is only a single state maintained between FragmentContext and FragmentExecutor
> - Clean up FragmentExecutor run() method to better manage error states and have only single terminal point (avoiding multiple messages to Foreman).
> - Add new CANCELLATION_REQUESTED state for FragmentState.
> - Move all users of isCancelled or isFailed in main code to use shouldContinue()
> - Update receivingFragmentFinished message to not cancel fragment (only inform root operator of cancellation)
> 
> WorkManager Updates
> - Add new afterExecute command to the WorkManager ExecutorService so that we get log entries if a thread leaks an exception. (Otherwise logs don't show these exceptions and they only go to standard out.)
> 
> Foreman/QueryManager
> - Extract listenable interfaces into anonymous inner classes from body of Foreman
> 
> QueryManager
> - Update QueryManager to track completed nodes rather than completed fragments using NodeTracker
> - Update DrillbitStatusListener to decrement expected completion messages on Nodes that have died to avoid query hang when a node dies
> 
> FragmentData/MinorFragmentProfile
> - Add ability to track last status update as well as last time fragment made progress
> 
> AbstractRecordBatch
> - Update awareness of current cancellation state to avoid cancellation delays
> 
> Misc. Other changes
> - Move ByteCode optimization code to only record assembly and code as trace messages
> - Update SimpleRootExec to create fake ExecutorState to make existing tests work.
> - Update sort to exit prematurely in the case that the fragment was asked to cancel.
> - Add finals to all edited files.
> - Modify control handler and FragmentManager to directly support receivingFragmentFinished
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/drill/common/DeferredException.java 99f18f1 
>   exec/java-exec/src/main/java/org/apache/drill/exec/ops/BufferManager.java 2d22d84 
>   exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java da2229c 
>   exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java 2bb29e5 
>   exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java e2bcec3 
>   exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 23ef0d3 
>   exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentData.java 433ab26 
>   exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java 8626d5b 
>   exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java a7e6c46 
>   protocol/src/main/java/org/apache/drill/exec/proto/SchemaUserBitShared.java f72d5e1 
>   protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java ac1bcbb 
>   protocol/src/main/java/org/apache/drill/exec/proto/beans/MinorFragmentProfile.java 5cd71f9 
>   protocol/src/main/protobuf/UserBitShared.proto 2938114 
> 
> Diff: https://reviews.apache.org/r/33115/diff/
> 
> 
> Testing
> -------
> 
> Regression & Unit, more manual testing planned before final patch.
> 
> 
> Thanks,
> 
> Jacques Nadeau
> 
>


--===============1684396001016115980==--