impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Robinson <he...@cloudera.com>
Subject Re: [Impala-ASF-CR] IMPALA-2905: Handle coordinator fragment lifecycle like all others
Date Tue, 11 Oct 2016 23:53:06 GMT
Supposed to be a draft; will post the final patch in a short while.

On 11 October 2016 at 16:44, Henry Robinson (Code Review) <
gerrit@cloudera.org> wrote:

> Henry Robinson has uploaded a new patch set (#11).
>
> Change subject: IMPALA-2905: Handle coordinator fragment lifecycle like
> all others
> ......................................................................
>
> IMPALA-2905: Handle coordinator fragment lifecycle like all others
>
> The plan-root fragment instance that runs on the coordinator should be
> handled like all others: started via RPC and run asynchronously. Without
> this, the fragment requires special-case code throughout the
> coordinator, and does not show up in system metrics etc.
>
> This patch adds a new sink type, PlanRootSink, to the root fragment
> instance so that the coordinator can pull row batches that are pushed by
> the root instance. The coordinator signals completion to the fragment
> instance via closing the consumer side of the sink, whereupon the
> instance is free to complete.
>
> Since the root instance now runs asynchronously wrt to the coordinator,
> we add several coordination methods to allow the coordinator to wait for
> a point in the instance's execution to be hit - e.g. to wait until the
> instance has been opened.
>
> Done in this patch:
>
> * Add PlanRootSink
> * Add coordination to PFE to allow coordinator to observe lifecycle
> * Make FragmentMgr a singleton
> * Removed dead code from Coordinator::Wait() and elsewhere.
> * Moved result output exprs out of QES and into PlanRootSink.
> * Remove special-case limit-based teardown of coordinator fragment, and
>   supporting functions in PlanFragmentExecutor.
> * Simplified lifecycle of PlanFragmentExecutor by separating Open() into
>   OpenPlan() and Exec(), the latter of which drives the sink by reading
>   rows from the plan tree.
> * Add child profile to PlanFragmentExecutor to measure time spent in
>   each lifecycle phase.
> * Removed dependency between InitExecProfiles() and starting root fragment.
>
> Not yet done:
>
> * Fix planner tests to reflect new sink added at root.
>
> Change-Id: Ibb0064ec2f085fa3a5598ea80894fb489a01e4df
> ---
> M be/src/exec/CMakeLists.txt
> M be/src/exec/data-sink.cc
> A be/src/exec/plan-root-sink.cc
> A be/src/exec/plan-root-sink.h
> M be/src/runtime/coordinator.cc
> M be/src/runtime/coordinator.h
> M be/src/runtime/exec-env.cc
> M be/src/runtime/exec-env.h
> M be/src/runtime/plan-fragment-executor.cc
> M be/src/runtime/plan-fragment-executor.h
> M be/src/scheduling/query-schedule.cc
> M be/src/scheduling/query-schedule.h
> M be/src/scheduling/simple-scheduler.cc
> M be/src/service/fragment-exec-state.cc
> M be/src/service/fragment-exec-state.h
> M be/src/service/fragment-mgr.cc
> M be/src/service/impala-beeswax-server.cc
> M be/src/service/impala-hs2-server.cc
> M be/src/service/impala-internal-service.h
> M be/src/service/impala-server.cc
> M be/src/service/impala-server.h
> M be/src/service/query-exec-state.cc
> M be/src/service/query-exec-state.h
> A be/src/service/query-result-set.h
> M be/src/testutil/in-process-servers.cc
> M common/thrift/DataSinks.thrift
> M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
> A fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
> M fe/src/main/java/org/apache/impala/planner/Planner.java
> M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
> M tests/failure/test_failpoints.py
> M tests/hs2/test_json_endpoints.py
> 32 files changed, 1,011 insertions(+), 898 deletions(-)
>
>
>   git pull ssh://gerrit.cloudera.org:29418/Impala-ASF
> refs/changes/02/4402/11
> --
> To view, visit http://gerrit.cloudera.org:8080/4402
> To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
>
> Gerrit-MessageType: newpatchset
> Gerrit-Change-Id: Ibb0064ec2f085fa3a5598ea80894fb489a01e4df
> Gerrit-PatchSet: 11
> Gerrit-Project: Impala-ASF
> Gerrit-Branch: master
> Gerrit-Owner: Henry Robinson <henry@cloudera.com>
> Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
> Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
> Gerrit-Reviewer: Marcel Kornacker <marcel@cloudera.com>
> Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
>
> --
> You received this message because you are subscribed to the Google Groups
> "impala-cr" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to impala-cr+unsubscribe@cloudera.com.
> For more options, visit https://groups.google.com/a/cloudera.com/d/optout.
>



-- 
Henry Robinson
Software Engineer
Cloudera
415-994-6679

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message