impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikramjeet Vig (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4080 [part 7]: Codegen once per fragment
Date Wed, 01 Apr 2020 16:56:32 GMT
Hello Tim Armstrong, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/15408

to look at the new patch set (#12).

Change subject: IMPALA-4080 [part 7]: Codegen once per fragment
......................................................................

IMPALA-4080 [part 7]: Codegen once per fragment

This is the final patch:
- Removes all temporary changes that called codegen on plan nodes and
  data sink through the exec node.
- Introduces FragmentState class that basically contains the plan tree,
  data sink config and the codegen state for every fragment.
- Every FragmentState object has its own runtime profile that helps to
  map out every codegen object's profile to its fragment id.
- Plan nodes are generated before fragment instance threads are spawned.
- The first fragment instance thread to call codegen on its parent
  parent fragment does the actual codegen work and the rest of the
  instance threads wait till codegen is complete.
- This very closely mimics the timing of the state machines both for
  query state and the fragment instance states.

Testing:
- Ran exhaustive tests successfully.

An example of how the codegen in the profile looks:

Per Node Profiles:
  localhost:22002:
     - AdmissionSlots: 1 (1)
     - BloomFilterBytes: 16.00 MB (16777216)
     - ScratchBytesRead: 0
     - ScratchBytesWritten: 0
     - ScratchFileUsedBytes: 0
     - ScratchReads: 0 (0)
     - ScratchWrites: 0 (0)
     - TotalEncryptionTime: 0.000ns
     - TotalReadBlockTime: 0.000ns
    Buffer pool:
       - AllocTime: 153.663us
       - CumulativeAllocationBytes: 16.00 MB (16777216)
       - CumulativeAllocations: 8 (8)
       - PeakReservation: 16.00 MB (16777216)
       - PeakUnpinnedBytes: 0
       - PeakUsedReservation: 16.00 MB (16777216)
       - ReadIoBytes: 0
       - ReadIoOps: 0 (0)
       - ReadIoWaitTime: 0.000ns
       - ReservationLimit: 16.00 MB (16777216)
       - SystemAllocTime: 133.841us
       - WriteIoBytes: 0
       - WriteIoOps: 0 (0)
       - WriteIoWaitTime: 0.000ns
    Fragment F02:
      CodeGen:(Total: 695.207ms, non-child: 0.000ns, % non-child: 0.00%)
         - CodegenInvoluntaryContextSwitches: 102 (102)
         - CodegenTotalWallClockTime: 695.180ms
           - CodegenSysTime: 0.000ns
           - CodegenUserTime: 688.998ms
         - CodegenVoluntaryContextSwitches: 22 (22)
         - CompileTime: 39.657ms
         - IrGenerationTime: 27.443ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 2.53 MB (2647876)
         - NumFunctions: 161 (161)
         - NumInstructions: 7.13K (7128)
         - OptimizationTime: 596.075ms
         - PeakMemoryUsage: 3.48 MB (3649536)
         - PrepareTime: 30.782ms
    Fragment F00:
      CodeGen:(Total: 145.314ms, non-child: 0.000ns, % non-child: 0.00%)
         - CodegenInvoluntaryContextSwitches: 11 (11)
         - CodegenTotalWallClockTime: 145.304ms
           - CodegenSysTime: 3.944ms
           - CodegenUserTime: 137.738ms
         - CodegenVoluntaryContextSwitches: 4 (4)
         - CompileTime: 6.697ms
         - IrGenerationTime: 3.173ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 2.53 MB (2647876)
         - NumFunctions: 41 (41)
         - NumInstructions: 1.02K (1024)
         - OptimizationTime: 119.623ms
         - PeakMemoryUsage: 512.00 KB (524288)
         - PrepareTime: 14.405ms

Change-Id: I3aef8bc621f96caafe9a1c378617a2987e4ad452
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/benchmarks/hash-benchmark.cc
M be/src/codegen/llvm-codegen-test.cc
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
M be/src/exec/aggregation-node-base.cc
M be/src/exec/aggregation-node-base.h
M be/src/exec/aggregator.cc
M be/src/exec/aggregator.h
M be/src/exec/analytic-eval-node.cc
M be/src/exec/analytic-eval-node.h
M be/src/exec/blocking-join-node.cc
M be/src/exec/blocking-join-node.h
M be/src/exec/data-sink.cc
M be/src/exec/data-sink.h
M be/src/exec/exchange-node.cc
M be/src/exec/exchange-node.h
M be/src/exec/exec-node.cc
M be/src/exec/exec-node.h
M be/src/exec/grouping-aggregator.cc
M be/src/exec/grouping-aggregator.h
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-avro-scanner.h
M be/src/exec/hdfs-columnar-scanner.cc
M be/src/exec/hdfs-columnar-scanner.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-sequence-scanner.h
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/hdfs-text-scanner.h
M be/src/exec/join-builder.cc
M be/src/exec/join-builder.h
M be/src/exec/nested-loop-join-builder.cc
M be/src/exec/nested-loop-join-builder.h
M be/src/exec/nested-loop-join-node.cc
M be/src/exec/nested-loop-join-node.h
M be/src/exec/non-grouping-aggregator.cc
M be/src/exec/non-grouping-aggregator.h
M be/src/exec/partial-sort-node.cc
M be/src/exec/partial-sort-node.h
M be/src/exec/partitioned-hash-join-builder.cc
M be/src/exec/partitioned-hash-join-builder.h
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
M be/src/exec/scan-node.cc
M be/src/exec/scan-node.h
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
M be/src/exec/sort-node.cc
M be/src/exec/sort-node.h
M be/src/exec/subplan-node.cc
M be/src/exec/subplan-node.h
M be/src/exec/topn-node.cc
M be/src/exec/topn-node.h
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M be/src/exec/unnest-node.cc
M be/src/exec/unnest-node.h
M be/src/exprs/agg-fn.cc
M be/src/exprs/agg-fn.h
M be/src/exprs/expr-codegen-test.cc
M be/src/exprs/hive-udf-call.cc
M be/src/exprs/hive-udf-call.h
M be/src/exprs/is-not-empty-predicate.cc
M be/src/exprs/is-not-empty-predicate.h
M be/src/exprs/kudu-partition-expr.cc
M be/src/exprs/kudu-partition-expr.h
M be/src/exprs/scalar-expr.cc
M be/src/exprs/scalar-expr.h
M be/src/exprs/scalar-fn-call.cc
M be/src/exprs/scalar-fn-call.h
M be/src/exprs/slot-ref.cc
M be/src/exprs/slot-ref.h
M be/src/exprs/tuple-is-null-predicate.cc
M be/src/exprs/tuple-is-null-predicate.h
M be/src/exprs/valid-tuple-id.cc
M be/src/exprs/valid-tuple-id.h
M be/src/runtime/CMakeLists.txt
M be/src/runtime/data-stream-test.cc
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/fragment-instance-state.h
A be/src/runtime/fragment-state.cc
A be/src/runtime/fragment-state.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/runtime-state.cc
M be/src/runtime/runtime-state.h
M be/src/runtime/test-env.cc
M be/src/service/fe-support.cc
M be/src/udf/udf-internal.h
M be/src/udf/udf.cc
M be/src/util/tuple-row-compare.cc
M be/src/util/tuple-row-compare.h
100 files changed, 930 insertions(+), 625 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/15408/12
-- 
To view, visit http://gerrit.cloudera.org:8080/15408
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3aef8bc621f96caafe9a1c378617a2987e4ad452
Gerrit-Change-Number: 15408
Gerrit-PatchSet: 12
Gerrit-Owner: Bikramjeet Vig <bikramjeet.vig@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bikramjeet.vig@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>

Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message