impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Becker (Code Review)" <>
Subject [Impala-ASF-CR] WIP: Asynchronous code generation
Date Thu, 09 Apr 2020 22:53:27 GMT
Daniel Becker has uploaded a new patch set (#30). ( )

Change subject: WIP: Asynchronous code generation

WIP: Asynchronous code generation

This commit introduces optional asynchronous code generation.

Asynchronous code generation means that instead of waiting for codegen
to finish, the query starts in interpreted mode while codegen is done on
another thread.

All the function pointers that point to codegen'd functions are changed
to be atomic, wrapped in a CodegenFnPtr. These are initialised to
nullptr and as long as they are nullptr, the corresponding interpreted
functions are used (as before). When code generation is ready, the
funtion pointers are set by the codegen thread. No synchronisation is
needed as the function pointers are atomic and it is not a problem if,
at a given moment, only a subset of the codegen'd function pointers are
set and the rest are interpreted.

Asynchronous code generation can be turned on using the ASYNC_CODEGEN
boolean query option.

TODO: The default should be synchronous codegen for now.
TODO: Testing.
TODO: Benchmarks.

Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b
M be/src/benchmarks/
A be/src/codegen/codegen-fn-ptr.h
M be/src/codegen/
M be/src/codegen/
M be/src/codegen/
M be/src/codegen/llvm-codegen.h
M be/src/exec/
M be/src/exec/grouping-aggregator.h
M be/src/exec/
M be/src/exec/hdfs-avro-scanner.h
M be/src/exec/
M be/src/exec/hdfs-columnar-scanner.h
M be/src/exec/
M be/src/exec/
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/
M be/src/exec/hdfs-scanner.h
M be/src/exec/
M be/src/exec/
M be/src/exec/
M be/src/exec/non-grouping-aggregator.h
M be/src/exec/parquet/
M be/src/exec/
M be/src/exec/
M be/src/exec/partitioned-hash-join-builder.h
M be/src/exec/
M be/src/exec/
M be/src/exec/partitioned-hash-join-node.h
M be/src/exec/
M be/src/exec/select-node.h
M be/src/exec/
M be/src/exec/topn-node.h
M be/src/exec/
M be/src/exec/union-node.h
M be/src/exprs/
M be/src/exprs/
M be/src/exprs/scalar-expr.h
M be/src/exprs/scalar-expr.inline.h
M be/src/exprs/
M be/src/exprs/scalar-fn-call.h
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/fragment-state.h
M be/src/runtime/
M be/src/runtime/krpc-data-stream-sender.h
M be/src/service/
M be/src/service/query-options.h
M be/src/util/
M be/src/util/tuple-row-compare.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
A tests/query_test/
M tests/query_test/
M tests/query_test/
54 files changed, 930 insertions(+), 432 deletions(-)

  git pull ssh:// refs/changes/05/15105/30
To view, visit
To unsubscribe, visit

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b
Gerrit-Change-Number: 15105
Gerrit-PatchSet: 30
Gerrit-Owner: Daniel Becker <>
Gerrit-Reviewer: Bikramjeet Vig <>
Gerrit-Reviewer: Csaba Ringhofer <>
Gerrit-Reviewer: Daniel Becker <>
Gerrit-Reviewer: Impala Public Jenkins <>
Gerrit-Reviewer: Tim Armstrong <>
Gerrit-Reviewer: Todd Lipcon <>
Gerrit-Reviewer: Zoltan Borok-Nagy <>

  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message