From reviews-return-77255-archive-asf-public=cust-asf.ponee.io@impala.apache.org Thu Apr 9 22:53:33 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 33CAD180634 for ; Fri, 10 Apr 2020 00:53:33 +0200 (CEST) Received: (qmail 65508 invoked by uid 500); 9 Apr 2020 22:53:32 -0000 Mailing-List: contact reviews-help@impala.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@impala.apache.org Received: (qmail 65489 invoked by uid 99); 9 Apr 2020 22:53:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Apr 2020 22:53:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id C1C9A1A329C for ; Thu, 9 Apr 2020 22:53:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.574 X-Spam-Level: X-Spam-Status: No, score=0.574 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=0.2, KAM_DMARC_STATUS=0.01, RDNS_DYNAMIC=0.363, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 7wyJYgd1zU8w for ; Thu, 9 Apr 2020 22:53:28 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=75.101.130.251; helo=ip-10-146-233-104.ec2.internal; envelope-from=gerrit@cloudera.org; receiver= Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 49A27BB8C2 for ; Thu, 9 Apr 2020 22:53:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id 039MrRDr001944; Thu, 9 Apr 2020 22:53:27 GMT Message-Id: <202004092253.039MrRDr001944@ip-10-146-233-104.ec2.internal> X-Gerrit-PatchSet: 30 Date: Thu, 9 Apr 2020 22:53:27 +0000 From: "Daniel Becker (Code Review)" To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org X-Gerrit-MessageType: newpatchset Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_WIP=3A_Asynchronous_code_generation=0A?= X-Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b X-Gerrit-Change-Number: 15105 X-Gerrit-ChangeURL: X-Gerrit-Commit: 4c9b547b6b895ca73976fb45561077f56f13f32b In-Reply-To: References: Reply-To: daniel.becker@cloudera.com, impala-cr@cloudera.com, marcelk@gmail.com, vihang@cloudera.com, amsinha@cloudera.com, reviews@impala.incubator.apache.org, progers@cloudera.com MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.14.15 Content-Type: multipart/alternative; boundary="Gl4OBfqn0A8="; charset=UTF-8 --Gl4OBfqn0A8= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Daniel Becker has uploaded a new patch set (#30)=2E ( http://gerrit=2Ecloud= era=2Eorg:8080/15105 ) Change subject: WIP: Asynchronous code generation = =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E= =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E= =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E WIP: Asynchro= nous code generation This commit introduces optional asynchronous code gen= eration=2E Asynchronous code generation means that instead of waiting for = codegen to finish, the query starts in interpreted mode while codegen is do= ne on another thread=2E All the function pointers that point to codegen'd = functions are changed to be atomic, wrapped in a CodegenFnPtr=2E These are = initialised to nullptr and as long as they are nullptr, the corresponding i= nterpreted functions are used (as before)=2E When code generation is ready,= the funtion pointers are set by the codegen thread=2E No synchronisation i= s needed as the function pointers are atomic and it is not a problem if, at= a given moment, only a subset of the codegen'd function pointers are set a= nd the rest are interpreted=2E Asynchronous code generation can be turned = on using the ASYNC_CODEGEN boolean query option=2E TODO: The default shoul= d be synchronous codegen for now=2E TODO: Testing=2E TODO: Benchmarks=2E C= hange-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b --- M be/src/benchmarks= /hash-benchmark=2Ecc A be/src/codegen/codegen-fn-ptr=2Eh M be/src/codegen/g= en_ir_descriptions=2Epy M be/src/codegen/llvm-codegen-test=2Ecc M be/src/co= degen/llvm-codegen=2Ecc M be/src/codegen/llvm-codegen=2Eh M be/src/exec/gro= uping-aggregator=2Ecc M be/src/exec/grouping-aggregator=2Eh M be/src/exec/h= dfs-avro-scanner=2Ecc M be/src/exec/hdfs-avro-scanner=2Eh M be/src/exec/hdf= s-columnar-scanner=2Ecc M be/src/exec/hdfs-columnar-scanner=2Eh M be/src/ex= ec/hdfs-orc-scanner=2Ecc M be/src/exec/hdfs-scan-node-base=2Ecc M be/src/ex= ec/hdfs-scan-node-base=2Eh M be/src/exec/hdfs-scanner=2Ecc M be/src/exec/hd= fs-scanner=2Eh M be/src/exec/hdfs-sequence-scanner=2Ecc M be/src/exec/hdfs-= text-scanner=2Ecc M be/src/exec/non-grouping-aggregator=2Ecc M be/src/exec/= non-grouping-aggregator=2Eh M be/src/exec/parquet/hdfs-parquet-scanner=2Ecc= M be/src/exec/partitioned-hash-join-builder-ir=2Ecc M be/src/exec/partitio= ned-hash-join-builder=2Ecc M be/src/exec/partitioned-hash-join-builder=2Eh = M be/src/exec/partitioned-hash-join-node-ir=2Ecc M be/src/exec/partitioned-= hash-join-node=2Ecc M be/src/exec/partitioned-hash-join-node=2Eh M be/src/e= xec/select-node=2Ecc M be/src/exec/select-node=2Eh M be/src/exec/topn-node= =2Ecc M be/src/exec/topn-node=2Eh M be/src/exec/union-node=2Ecc M be/src/ex= ec/union-node=2Eh M be/src/exprs/expr-codegen-test=2Ecc M be/src/exprs/scal= ar-expr=2Ecc M be/src/exprs/scalar-expr=2Eh M be/src/exprs/scalar-expr=2Ein= line=2Eh M be/src/exprs/scalar-fn-call=2Ecc M be/src/exprs/scalar-fn-call= =2Eh M be/src/runtime/fragment-instance-state=2Ecc M be/src/runtime/fragmen= t-state=2Ecc M be/src/runtime/fragment-state=2Eh M be/src/runtime/krpc-data= -stream-sender=2Ecc M be/src/runtime/krpc-data-stream-sender=2Eh M be/src/s= ervice/query-options=2Ecc M be/src/service/query-options=2Eh M be/src/util/= tuple-row-compare=2Ecc M be/src/util/tuple-row-compare=2Eh M common/thrift/= ImpalaInternalService=2Ethrift M common/thrift/ImpalaService=2Ethrift A tes= ts/query_test/test_async_codegen=2Epy M tests/query_test/test_queries=2Epy = M tests/query_test/test_query_mem_limit=2Epy 54 files changed, 930 insertio= ns(+), 432 deletions(-) git pull ssh://gerrit=2Ecloudera=2Eorg:29418/Im= pala-ASF refs/changes/05/15105/30 -- To view, visit http://gerrit=2Ecloude= ra=2Eorg:8080/15105 To unsubscribe, visit http://gerrit=2Ecloudera=2Eorg:80= 80/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Messag= eType: newpatchset Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194a= a6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 30 Gerrit-Owner: Daniel Be= cker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-R= eviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy= --Gl4OBfqn0A8=--