impala-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tarmstr...@apache.org
Subject [7/8] impala git commit: IMPALA-6070: Parallelize another bit of data load.
Date Thu, 14 Dec 2017 17:48:44 GMT
IMPALA-6070: Parallelize another bit of data load.

The two Kudu loads and Hive UDFs can all run in parallel. This
should shave about 4 minutes off of the data load. (Current
timings are 3.5, 4, and 0.6 minutes, see below.)

I've run dataload with this change many times.

   Loading Kudu functional (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu.log)...
     Loading workload 'functional-query' using exploration strategy 'core' in table formats
'kudu/none/none' OK (Took: 3 min 29 sec)
   Loading Kudu TPCH (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu-tpch.log)...
     Loading workload 'tpch' using exploration strategy 'core' in table formats 'kudu/none/none'
OK (Took: 4 min 0 sec)
   Loading Hive UDFs (logging to /home/ubuntu/Impala/logs/data_loading/build-and-copy-hive-udfs.log)...
     Loading Hive UDFs OK (Took: 0 min 41 sec)

Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
Reviewed-on: http://gerrit.cloudera.org:8080/8822
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Impala Public Jenkins


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/11dbb395
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/11dbb395
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/11dbb395

Branch: refs/heads/master
Commit: 11dbb3952a1c598f27de281c5020ed2df325d6e8
Parents: 5c593be
Author: Philip Zeyliger <philip@cloudera.com>
Authored: Tue Dec 12 15:38:54 2017 -0800
Committer: Impala Public Jenkins <impala-public-jenkins@gerrit.cloudera.org>
Committed: Thu Dec 14 02:28:40 2017 +0000

----------------------------------------------------------------------
 testdata/bin/create-load-data.sh | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/11dbb395/testdata/bin/create-load-data.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/create-load-data.sh b/testdata/bin/create-load-data.sh
index 099fe59..df6622a 100755
--- a/testdata/bin/create-load-data.sh
+++ b/testdata/bin/create-load-data.sh
@@ -507,13 +507,14 @@ fi
 
 if $KUDU_IS_SUPPORTED; then
   # Tests depend on the kudu data being clean, so load the data from scratch.
-  run-step "Loading Kudu functional" load-kudu.log \
+  run-step-backgroundable "Loading Kudu functional" load-kudu.log \
         load-data "functional-query" "core" "kudu/none/none" force
-  run-step "Loading Kudu TPCH" load-kudu-tpch.log \
+  run-step-backgroundable "Loading Kudu TPCH" load-kudu-tpch.log \
         load-data "tpch" "core" "kudu/none/none" force
 fi
-run-step "Loading Hive UDFs" build-and-copy-hive-udfs.log \
+run-step-backgroundable "Loading Hive UDFs" build-and-copy-hive-udfs.log \
     build-and-copy-hive-udfs
+run-step-wait-all
 run-step "Running custom post-load steps" custom-post-load-steps.log \
     custom-post-load-steps
 


Mime
View raw message