impala-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mi...@apache.org
Subject impala git commit: Revert "IMPALA-3887: Wait for HDFS replication in data loading"
Date Wed, 20 Dec 2017 18:16:23 GMT
Repository: impala
Updated Branches:
  refs/heads/master 061287d91 -> 2fb11fb73


Revert "IMPALA-3887: Wait for HDFS replication in data loading"

Using fsck breaks non-HDFS builds: local, S3, and Isilon.

This reverts commit 5a7c10ec3d500c1bb582e985cb299e7dbcfe5520.

Change-Id: I0b12a42049543ca0b267b5146a0bbcdd2316abfc
Reviewed-on: http://gerrit.cloudera.org:8080/8880
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Impala Public Jenkins


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/2fb11fb7
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/2fb11fb7
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/2fb11fb7

Branch: refs/heads/master
Commit: 2fb11fb7326b11c9c5c1b1bab811333d91d8640c
Parents: 061287d
Author: David Knupp <dknupp@cloudera.com>
Authored: Tue Dec 19 10:53:49 2017 -0800
Committer: Impala Public Jenkins <impala-public-jenkins@gerrit.cloudera.org>
Committed: Tue Dec 19 23:26:29 2017 +0000

----------------------------------------------------------------------
 testdata/bin/create-load-data.sh | 19 -------------------
 1 file changed, 19 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/2fb11fb7/testdata/bin/create-load-data.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/create-load-data.sh b/testdata/bin/create-load-data.sh
index 2111ba1..df6622a 100755
--- a/testdata/bin/create-load-data.sh
+++ b/testdata/bin/create-load-data.sh
@@ -449,23 +449,6 @@ function copy-and-load-ext-data-source {
     ${IMPALA_HOME}/testdata/bin/create-data-source-table.sql
 }
 
-function wait-hdfs-replication {
-  FAIL_COUNT=0
-  while [[ "$FAIL_COUNT" -ne "6" ]] ; do
-    FSCK_OUTPUT="$(hdfs fsck /test-warehouse)"
-    echo "$FSCK_OUTPUT"
-    if grep "Under-replicated blocks:[[:space:]]*0" <<< "$FSCK_OUTPUT"; then
-      return
-    fi
-    let FAIL_COUNT="$FAIL_COUNT"+1
-    sleep 5
-  done
-  echo "Some HDFS blocks are still under replicated after 30s."
-  echo "Some tests cannot pass without fully replicated blocks (IMPALA-3887)."
-  echo "Failing the data loading."
-  exit 1
-}
-
 # For kerberized clusters, use kerberos
 if ${CLUSTER_DIR}/admin is_kerberized; then
   LOAD_DATA_ARGS="${LOAD_DATA_ARGS} --use_kerberos --principal=${MINIKDC_PRINC_HIVE}"
@@ -559,5 +542,3 @@ run-step "Computing table stats" compute-table-stats.log \
     ${IMPALA_HOME}/testdata/bin/compute-table-stats.sh
 
 run-step "Copying auth policy file" copy-auth-policy.log copy-auth-policy
-
-run-step "Waiting for HDFS replication" wait-hdfs-replication.log wait-hdfs-replication


Mime
View raw message