flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-10801) Fix sql client integrate elasticsearch connector test failure
Date Fri, 09 Nov 2018 08:53:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681084#comment-16681084
] 

ASF GitHub Bot commented on FLINK-10801:
----------------------------------------

pnowojski closed pull request #7060: [FLINK-10801][e2e] Retry verify_result_hash in elastichsearch-common
URL: https://github.com/apache/flink/pull/7060
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/flink-end-to-end-tests/test-scripts/common.sh b/flink-end-to-end-tests/test-scripts/common.sh
index 36cc0df142c..6b8bd061f15 100644
--- a/flink-end-to-end-tests/test-scripts/common.sh
+++ b/flink-end-to-end-tests/test-scripts/common.sh
@@ -451,6 +451,16 @@ function cancel_job {
 }
 
 function check_result_hash {
+  local error_code=0
+  check_result_hash_no_exit "$@" || error_code=$?
+
+  if [ "$error_code" != "0" ]
+  then
+    exit $error_code
+  fi
+}
+
+function check_result_hash_no_exit {
   local name=$1
   local outfile_prefix=$2
   local expected=$3
@@ -462,18 +472,19 @@ function check_result_hash {
     actual=$(LC_ALL=C sort $outfile_prefix* | md5sum | awk '{print $1}')
   else
     echo "Neither 'md5' nor 'md5sum' binary available."
-    exit 2
+    return 2
   fi
   if [[ "$actual" != "$expected" ]]
   then
     echo "FAIL $name: Output hash mismatch.  Got $actual, expected $expected."
     echo "head hexdump of actual:"
     head $outfile_prefix* | hexdump -c
-    exit 1
+    return 1
   else
     echo "pass $name"
     # Output files are left behind in /tmp
   fi
+  return 0
 }
 
 # This function starts the given number of task managers and monitors their processes.
diff --git a/flink-end-to-end-tests/test-scripts/elasticsearch-common.sh b/flink-end-to-end-tests/test-scripts/elasticsearch-common.sh
index 8bb9c42ab69..695905222f5 100644
--- a/flink-end-to-end-tests/test-scripts/elasticsearch-common.sh
+++ b/flink-end-to-end-tests/test-scripts/elasticsearch-common.sh
@@ -83,23 +83,34 @@ function verify_result_hash {
   local name=$1
   local index=$2
   local numRecords=$3
-  local hash=$4
+  local expectedHash=$4
 
-  while : ; do
+  for i in {1..30}
+  do
+    local error_code=0
+
+    echo "Result verification attempt $i..."
     curl "localhost:9200/${index}/_search?q=*&pretty" > $TEST_DATA_DIR/es_output ||
true
 
-    if [ -n "$(grep "\"total\" : $numRecords" $TEST_DATA_DIR/es_output)" ]; then
-      break
-    else
-      echo "Waiting for Elasticsearch records ..."
+    # remove meta information
+    sed '2,9d' $TEST_DATA_DIR/es_output > $TEST_DATA_DIR/es_content
+
+    check_result_hash_no_exit "$name" $TEST_DATA_DIR/es_content "$expectedHash" || error_code=$?
+
+    if [ "$error_code" != "0" ]
+    then
+      echo "Result verification attempt $i has failed"
       sleep 1
+    else
+      break
     fi
   done
 
-  # remove meta information
-  sed '2,9d' $TEST_DATA_DIR/es_output > $TEST_DATA_DIR/es_content
-
-  check_result_hash "$name" $TEST_DATA_DIR/es_content "$hash"
+  if [ "$error_code" != "0" ]
+  then
+    echo "All result verification attempts have failed"
+    exit $error_code
+  fi
 }
 
 function shutdown_elasticsearch_cluster {


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Fix sql client integrate elasticsearch connector test failure
> -------------------------------------------------------------
>
>                 Key: FLINK-10801
>                 URL: https://issues.apache.org/jira/browse/FLINK-10801
>             Project: Flink
>          Issue Type: Bug
>          Components: E2E Tests
>            Reporter: vinoyang
>            Assignee: Piotr Nowojski
>            Priority: Major
>              Labels: pull-request-available
>
> It usually reports : 
> {code:java}
> FAIL SQL Client Elasticsearch Upsert: Output hash mismatch. Got 6187222e109ee9222e6b2f117742070c,
expected 982cb32908def9801e781381c1b8a8db.
> head hexdump of actual:
> 0000000 { \n " h i t s " : { \n 
> 0000010 " t o t a l " : 3 , \n
> 0000020 " m a x _ s c o r e " 
> 0000030 : 1 . 0 , \n " h i t s
> 0000040 " : [ \n { \n 
> 0000050 " _ i n d e x " :
> 0000060 " m y _ u s e r s " , \n 
> 0000070 " _ t y p e " : "
> 0000080 u s e r " , \n "
> 0000090 _ i d " : " 1 _ B o b "
> 00000a0 , \n " _ s c o r
> 00000b0 e " : 1 . 0 , \n 
> 00000ba
> {code}
> the actual hash means : 
> {code:java}
> {
>   "hits" : {
>     "total" : 3,
>     "max_score" : 1.0,
>     "hits" : [
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "1_Bob  ",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 1,
>           "user_name" : "Bob  ",
>           "user_count" : 1
>         }
>       },
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "22_Alice",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 22,
>           "user_name" : "Alice",
>           "user_count" : 1
>         }
>       },
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "42_Greg ",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 42,
>           "user_name" : "Greg ",
>           "user_count" : 3
>         }
>       }
>     ]
>   }
> }
> {code}
> the expected hash code means : 
> {code:java}
> {
>   "hits" : {
>     "total" : 3,
>     "max_score" : 1.0,
>     "hits" : [
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "1_Bob  ",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 1,
>           "user_name" : "Bob  ",
>           "user_count" : 2
>         }
>       },
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "22_Alice",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 22,
>           "user_name" : "Alice",
>           "user_count" : 1
>         }
>       },
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "42_Greg ",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 42,
>           "user_name" : "Greg ",
>           "user_count" : 3
>         }
>       }
>     ]
>   }
> }
> {code}
> It seems that the user count for "Bob" is off by 1.
> The speculation is due to the premature acquisition of aggregated statistics from Elasticsearch.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message