flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-10801) Fix sql client integrate elasticsearch connector test failure
Date Thu, 08 Nov 2018 14:25:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679829#comment-16679829
] 

ASF GitHub Bot commented on FLINK-10801:
----------------------------------------

twalthr commented on a change in pull request #7060: [FLINK-10801][e2e] Retry verify_result_hash
in elastichsearch-common
URL: https://github.com/apache/flink/pull/7060#discussion_r231904591
 
 

 ##########
 File path: flink-end-to-end-tests/test-scripts/elasticsearch-common.sh
 ##########
 @@ -83,23 +83,33 @@ function verify_result_hash {
   local name=$1
   local index=$2
   local numRecords=$3
-  local hash=$4
+  local expectedHash=$4
 
-  while : ; do
+  local error_code=0
+
+  for i in {1..30}; do
+    echo "Result verification attempt $i..."
     curl "localhost:9200/${index}/_search?q=*&pretty" > $TEST_DATA_DIR/es_output ||
true
 
-    if [ -n "$(grep "\"total\" : $numRecords" $TEST_DATA_DIR/es_output)" ]; then
-      break
-    else
-      echo "Waiting for Elasticsearch records ..."
+    # remove meta information
+    sed '2,9d' $TEST_DATA_DIR/es_output > $TEST_DATA_DIR/es_content
+
+    check_result_hash_no_exit "$name" $TEST_DATA_DIR/es_content "$expectedHash" || result=$?
 
 Review comment:
   Shouldn't `result=$?` be `error_code=$?`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Fix sql client integrate elasticsearch connector test failure
> -------------------------------------------------------------
>
>                 Key: FLINK-10801
>                 URL: https://issues.apache.org/jira/browse/FLINK-10801
>             Project: Flink
>          Issue Type: Bug
>          Components: E2E Tests
>            Reporter: vinoyang
>            Assignee: Piotr Nowojski
>            Priority: Major
>              Labels: pull-request-available
>
> It usually reports : 
> {code:java}
> FAIL SQL Client Elasticsearch Upsert: Output hash mismatch. Got 6187222e109ee9222e6b2f117742070c,
expected 982cb32908def9801e781381c1b8a8db.
> head hexdump of actual:
> 0000000 { \n " h i t s " : { \n 
> 0000010 " t o t a l " : 3 , \n
> 0000020 " m a x _ s c o r e " 
> 0000030 : 1 . 0 , \n " h i t s
> 0000040 " : [ \n { \n 
> 0000050 " _ i n d e x " :
> 0000060 " m y _ u s e r s " , \n 
> 0000070 " _ t y p e " : "
> 0000080 u s e r " , \n "
> 0000090 _ i d " : " 1 _ B o b "
> 00000a0 , \n " _ s c o r
> 00000b0 e " : 1 . 0 , \n 
> 00000ba
> {code}
> the actual hash means : 
> {code:java}
> {
>   "hits" : {
>     "total" : 3,
>     "max_score" : 1.0,
>     "hits" : [
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "1_Bob  ",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 1,
>           "user_name" : "Bob  ",
>           "user_count" : 1
>         }
>       },
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "22_Alice",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 22,
>           "user_name" : "Alice",
>           "user_count" : 1
>         }
>       },
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "42_Greg ",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 42,
>           "user_name" : "Greg ",
>           "user_count" : 3
>         }
>       }
>     ]
>   }
> }
> {code}
> the expected hash code means : 
> {code:java}
> {
>   "hits" : {
>     "total" : 3,
>     "max_score" : 1.0,
>     "hits" : [
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "1_Bob  ",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 1,
>           "user_name" : "Bob  ",
>           "user_count" : 2
>         }
>       },
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "22_Alice",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 22,
>           "user_name" : "Alice",
>           "user_count" : 1
>         }
>       },
>       {
>         "_index" : "my_users",
>         "_type" : "user",
>         "_id" : "42_Greg ",
>         "_score" : 1.0,
>         "_source" : {
>           "user_id" : 42,
>           "user_name" : "Greg ",
>           "user_count" : 3
>         }
>       }
>     ]
>   }
> }
> {code}
> It seems that the user count for "Bob" is off by 1.
> The speculation is due to the premature acquisition of aggregated statistics from Elasticsearch.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message