flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-10357) Streaming File Sink end-to-end test failed with mismatch
Date Tue, 30 Oct 2018 08:34:05 GMT

    [ https://issues.apache.org/jira/browse/FLINK-10357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668343#comment-16668343
] 

ASF GitHub Bot commented on FLINK-10357:
----------------------------------------

GJL commented on a change in pull request #6907: [FLINK-10357][tests] Improve StreamingFileSink
E2E test stability.
URL: https://github.com/apache/flink/pull/6907#discussion_r229213320
 
 

 ##########
 File path: flink-end-to-end-tests/test-scripts/test_streaming_file_sink.sh
 ##########
 @@ -45,47 +40,55 @@ function wait_for_restart {
 }
 
 ###################################
-# Wait a specific number of successful checkpoints
-# to have happened
+# Get all lines in part files and sort them numerically.
 #
 # Globals:
-#   None
+#   OUTPUT_PATH
 # Arguments:
-#   $1: the job id
-#   $2: the number of expected successful checkpoints
-#   $3: timeout in seconds
+#   None
 # Returns:
 #   None
 ###################################
-function wait_for_number_of_checkpoints {
-    local job_id=$1
-    local expected_num_checkpoints=$2
-    local timeout=$3
-    local count=0
-
-    echo "Starting to wait for completion of ${expected_num_checkpoints} checkpoints"
-    while (($(get_completed_number_of_checkpoints ${job_id}) < ${expected_num_checkpoints}));
do
+function get_complete_result {
+    find "${OUTPUT_PATH}" -type f \( -iname "part-*" \) -exec cat {} + | sort -g
+}
 
-        if [[ ${count} -gt ${timeout} ]]; then
-            echo "A timeout occurred waiting for successful checkpoints"
+###################################
+# Waits until a number of values have been written within a timeout.
+# If the timeout expires, exit with return code 1.
+#
+# Globals:
+#   None
+# Arguments:
+#   $1: the number of expected values
+#   $2: timeout in seconds
+# Returns:
+#   None
+###################################
+function wait_for_complete_result {
+    local expected_number_of_values=$1
+    local polling_timeout=$2
+    local polling_interval=1
+    local seconds_elapsed=0
+
+    local number_of_values=$(get_complete_result | tail -1)
 
 Review comment:
   I will just initialize `number_of_values` with `0`. The loop will run at least once, we
we will see the output of `echo "Number of produced values ${number_of_values}/${expected_number_of_values}"`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Streaming File Sink end-to-end test failed with mismatch
> --------------------------------------------------------
>
>                 Key: FLINK-10357
>                 URL: https://issues.apache.org/jira/browse/FLINK-10357
>             Project: Flink
>          Issue Type: Bug
>          Components: Tests
>    Affects Versions: 1.6.0
>            Reporter: Till Rohrmann
>            Assignee: Gary Yao
>            Priority: Critical
>              Labels: pull-request-available, test-stability
>             Fix For: 1.6.3, 1.7.0
>
>         Attachments: flink-logs.tgz, flink-streaming-file-sink-logs.tgz
>
>
> The {{Streaming File Sink end-to-end test}} failed on an Amazon instance with the following
result: 
> {code}
> FAIL File Streaming Sink: Output hash mismatch.  Got f2000bbc18a889dc8ec4b6f2b47bf9f5,
expected 6727342fdd3aae2129e61fc8f433fb6f.
> head hexdump of actual:
> 0000000   0  \n   1  \n   2  \n   3  \n   4  \n   5  \n   6  \n   7  \n
> 0000010   8  \n   9  \n
> 0000014
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message