flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9594) Add documentation for e2e test changes introduced with FLINK-9257
Date Mon, 18 Jun 2018 08:08:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515473#comment-16515473
] 

ASF GitHub Bot commented on FLINK-9594:
---------------------------------------

Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/6172#discussion_r195992920
  
    --- Diff: flink-end-to-end-tests/README.md ---
    @@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir> flink-end-to-end-tests/test-scripts/test_batch_wordcount
     
     ## Writing Tests
     
    -Have a look at test_batch_wordcount.sh for a very basic test and
    -test_streaming_kafka010.sh for a more involved example. Whenever possible, try
    -to put new functionality in common.sh so that it can be reused by other tests.
    +### Examples
    +Have a look at `test_batch_wordcount.sh` for a very basic test and
    +`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try
    +to put new functionality in `common.sh` so that it can be reused by other tests.
    +
    +### Adding a test case
    +In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh`
and / or `test-scripts/run-pre-commit-tests.sh`
    +
    +Adding a new test case generally follows the following pattern
    +
    +```sh
    +run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2"
    +```
    +
    +_Note: If you want to parameterize your tests please do so by adding multiple test cases
with parameters as arguments to the nightly / pre-commit test suites. This allows the test
runner to do a cleanup in between each individual test and also to fail those tests individually._
    +
    +### Passing your test
    +A test is considered to have passed if it:
    +- has exit code 0
    +- there are no non-empty .out files (nothing was written to stdout / stderr by your flink
program)
    +- there are no exceptions in the log files
    +- there are no errors in the log files
    +
    +### Failing your test
    +A test is considered to have failed if it:
    +- exited with non-zero exit code
    +- has non-empty *.out files (something was written to stdout / stderr by your flink program)
    +- there are errors in the log files
    +- there are exceptions in the log files
    +
    +_There is a whitelist for exceptions and errors that do not lead to failure, they can
be found in the `check_logs_for_errors` and `check_logs_for_exceptions` in `test-scripts/common.sh`_
    +
    +Please note that a previously supported pattern where you could assign a value the global
variable `PASS` to have your tests fail **is not supported anymore**
    +
    +### Cleanup
    +The test runner performs a cleanup after each test case, which includes:
    +- Stopping the cluster
    +- Killing all task and job managers
    +- Reverting config to default (if changed before)
    +- Cleaning up log and temp directories
    +
    +If your test case should do some *additional* cleanup, a common pattern is to trap a
`test_cleanup` function to `EXIT` in your test case like so:
    --- End diff --
    
    I would explicitly state that this typically consists of shutting down external systems
used by the e2e test, e.g. Kafka / Elasticsearch.


> Add documentation for e2e test changes introduced with FLINK-9257
> -----------------------------------------------------------------
>
>                 Key: FLINK-9594
>                 URL: https://issues.apache.org/jira/browse/FLINK-9594
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Florian Schmidt
>            Assignee: Florian Schmidt
>            Priority: Minor
>
> There were some changes introduced with FLINK-9257 in how end-to-end tests are structured
and how we handle triggering failure from a test case. This should be documented for future
developers writing e2e tests



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message