kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neha Narkhede (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-440) Create a regression test framework for distributed environment testing
Date Mon, 13 Aug 2012 16:42:38 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433272#comment-13433272

Neha Narkhede commented on KAFKA-440:

Thanks for patch v2! Overall, it looks like a good start. Here are some review comments -

1. Standardize utils into system_test/utils. Remove lib. The libraries that go in here should
be available to all other scripts through an 'import library' statement.
1.1. Rename li_reg_test_helper to KafkaSystemTestUtils. This has Kafka specific helper APIs.
Other util APIs can go in system_test/utils/SystemTestUtils.
1.2. Can we include the logger in there as well ? Every script should be able to import that.

1.3. Rename RtLogging to Logger.
1.4. We shouldn't have to pass around the rtLogger object everywhere. I hope this can be available
as a module level static variable. Once I import Logger, I should just be able to use a variable
named logger to log statements
1.5. Rename RegTestEnv to SystemTestEnv. Let's add all useful environment variables here like
– kafkaBaseDir, systemTestBaseDir, testSuiteBaseDir, testCaseBaseDir, testCaseLogDir etc
1.6. replica_basic_test.py currently doesn't have access to the above environment variables
which makes it awkward to use. 

2. Rename reg_test* to SystemTestUtils or something like that

3. Rename suite_replication to replication_test_suite. You  might want to name every test
suite as *_test_suite so that the scripts can detect that as a test suite.

4. Rename reg_test_driver to test_driver.

5. It will be good to follow some Python style guide like this one - http://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles
5.1. Package names should be all lower case and *preferably* without underscores, al though
sometimes this is difficult to avoid. So something like reg_test_helper could just be utils/testutils.
5.2. Class names should be CamelCase, with first letter capitalized.
5.3. Function names should be all lower case with words separated by underscores.
5.4. Constants should be defined at module level with all letters capitalized and separated
with underscores.

6. A lot of commands need to prepend 'ssh host' to them. How about having a helper API run_at_host_command(host,
command) that will do this and return the command string ?

7. Also, all commands need to specify the host to run on, the path to the script, some arguments
to that script and an output file. It will be nice to add a helper API to do that.

8. It will be good to structure the testcase/logs directory by role-entityid. The reason is
that we want to collect metrics and plot graphs for every entity. These will involve generating
several csv/svg/png files per entity. Instead of putting them all in one testcase/logs directory,
how about having the following structure -
    1. testcase/logs 
    2. |__ zookeeper-0
    3.      |__ metrics
    4.      ||__ dashboards
    5. |__ kafka-1
    6.      |__ metrics
    7.      |__ dashboards

9. To make it easier to use the directory structure above, I think we should have access to
helper APIs that given the entity id can return the path to the metrics/dashboards directories.
This will be used by the APIs that collect metrics and plot graphs.

10. cluster_config.json describes each entity with a set of properties. It might be easier
to define a class called TestEntity that has all these properties. On startup, the test will
read cluster_config.json and create a map/list of TestEntity objects. The test scripts should
have access to these.

11. How about having lifecycle management for all testcases in a test suite similar to Junit
? For example, all test cases in one test suite can have a setup and teardown method, where
common tasks can be performed. However, you can probably do this as part of another JIRA.
> Create a regression test framework for distributed environment testing
> ----------------------------------------------------------------------
>                 Key: KAFKA-440
>                 URL: https://issues.apache.org/jira/browse/KAFKA-440
>             Project: Kafka
>          Issue Type: Task
>            Reporter: John Fung
>            Assignee: John Fung
>         Attachments: kafka-440-v1.patch, kafka-440-v2.patch
> Initial requirements:
> 1. The whole test framework is preferably coded in Python (a common scripting language
which has well supported features)
> 2. Test framework driver should be generic (distributed environment can be local host)
> 3. Test framework related configurations are defined in JSON format
> 4. Test environment, suite, case definitions may be defined in the following levels:
>   4-a entity_id is used as a key for looking up related config from different levels
>   4-b Cluster level defines: entity_id, hostname, kafka_home, java_home, ...
>   4-c Test suite / case level defines:
>     4-c-1 zookeeper: entity_id, clientPort, dataDir, log_filename, config_filename
>     4-c-2 broker: entity_id, port, log.file.size, log.dir, log_filename, config_filename
>     4-c-3 producer: entity_id, topic, threads, compression-codec, message-size, log_filename,

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message