apr-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r..@apache.org
Subject cvs commit: apr/test README
Date Mon, 16 Sep 2002 01:53:08 GMT
rbb         2002/09/15 18:53:08

  Added:       test     README
  Log:
  Add a README describing how to use the test suite.
  
  Revision  Changes    Path
  1.1                  apr/test/README
  
  Index: README
  ===================================================================
  Writing APR tests
  
  All APR tests should be executable in 2 ways, as an individual program, or
  as a part of the full test suite.  The full test suite is controlled with
  the testall program.  At the beginning of the testall.c file, there is an
  array of functions called tests.  The testall program loops through this 
  array calling each function.  Each function returns a CuSuite variable, which
  is then added to the SuiteList.  Once all Suites have been added, the SuiteList
  is executed, and the output is printed to the screen.  All functions in the
  array should follow the same basic format:
  
  The Full Suite
  --------------
  
  /* The driver function.  This must return a CuSuite variable, which will
   * then be used to actually run the tests.  Essentially, all Suites are a
   * collection of tests.  The driver will take each Suite, and put it in a
   * SuiteList, which is a collection of Suites.
   */
  CuSuite *testtime(void)
  {
      /* The actual suite, this must be created for each test program.  Please
       * give it a useful name, that will inform the user of the feature being
       * tested.
       */
      CuSuite *suite = CuSuiteNew("Test Time");
  
      /* Each function must be added to the suite.  Each function represents
       * a single test.  It is possible to test multiple features in a single
       * function, although no tests currently do that.
       */
      SUITE_ADD_TEST(suite, test_now);
      SUITE_ADD_TEST(suite, test_gmtstr);
      SUITE_ADD_TEST(suite, test_localstr);
      SUITE_ADD_TEST(suite, test_exp_get_gmt);
      SUITE_ADD_TEST(suite, test_exp_get_lt);
      SUITE_ADD_TEST(suite, test_imp_gmt);
      SUITE_ADD_TEST(suite, test_rfcstr);
      SUITE_ADD_TEST(suite, test_ctime);
      SUITE_ADD_TEST(suite, test_strftime);
      SUITE_ADD_TEST(suite, test_strftimesmall);
      SUITE_ADD_TEST(suite, test_exp_tz);
      SUITE_ADD_TEST(suite, test_strftimeoffset);
  
      /* You must return the suite so that the driver knows which suites to
       * run.
       */
      return suite;
  }
  
  Building the full driver
  ------------------------
  
  All you need to do to build the full driver is run:
  
      make testall
  
  To run it, run:
  
      ./testall
  
  Individual Program
  ------------------
  
  The second way to run a test program is as an individual program.  This is
  done with another function, getsuite.  This function should always look like:
  
  #ifdef SINGLE_PROG
  CuSuite *getsuite(void)
  {
      return testtime();
  }
  #endif
  
  Notice that all this function does is return the suite that is generated
  in the function used by the full driver.  This is very important.
  
  Also, notice that this function is protected by "#ifdef SINGLE_PROG".  All
  test programs should include this function.  The individual program will
  call this function to get the suite, and then it will run the suite and
  print the output.
  
  The reason that this function must be protected, is that all test programs
  have this function, and when they are linked into the full driver, they
  conflict with each other.  I hope to fix this problem soon.
  
  Building individual test programs
  ---------------------------------
  
  To build individual test programs, you must do the following (for example):
  
      MY_CFLAGS=-DSINGLE_PROGRAM make testtime
  
  To run the test, run:
  
      ./testtime
  
  Most people should just build the full test driver.
  
  Reading the test suite output
  -----------------------------
  
  Once you run the test suite, you will get output like:
  
  All APR Tests:
      Test Strings:       ....
      Test Time:  ............
  
  16 tests run:  16 passed, 0 failed, 0 not implemented.
  
  There are a couple of things to look at with this.  First, if you look at the
  first function in this document, you should notice that the string passed to
  the CuSuiteNew function is in the output.  That is why the string should
  explain the feature you are testing.
  
  Second, this test passed completely.  This is obvious in two ways.  First, and
  most obvious, the summary line tells you that 16 tests were run and 16 tests
  passed.  However, the results can also be found in the lines above.  Every
  '.' in the output represents a passed test.
  
  If a test fails, the output will look like:
  
  All APR Tests:
      Test Strings:       ....
      Test Time:  ..F.........
  
  16 tests run:  15 passed, 1 failed, 0 not implemented.
  
  This is not very useful, because you don't know which test failed.  However,
  once you know that a test failed, you can run the suite again, with the
  -v option.  If you do this, you will get something like:
  
  All APR Tests:
      Test Strings:       ....
      Test Time:  ..F.........
  
  16 tests run:  15 passed, 1 failed, 0 not implemented.
  Failed tests:
  1) test_localstr: assert failed
  
  In this case, we know the test_localstr function failed, and there is an
  Assert in this that failed (I modified the test to fail for this document).
  Now, you can look at what that test does, and why it would have failed.
  
  There is one other possible output for the test suite (run with -v):
  
  All APR Tests:
      Test Strings:       ....
      Test Time:  ..N.........
  
  16 tests run:  15 passed, 0 failed, 1 not implemented.
  
  Not Implemented tests:
  
  Not Implemented tests:
  1) test_localstr: apr_time_exp_lt not implemented on this platform
  
  The 'N' means that a function has returned APR_ENOTIMPL.  This should be 
  treated as an error, and the function should be implemented as soon as
  possible.
  
  Adding New test Suites to the full driver
  -------------------------------------------
  
  To add a new Suite to the full driver, you must make a couple of modifications.
  
  1)  Edit test_apr.h, and add the prototype for the function.
  2)  Edit testall.c, and add the function to the tests array.
  3)  Increase the NUM_TESTS macro in testall.c
  4)  Edit Makefile.in, and add the .lo file to the testall target.
  
  Once those four things are done, your tests will automatically be added
  to the suite.
  
  Writing tests
  -------------
  
  There are a couple of rules for writing good tests for the test suite.
  
  1)  All tests can determine for themselves if it passed or not.  This means
  that there is no reason for the person running the test suite to interpret
  the results of the tests.
  2)  Never use printf to add to the output of the test suite.  The suite
  library should be able to print all of the information required to debug
  a problem.
  3)  Functions should be tested with both positive and negative tests.  This
  means that you should test things that should both succeed and fail.
  
  An example test
  ---------------
  
  Finally, we will look at a quick test:
  
  /* All tests are passed a CuTest variable.  This is how the suite determines
   * if the test succeeded or failed.
   */
  static void test_localstr(CuTest *tc)
  {
      apr_status_t rv;
      apr_time_exp_t xt;
      time_t os_now;
  
      rv = apr_time_exp_lt(&xt, now);
      os_now = now / APR_USEC_PER_SEC;
     
      /* If the function can return APR_ENOTIMPL, then you should check for it.
       * This allows platform implementors to know if they have to implement
       * the function.
       */
      if (rv == APR_ENOTIMPL) {
          CuNotImpl(tc, "apr_time_exp_lt");
      }
  
      /* It often helps to ensure that the return code was APR_SUCESS.  If it
       * wasn't, then we know the test failed.
       */
      CuAssertTrue(tc, rv == APR_SUCCESS);
  
      /* Now that we know APR thinks it worked properly, we need to check the
       * output to ensure that we got what we expected.
       */
      CuAssertStrEquals(tc, "2002-08-14 12:05:36.186711 -25200 [257 Sat] DST",
                        print_time(p, &xt));
  }
  
  Notice, the same test can fail for any of a number of reasons.  The first 
  test to fail ends the test.
  
  CuTest
  ------
  
  CuTest is an open source test suite written by Asim Jalis.  It has been 
  released under the zlib/libpng license.  That license can be found in the
  CuTest.c and CuTest.h files.
  
  The version of CuTest that is included in the APR test suite has been modified
  from the original distribution in the following ways:
  
  1)  The original distribution does not have a -v flag, the details are always
  printed.
  2)  The NotImplemented result does not exist.
  3)  SuiteLists do not exist.  In the original distribution, you can add suites
  to suites, but it just adds the tests in the first suite to the list of tests
  in the original suite.  The output wasn't as detailed as I wanted, so I created
  SuiteLists.
  
  The first two modifications have been sent to the original author of CuTest,
  but they have not been integrated into the base distribution.  The SuiteList
  changes will be sent to the original author soon.
  
  The modified version of CuTest is not currently in any CVS or Subversion
  server.  In time, it will be hosted at rkbloom.net.
  
  There are currently no docs for how to write tests, but the teststr and 
  testtime programs should give an idea of how it is done.  In time, a document
  should be written to define how tests are written.
  
  
  
  

Mime
View raw message