openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul Tripathi <>
Subject Re: BDD Test Cases Contribution to complement existing test cases and coverage
Date Mon, 13 Aug 2018 10:00:16 GMT
Hi Markus,

I am yet to get the permission to add the proposal to the OW Wiki but though of sharing it

What is Karate?
Karate is based on Cucumber which is a BDD framework.

What is BDD Testing?
BDD Testing is a testing approach based on Behavioural Driven Development. It uses Given ,When
,Then statements to create test scenario's. Example:

Feature:  Get List of actions based on the NameSpace

Scenario: As a user I want to get the list of actions available for the given namespace
    * def path = '/api/v1/namespaces/'+nameSpace+'/actions?limit=30&skip=0'
    Given url BaseUrl+path
    And header Authorization = Auth
    And header Content-Type = 'application/json'
    When method get
    Then status 200
    And def json = response
 The above example tests that the List Actions API's is returning a 200 ok . A lot more assertions
can be put as per the requirements. Karate has some very good in built functions for 
 asserting the nested JSON's
TEST CASE-2(This example shows a simple smoke test on all the wsk user functions)

Feature: This feature file will test all the wsk functions

    * configure ssl = true
    * def nameSpace = 'guest'
    * def params = '?blocking=true&result=false'
    * def scriptcode = call read('classpath:com/karate/openwhisk/functions/hello-world.js')
    * def base64encoding = read('classpath:com/karate/openwhisk/utils/base64.js')

  Scenario: TC01-As a user I want to all the wsk functions available to the user and check
if they give the proper response
    # Get User Auth
    * def getNSCreds = call read('classpath:com/karate/openwhisk/wskadmin/get-user.feature')
    * def result = getNSCreds.result
    * def Auth = base64encoding(result)
    * print "Got the Creds for the guest user"
    * print Auth
    # Create an Action .Create an action for the above defined guest name
    #* def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)', actionName:'Dammyyy'}
    * def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
    * def actionName = createAction.actName
    * print actionName
    * print "Successfully Created an action"
    # Get Action Details
    * def actionDetails = call read('classpath:com/karate/openwhisk/wskactions/get-action.feature')
{nameSpace:'#(nameSpace)' ,Auth:'#(Auth)',actionName:'#(actionName)'}
    * print "Successfully got the action details"
    #Invoke Action
    * def invokeAction = call read('classpath:com/karate/openwhisk/wskactions/invoke-action.feature')
{params:'#(params)',requestBody:'',nameSpace:'#(nameSpace)' ,Auth:'#(Auth)',actionName:'#(actionName)'}
    * def actID = invokeAction.activationId
    * print  = "Successfully invoked the action"
    * def webhooks = callonce read('classpath:com/karate/openwhisk/utils/sleep.feature') {sheepCount:'20'}
    #Get Activation details
    * def getActivationDetails = call read('classpath:com/karate/openwhisk/wskactions/get-activation-details.feature')
{ activationId: '#(actID)' ,Auth:'#(Auth)'}
    * print "Successfully pulled the activation details"
    # Update Action
    * def updateAction = call read('classpath:com/karate/openwhisk/wskactions/update-action.feature')
{actionName:'#(actionName)',script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
    * def actionName = createAction.actName
    * print actionName
    * print "Successfully updated the action"
    # List Action
    * def listActions = call read('classpath:com/karate/openwhisk/wskactions/list-action.feature')
{nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
    * print "Successfully pulled up the list of actions"
    # Delete Action
    * def deleteAction = call read('classpath:com/karate/openwhisk/wskactions/delete-action.feature')
{actionName:'#(actionName)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
    * print "Successfully deleted the action"

Some key features of BDD testing approach:

1) Shifting from thinking in “tests” to thinking in “behavior”
2) Collaboration between Business stakeholders, Business Analysts, QA Team and developers
3) Ubiquitous language, it is easy to describe
4) Driven by Business Value
5) Extends Test Driven Development (TDD) by utilizing natural language that non technical
stakeholders can understand
6) It enables to create test cases even before development starts and also helps bridging
the Gap between what was required and what was actually delivered.

Why we should consider doubling our test-coverage (and thus effort)?
As OpenWhisk is an opensource Serverless Platform, it offers user a wide range of use cases.
If we implement a BDD Testing Framework it would give us an ability to create
various test flows/scenarios  which mimics the user behaviour. This might help us uncover
issues which we might not catch in Unit/System level tests.
Also as writing tests in Karate does not require in-depth coding knowledge we can expect a
lot more contributions in terms of testing. While our existing test cases can serve as Unit/System
levels tests,
the Karate Framework could focus more on testing Business Logics/Work Flows.
Also,now that Karate offers Gatling support we can utilize the test feature files for Performance
Test Scenarios.

How do these tests relate to the other system tests we have written in Scala
We have already covered most of the test cases in the Karate Framework and are in Progress
to create the other test cases as well. Also we are creating units/modules derived from the
Scala based Tests
which is a one time effort as those units can be utilised in any of the test scenarios in
Also keeping in mind a testers perspective we have structured/grouped the tests in the way
it can be tested. Like Smoke Tests(These would just contain the basic tests which would ensure
only that that
system is working fine, Regression Tests (Covering In-depth Testing)

Below is the relation between a test case written in Scala and Karate for your reference .The
two tests give the same result.

Test Case from (

  behavior of "Wsk REST"

  it should "reject creating duplicate entity" in withAssetCleaner(wskprops) { (wp, assetHelper)
    val name = "testDuplicateCreate"
    assetHelper.withCleaner(wsk.trigger, name) { (trigger, _) =>
    assetHelper.withCleaner(wsk.action, name, confirmDelete = false) { (action, _) =>
      action.create(name, defaultAction, expectedExitCode = Conflict.intValue)
 Test Case from (

    Scenario: As a user i want to verify the creation of duplicate entity
    #create action.
    * def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
    * def actionName = createAction.actName
    * match createAction.responseStatusCode == 200
    * print "Successfully Created an action"
    #recreate the duplicate action with the same name
    * def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)' ,actionName:'#(actionName)'}
    * def actionName = createAction.actName
    * match createAction.responseStatusCode == 409
    * match createAction.response.error == "resource already exists"

Also if needed we will be happy to give a demo in the OW Bi-Weekly meeting.

On 13/08/18, 12:34 PM, "Markus Thömmes" <> wrote:

    Hi Rahul,
    thanks a lot for the contribution and the work you've put into this!
    Very much appreciated.
    I'm not familiar with Karate or BDD at all. Could you go into a little
    bit more detail on what the benefit of this approach are and why we
    should consider doubling our test-coverage (and thus effort)? How do
    these tests relate to the other system tests we have written in Scala
    Am Mo., 13. Aug. 2018 um 07:55 Uhr schrieb Rahul Tripathi
    > Hi All,
    > To completed the existing OpenWhisk Test I have raised this pull request which has
Karate Based BDD test cases along with Gatling Support for performance testing. The idea is
to enable the testers to contribute the user workflows in the form of BDD scenarios.
    > Request you to kindly review and provide your valuable feedback.
    > Thanks,
    > Rahul
    > Thanks,
    > Rahul

View raw message