openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Gencur <mgen...@redhat.com>
Subject Re: BDD Test Cases Contribution to complement existing test cases and coverage
Date Tue, 14 Aug 2018 11:29:37 GMT
Hi Rahul/all,
this is certainly an interesting approach to testing. Let me mention 
some points.

The test structure is more flat - either all commands at the same level 
or at most one/two calls to "features" in separate files. This gives the 
user more insight into all details of the request/response. However, it 
is questionable is the test itself is more readable than the Scala code. 
The tests are a mix of Gherkin Feature file commands, embedded Java 
snippets, and JavaScript as opposed to just Scala in the current test suite.

The tests in the Scala test suite are already written in the way: 
"Component XY should do this and that" which is close to BDD.

I guess the most important point is whether OpenWhisk does BDD which is 
really a style of development where behaviour is defined first as well 
as tests with exact specification and coding follows. Does OpenWhisk 
development work in this way or want to work this way? Then these tests 
might be useful, especially for new features and test cases. I don't see 
so much value in re-writing the existing tests because the community is 
familiar with them and not sure how many people are familiar with the 
Karate framework.

Perhaps other questions are:
* is there any code completion for the features files in some IDEs? 
Especially for the embedded code snippets.
* do the tests run more quickly or slower than the current scala test suite?

Cheers,
Martin Gencur
QE, Red Hat

On 13.8.2018 12:00, Rahul Tripathi wrote:
> Hi Markus,
>
> I am yet to get the permission to add the proposal to the OW Wiki but though of sharing
it here:
>
> What is Karate?
> Karate is based on Cucumber which is a BDD framework.
>
> What is BDD Testing?
> BDD Testing is a testing approach based on Behavioural Driven Development. It uses Given
,When ,Then statements to create test scenario's. Example:
>
>
> TEST CASE-1
> Feature:  Get List of actions based on the NameSpace
>
> Scenario: As a user I want to get the list of actions available for the given namespace
>      * def path = '/api/v1/namespaces/'+nameSpace+'/actions?limit=30&skip=0'
>      Given url BaseUrl+path
>      And header Authorization = Auth
>      And header Content-Type = 'application/json'
>      When method get
>      Then status 200
>      And def json = response
>      
>   The above example tests that the List Actions API's is returning a 200 ok . A lot more
assertions can be put as per the requirements. Karate has some very good in built functions
for
>   asserting the nested JSON's
>   
>   
> TEST CASE-2(This example shows a simple smoke test on all the wsk user functions)
>
> Feature: This feature file will test all the wsk functions
>
>    Background:
>      * configure ssl = true
>      * def nameSpace = 'guest'
>      * def params = '?blocking=true&result=false'
>      * def scriptcode = call read('classpath:com/karate/openwhisk/functions/hello-world.js')
>      * def base64encoding = read('classpath:com/karate/openwhisk/utils/base64.js')
>
>    Scenario: TC01-As a user I want to all the wsk functions available to the user and
check if they give the proper response
>      # Get User Auth
>      * def getNSCreds = call read('classpath:com/karate/openwhisk/wskadmin/get-user.feature')
{nameSpace:'#(nameSpace)'}
>      * def result = getNSCreds.result
>      * def Auth = base64encoding(result)
>      * print "Got the Creds for the guest user"
>      * print Auth
>      
>      # Create an Action .Create an action for the above defined guest name
>      #* def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)', actionName:'Dammyyy'}
>      * def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
>      * def actionName = createAction.actName
>      * print actionName
>      * print "Successfully Created an action"
>      
>      # Get Action Details
>      * def actionDetails = call read('classpath:com/karate/openwhisk/wskactions/get-action.feature')
{nameSpace:'#(nameSpace)' ,Auth:'#(Auth)',actionName:'#(actionName)'}
>      * print "Successfully got the action details"
>      
>      #Invoke Action
>      * def invokeAction = call read('classpath:com/karate/openwhisk/wskactions/invoke-action.feature')
{params:'#(params)',requestBody:'',nameSpace:'#(nameSpace)' ,Auth:'#(Auth)',actionName:'#(actionName)'}
>      * def actID = invokeAction.activationId
>      * print  = "Successfully invoked the action"
>      * def webhooks = callonce read('classpath:com/karate/openwhisk/utils/sleep.feature')
{sheepCount:'20'}
>      
>      #Get Activation details
>      * def getActivationDetails = call read('classpath:com/karate/openwhisk/wskactions/get-activation-details.feature')
{ activationId: '#(actID)' ,Auth:'#(Auth)'}
>      * print "Successfully pulled the activation details"
>      
>      # Update Action
>      * def updateAction = call read('classpath:com/karate/openwhisk/wskactions/update-action.feature')
{actionName:'#(actionName)',script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
>      * def actionName = createAction.actName
>      * print actionName
>      * print "Successfully updated the action"
>      
>      # List Action
>      * def listActions = call read('classpath:com/karate/openwhisk/wskactions/list-action.feature')
{nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
>      * print "Successfully pulled up the list of actions"
>      
>      # Delete Action
>      * def deleteAction = call read('classpath:com/karate/openwhisk/wskactions/delete-action.feature')
{actionName:'#(actionName)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
>      * print "Successfully deleted the action"
>
>   
> Some key features of BDD testing approach:
>
> 1) Shifting from thinking in “tests” to thinking in “behavior”
> 2) Collaboration between Business stakeholders, Business Analysts, QA Team and developers
> 3) Ubiquitous language, it is easy to describe
> 4) Driven by Business Value
> 5) Extends Test Driven Development (TDD) by utilizing natural language that non technical
stakeholders can understand
> 6) It enables to create test cases even before development starts and also helps bridging
the Gap between what was required and what was actually delivered.
>
>
> Why we should consider doubling our test-coverage (and thus effort)?
> As OpenWhisk is an opensource Serverless Platform, it offers user a wide range of use
cases. If we implement a BDD Testing Framework it would give us an ability to create
> various test flows/scenarios  which mimics the user behaviour. This might help us uncover
issues which we might not catch in Unit/System level tests.
> Also as writing tests in Karate does not require in-depth coding knowledge we can expect
a lot more contributions in terms of testing. While our existing test cases can serve as Unit/System
levels tests,
> the Karate Framework could focus more on testing Business Logics/Work Flows.
> Also,now that Karate offers Gatling support we can utilize the test feature files for
Performance Test Scenarios.
>
>
> How do these tests relate to the other system tests we have written in Scala
> today?
> We have already covered most of the test cases in the Karate Framework and are in Progress
to create the other test cases as well. Also we are creating units/modules derived from the
Scala based Tests
> which is a one time effort as those units can be utilised in any of the test scenarios
in future.
> Also keeping in mind a testers perspective we have structured/grouped the tests in the
way it can be tested. Like Smoke Tests(These would just contain the basic tests which would
ensure only that that
> system is working fine, Regression Tests (Covering In-depth Testing)
>
>
> Below is the relation between a test case written in Scala and Karate for your reference
.The two tests give the same result.
>
> Test Case from (https://github.com/apache/incubator-openwhisk/blob/master/tests/src/test/scala/system/basic/WskRestBasicTests.scala)
>
>    behavior of "Wsk REST"
>
>    it should "reject creating duplicate entity" in withAssetCleaner(wskprops) { (wp,
assetHelper) =>
>      val name = "testDuplicateCreate"
>      assetHelper.withCleaner(wsk.trigger, name) { (trigger, _) =>
>        trigger.create(name)
>      }
>      assetHelper.withCleaner(wsk.action, name, confirmDelete = false) { (action, _) =>
>        action.create(name, defaultAction, expectedExitCode = Conflict.intValue)
>      }
>    }
>    
>    
>    
>   Test Case from (https://github.com/apache/incubator-openwhisk/blob/9d87202a9f91d6274b1922dd125fc321d1d92a3a/tests/functional/karate_tests/src/test/java/com/karate/openwhisk/actionbasictests/action-basic-tests.feature)
>    
>      Scenario: As a user i want to verify the creation of duplicate entity
>      #create action.
>      * def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
>      * def actionName = createAction.actName
>      * match createAction.responseStatusCode == 200
>      * print "Successfully Created an action"
>      
>      #recreate the duplicate action with the same name
>      * def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)' ,actionName:'#(actionName)'}
>      * def actionName = createAction.actName
>      * match createAction.responseStatusCode == 409
>      * match createAction.response.error == "resource already exists"
>
> Also if needed we will be happy to give a demo in the OW Bi-Weekly meeting.
>      
>
> Thanks,
> Rahul
>   
>   
> On 13/08/18, 12:34 PM, "Markus Thömmes" <markusthoemmes@apache.org> wrote:
>
>      Hi Rahul,
>      
>      thanks a lot for the contribution and the work you've put into this!
>      Very much appreciated.
>      
>      I'm not familiar with Karate or BDD at all. Could you go into a little
>      bit more detail on what the benefit of this approach are and why we
>      should consider doubling our test-coverage (and thus effort)? How do
>      these tests relate to the other system tests we have written in Scala
>      today?
>      
>      Cheers,
>      Markus
>      Am Mo., 13. Aug. 2018 um 07:55 Uhr schrieb Rahul Tripathi
>      <rtripath@adobe.com.invalid>:
>      >
>      > Hi All,
>      >
>      > To completed the existing OpenWhisk Test I have raised this pull request which
has Karate Based BDD test cases along with Gatling Support for performance testing. The idea
is to enable the testers to contribute the user workflows in the form of BDD scenarios.
>      > Request you to kindly review and provide your valuable feedback.
>      >
>      >
>      > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fpull%2F3956&amp;data=02%7C01%7Crtripath%40adobe.com%7C155132631fa64e50d66b08d600eaf157%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636697406398620244&amp;sdata=%2BdrXbT9yi6IHPcUr4eCfWicKiLfLtDsCecxxGqpLimw%3D&amp;reserved=0
>      >
>      > Thanks,
>      > Rahul
>      >
>      >
>      >
>      >
>      > Thanks,
>      > Rahul
>      >
>      >
>      
>


Mime
View raw message