yunikorn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ayub Pathan <ayubpat...@apache.org>
Subject Re: [Discussion] Add integration testing framework and cases
Date Tue, 21 Apr 2020 00:28:09 GMT
Thanks Tao for initiating this thread, I am super excited to be part of
this discussion.

Here are my two cents..

Framework:

   1. Consider adding something like kubeconfigManager, as this allows
   extensibility to work with any cluster(not just with the one running
   locally).
   2. If I have understood correctly, the nature of the framework is
   behavior driven. If you agree, consider something which is Ginkgo
   <https://onsi.github.io/ginkgo/>. It provides many
   inbuilt extensions and third party integrations.
   3. Consider using Viper <https://github.com/spf13/viper> for
   configuration management, as this helps setting defaults, reading/writing
   config files, fileformat conversions, environment variables management etc.

I have some more thoughts around test cases as well. Let us set up a
meeting to discuss these points in detail, as Weiwei suggested.

Thanks
Ayub Khan

On Sun, Apr 19, 2020 at 9:08 PM Tao Yang <ytaofighting@gmail.com> wrote:

> Thanks Weiwei for the comments.
>
> Of course we should have some verification results in cases, my initial
> thought is to execute cases separately but your comments made me realized
> that maybe it's better to manage them together and provide functions as
> following:
> (1) We can keep loosely coupled relationships between framework and cases,
> for example, a new case can be registered into the framework and with
> specified type (such as benchmark, smoke, or chaos-monkey) and tags (like
> 'constraints', 'performance', 'fairness', 'function' ...), and provide an
> common entrance with which users can decide which cases to be included:
> all, specified type or tags, or specified one.
> (2) Each cases can return verification results (such as "All requirements
> of app1 are satisfied":succeed) to the framework and output useful
> information (which may helps to show execution process or locate the cause
> of a failure) into log file. Chaos-monkey tests can be managed in one case
> with type 'chaos-monkey' which has its own structure and can reuse tools in
> the framework.
> (3) After all desired test cases are done, a testing report with details of
> these cases can be generated on demand, an example can be:
> ```
> Case: allocation
> Tags: function
> Status: Failed
> Verifications:
>    Allocate pods with node constraint: Succeed
>    Allocate pods with affinity constraint: Succeed
>    Allocate pods with anti-affinity constraint: Failed
> ```
>
> Above is my rough idea for that, hope to hear your thoughts.
>
> Thanks,
> Tao
>
> Weiwei Yang <wwei@apache.org> 于2020年4月18日周六 上午5:44写道:
>
> > Hi Tao
> >
> > This is great, we are super interested.
> > Let's track this effort with
> > https://issues.apache.org/jira/browse/YUNIKORN-33.
> > A couple of comments
> >
> >    1. There might be one thing missing is, how we can make this
> >    sustainable as "tests". which means we can 1) run this easily, 2)
> > generate
> >    a result, and then 3) verify the result. Do we have #3 now? If not, we
> > need
> >    to think about how to design a proper algorithm to verify if each
> > scenario
> >    passes.
> >    2. Another thing to add is the chaos monkey tests. This can be phase-2
> >    but it might be good we can consider this now, to make sure our
> > structure
> >    can support that.
> >
> > We can set up some meetings next week to discuss this in more detail.
> > Thanks
> >
> > On Fri, Apr 17, 2020 at 12:49 AM Tao Yang <ytaofighting@gmail.com>
> wrote:
> >
> > > Hello everyone
> > >
> > > We are planning to add integration testing framework and initial test
> > cases
> > > in https://issues.apache.org/jira/browse/YUNIKORN-29, general thoughts
> > are
> > > as follows.
> > >
> > > Basic testing framework includes:
> > > 1. AppInfo struct: define basic information, requests and status of an
> > > application
> > > 2. AppManager interface and its implementations like
> > DeploymentAppManager:
> > > support useful operations like create/delete/refresh(status)/wait(for
> > apps
> > > to be satisfied or cleaned up) for testing.
> > > 3. CommonConfig struct: keep several common configurations for testing
> > like
> > > schedulerName, outputRootPath, namespace, queue etc.
> > > 4. Output tools like chart painter and file writer with specific format
> > > (like csv).
> > >
> > > Initial test cases:
> > > 1. Throughput : Request a certain number of pods via different
> schedulers
> > > and then record the distributions of scheduled pods, draw results of
> > > different schedulers on the same chart or write results into a file.
> > > 2. Node Fairness: Request a certain number of pods via different
> > schedulers
> > > and then record the distributions of node usage, draw results on charts
> > > separately for different schedulers and write results into a file.
> > > 3. Queue Fairness (Only for YuniKorn since K8s scheduler doesn't
> support
> > it
> > > yet): Prepare queues with different capacities, request pods with
> > different
> > > number or resource for these queues, then record the usage of queues,
> > draw
> > > results on the same chart and write results into a file.
> > >
> > > Implementation:
> > > 1. Package structure: add test/integration package in yunikorn-k8shim
> > > project, the source files with structures or tools of testing framework
> > > will be directly putted in this package, and test cases will be
> > seperately
> > > managed by case package in sub-directories: cases/<special-case> such
> as
> > > cases/throughput, cases/node-fairness, cases/queue-fairness
> > > 2. Configurations:  cases keep their specific configurations themselves
> > and
> > > all configurations can be maintained in a single yaml file together
> with
> > > common configurations, the config structure like this:
> > > ```
> > > common:
> > >   schedulerName: yunikorn
> > >   maxWaitSeconds: 60
> > >   queue: root.default
> > >   namespace: default
> > >   outputRootPath: /tmp
> > >
> > > cases:
> > >   throughput:
> > >     schedulerNames:
> > >       - yunikorn
> > >       - default-scheduelr
> > >     appName: throughput-test
> > >     numPods: 10000
> > >     ...
> > >   node-fairness:
> > >     ...
> > >   queue-fairness:
> > >     ...
> > > ```
> > > 3. Executions: all cases have main function themselves and need one
> > > command-line argument: configPath, so that we can directly run
> specified
> > > case and easily perform integration testing on different scenarios.
> > >
> > > Any comments and suggestions are welcome!
> > >
> > > Thanks,
> > > Tao
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message