ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artem Shutak <ashu...@gridgain.com>
Subject Re: Full API coverage enhancement
Date Mon, 08 Feb 2016 14:45:33 GMT
Sergey,

I think we should start more caches, like 1000 in one time. But we have to
have enough memory on our TC agents. As I know, empty cache is require
about 50 mb (without indexing), am I right?

You are right, I keep in mind that *backups* and *REPLICATED* mode make no
sense together, but we still have to test it in one node and multi node
cases—é

Any other *no sense* combinations?

I forgot about custom BinaryConfiguration at IgniteConfiguration for
BinaryMarshaller. So, at least 6 IgniteConfigurations.

-- Artem --

On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <skozlov@gridgain.com> wrote:

> Hi Artem
>
> It's good idea to create 20-30 cache configurations at once and then to
> iterate tests over those caches in parallel (but make sure that cache names
> are unique).
> Another point that some combinations make no sense like *backups *and
> *REPLICATED
> *cache
>
>
> On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <ashutak@gridgain.com> wrote:
>
> > Hi all,
> >
> > I have an update.
> >
> > I've started from *CacheConfiguration* permutations. I wrote out a list
> > with all CacheConfiguration setters and filtered it with Alexey G.
> >
> > Finally we have:
> >
> >    1. CacheMode - 3 variants
> >    2. CacheAtomicityMode - 2 variants
> >    3. CacheMemoryMode - 3 variants
> >    4. setLoadPreviousValue - 2 variants
> >    5. setReadFromBackup - 2 variants
> >    6. setStoreKeepBinary - 2 variants
> >    7. setRebalanceMode - SYNC and ASYNC (2 variants)
> >    8. setSwapEnabled - 2 variants
> >    9. setCopyOnRead - 2 variants
> >    10. NearConfiguration disabled / default NearConfiguration / custom
> >    NearConfiguration - 3 variants
> >    11. With and without a complex parameter. The complex parameter
> defines
> >    not-default Eviction policy and filter, cache store configuration
> >    (storeFactory and storeSessionListenerFactory), rebalancing
> > configuration,
> >    affinity function, offHeapMaxMemory, interceptor, topology validator
> and
> >    CacheEntryListener.
> >
> > I've run 123 Cache Full Api test cases for all permutations of parameters
> > 1-9 and got 256896 test cases (1152 configuration variants * 123 test
> > cases). All these tests take 4 hours 40 minutes. Not all tests pass, so
> MAY
> > BE when all tests will pass it will take less time (3,5 hours for
> example).
> >
> > As we can see the tests take a lot of time.
> >
> > The following permutation should be supported too:
> >
> >    1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1
> >    backups, 4 nodes and 2 backups - 3 variants
> >    2. Client and Server nodes - 2 variants
> >    3. Indexing enabled and disabled for cache - 2 variants
> >    4. IgniteConfiguration permutations - how many variants? I see at
> least
> >    4 (2 Marshallers, P2P).
> >
> > Plus we need add new test cases to test different key and value types and
> > etc.
> >
> > So, we need multiply more then 3,5-4,5 hours on ~250. If we will split
> all
> > tests on 250 suites and run on all 30 TC agents it will take about 30-40
> > hours. Ok, we can do it during weekends.
> >
> > I think it will take too much time.
> >
> > As an option we can start a cache for each configuration and run tests
> > concurrently. But we need to implement this opportunity in our test
> > framework.
> >
> > Any other thoughts how we can decrease time for tests?
> >
> > Thanks,
> > -- Artem --
> >
> > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <sboikov@gridgain.com>
> > wrote:
> >
> > > Artem,
> > >
> > > One more thing for new tests: I think test should start both server and
> > > client nodes and use Ignite API from all nodes.
> > >
> > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <ashutak@gridgain.com>
> > wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > Actually, I don't have a list with all the permutations.
> > > >
> > > > At first, we need to split in our discussion test cases and Ignite
> > > > configuration which should be covered.
> > > >
> > > > For example, new Full Api test cases for cache are based on old Full
> > Api
> > > > test cases. So, it need to think what the test cases was not covered
> > > > before.
> > > >
> > > > About Ignite configurations, I'm going to add permutation for each
> > > > IgniteConfiguration and CacheConfiguration property.
> > > >
> > > > By the way, the jira contains the following list of permutation (feel
> > > free
> > > > to add something):
> > > >
> > > > The following tests should be added (for functional blocks):
> > > >
> > > >    1. Interceptor
> > > >    2. Queries: continuous, scan, SQL, fields and text queries.
> > > >    3. cache events
> > > >    4. We should also test with Serializable, Externalizable, and
> plain
> > > >    Pojos for keys and values.
> > > >    5. The Pojo in the above test should contain an enum value
> > > >    6. We should also test Enums as keys and Enums as values
> > > >    7. All operations should have single-key and multi-key operations
> > > >
> > > > New tests should cover all combinations for following properties:
> > > >
> > > >    1. cache modes
> > > >    2. operation from client nodes and server nodes
> > > >    3. store enabled/disabled
> > > >    4. evicts sycn/non-sync
> > > >    5. eviction policies
> > > >    6. near on/off
> > > >    7. marshallers (+ Binary marshaller with different mappers)
> > > >    8. keys and values - externalizable, serializable, binaryzable,
> > "none
> > > of
> > > >    previous"
> > > >    9. classes available on servers: true/false
> > > >    10. Peer loading on/off
> > > >    11. Affinity functions
> > > >    12. expiry policies
> > > >
> > > >
> > > > Thanks,
> > > > -- Artem --
> > > >
> > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Artem, I think it is best to specify all the permutations here, so
> > > others
> > > > > can make additional suggestions. Otherwise, we cannot get a full
> > > picture.
> > > > >
> > > > > Thanks,
> > > > > D.
> > > > >
> > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <ashutak@gridgain.com
> >
> > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > I thought a little bit more and think we need to add a support
> for
> > > the
> > > > > > following permutations too (I've added these to the jira
> > > description):
> > > > > > - We should also test with Serializable, Externalizable, and
> plain
> > > > Pojos
> > > > > > for keys and values.
> > > > > > - The Pojo in the above test should contain an enum value
> > > > > > - We should also test Enums as keys and Enums as values
> > > > > > - All operations should have single-key and multi-key operations
> > > > > >
> > > > > > Maybe someone see any other permutation to be supported?
> > > > > >
> > > > > > -- Artem --
> > > > > >
> > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <
> > ashutak@gridgain.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Dmitriy,
> > > > > > >
> > > > > > > There is a branch at my fork and a pull request at Ignite.
See
> > > > comment
> > > > > > > about pull request at the ticket (PR-446).
> > > > > > >
> > > > > > > But I have to notice that the branch under hard development
and
> > you
> > > > it
> > > > > > can
> > > > > > > not work (have compilation or test errors) at some moments.
> > > > > > >
> > > > > > > Good luck!
> > > > > > >
> > > > > > > -- Artem --
> > > > > > >
> > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > > > > dsetrakyan@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Artem,
> > > > > > >>
> > > > > > >> This is great. I have noticed from the ticket that
you have
> > > created
> > > > > some
> > > > > > >> initial suite already. Is there a branch I can look
at it?
> > > > > > >>
> > > > > > >> D.
> > > > > > >>
> > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <
> > > ashutak@gridgain.com
> > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Igniters,
> > > > > > >> >
> > > > > > >> > I'm working on an enhancement of Full API coverage
[1] [2].
> > > > > > >> >
> > > > > > >> > Ignite already has Full API test, but currently
it's hard to
> > > test
> > > > > all
> > > > > > >> > configuration combinations.
> > > > > > >> >
> > > > > > >> > Feel free to add comments in the jira if you have
any
> thought.
> > > > > > >> >
> > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > > > >> > [2]
> > > > > > >>
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > -- Artem --
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message