accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Havanki <bhava...@cloudera.com>
Subject Re: Appropriate changes to increase test reliability
Date Tue, 22 Oct 2013 14:59:07 GMT
I'll look into our VM server farm some to see what can be done. It's super
overloaded, that's for sure.

The overridable timeout suggestion (thanks for that!) gave me an idea - I'm
playing with an option to add a multiplier to certain timeouts (the ones
bothering me) via the command line, i.e., run.py -f 3 to triple timeout
lengths. That can preserve the current timeout numbers in the tests but let
you scale them out for slower machines, VMs, etc.

Bill


On Mon, Oct 21, 2013 at 6:56 AM, Steve Loughran <stevel@hortonworks.com>wrote:

> On 18 October 2013 16:07, Bill Havanki <bhavanki@cloudera.com> wrote:
>
> > Hello all,
> >
> > We've often found while running Accumulo tests, at least under 1.4.x and
> > 1.5.x, that we need to tweak aspects of the tests to get them to run more
> > reliably. This is usually extensions to timeouts defined for how long
> tests
> > should be allowed to run, how long shutdowns are expected to take, and
> the
> > like.
> >
> > We suspect the need to make these changes arises from us running tests on
> > VMs, while the tests were perhaps originally aimed for execution on
> > dedicated physical resources.
> >
>
>
> it may just be slower machines: why not provide some overrideable timeouts.
>
>
> >
> > Should we provide patches for these sort of changes, so that others get
> > increased reliability? Or, conversely, does this indicate issues with our
> > test environments? How open are the tests to being changed for
> reliability?
> >
> >
>
> VMs have odd clock drift, especially if you don't have enough RAM to stop
> them being swapped out -time gets jerkier.
>
> Try adding more memory to your physical hosts, make sure they are getting
> host time not NTP, and that the TZs are consistent.
>
> remember: they're not real machines
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
// - - -
// Bill Havanki
// Solutions Architect, Cloudera
// - - -

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message