Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: 98.139.253.105 is neither permitted
 nor denied by domain of kihwal@yahoo-inc.com)
From: Kihwal Lee <kihwal@yahoo-inc.com>
To: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
        "yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
CC: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
        hwx-hdfs-dev <hwx-hdfs-dev@hortonworks.com>
Subject: Re: Policy on adding timeouts to tests
Thread-Topic: Policy on adding timeouts to tests
Thread-Index: AQHPWACM/SWRlag6R0SJf+hr+G5KTZsTeOAAgAI2OoCAAhZTAA==
Date: Fri, 18 Apr 2014 17:16:38 +0000
Message-ID: <CF76C049.1698C%kihwal@yahoo-inc.com>
References: 
 <CALwhT97iPsqm8-nXZh6Rjwr+1=33yLZ9EE3=Tw4UBexMk8=gtg@mail.gmail.com>
 <CABCYYb_tbEMS86XzkPnoNW9GHTNOs1qcUnDUjWGJiHBBtyVi9Q@mail.gmail.com>
 <A1F10FA3-E00E-45E3-8AF4-8BA5AB26A574@apache.org>
In-Reply-To: <A1F10FA3-E00E-45E3-8AF4-8BA5AB26A574@apache.org>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.4.1.140326
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A550C7B1E4814242A182D7A3FD9059BD@yforest.corp.yahoo.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

The most common mistake is making incorrect assumptions on test/component
run-time. We often forget how slower/faster things can be on different
platforms and how the load on the machine at the time of test execution
affect the run time.  If not sure, leave it to surefire. If things run
generally slow on certain environment and the 10 min timeout generates
many false positives, it is easier to adjust. But early detection of
failures is still preferred, if possible.

Some are using timeout for validating performance in unit tests.  This
should not be done in unit tests. Also, many test issues have been due to
incorrect assumptions on the ordering/timing of events and resulting state
updates. These test cases usually timeout waiting for certain state to be
reached.

Kihwal

On 4/16/14, 11:24 PM, "Vinod Kumar Vavilapalli" <vinodkv@apache.org> wrote:

>The other advantage of timeout is early failure - earlier than the uber
>10 min timeout that seems to exist in the build files. Usually the
>test-writer has a general idea of how long the test is supposed to run
>and if that doesn't happen, we can fail early. Clearly, this involves
>choosing a reasonable timeout so that the test can pass on local
>machines, different OSes and/or in VMs.
>
>+Vinod
>
>On Apr 15, 2014, at 11:37 AM, Chris Nauroth <cnauroth@hortonworks.com>
>wrote:
>
>> +common-dev, hdfs-dev
>>=20
>> My understanding of the current situation is that we had a period where
>>we
>> tried to enforce adding timeouts on all new tests in patches, but it
>>caused
>> trouble, and now we're back to not requiring it.  Jenkins test-patch
>>isn't
>> checking for it anymore.
>>=20
>> I don't think patches are getting rejected for using timeouts though.
>>=20
>> The difficulty is that execution time is quite sensitive to the build
>> environment.  (Consider top-of-the-line server hardware used in build
>> infrastructure vs. a dev running a VirtualBox VM with 1 dedicated CPU,
>>2 GB
>> RAM and slow virtualized disk.)  When we were enforcing timeouts, it was
>> quite common to see follow-up patches tuning up the timeout settings to
>> make tests work reliably in a greater variety of environments.  At that
>> point, the benefit of using the timeout becomes questionable, because
>>now
>> the fast machine is running with the longer timeout too.
>>=20
>> Chris Nauroth
>> Hortonworks
>> http://hortonworks.com/
>>=20
>>=20
>>=20
>> On Mon, Apr 14, 2014 at 9:41 AM, Karthik Kambatla
>><kasha@cloudera.com>wrote:
>>=20
>>> Hi folks
>>>=20
>>> Just wanted to check what our policy for adding timeouts to tests is.
>>>Do we
>>> encourage/discourage using timeouts for tests? If we discourage using
>>> timeouts for tests in general, are we okay with adding timeouts for a
>>>few
>>> tests where we explicitly want the test to fail if it takes longer
>>>than a
>>> particular amount of time?
>>>=20
>>> Thanks
>>> Karthik
>>>=20
>>=20
>> --=20
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or
>>entity to=20
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the
>>reader=20
>> of this message is not the intended recipient, you are hereby notified
>>that=20
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>>immediately=20
>> and delete it from your system. Thank You.
>
>
>--=20
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity
>to=20
>which it is addressed and may contain information that is confidential,
>privileged and exempt from disclosure under applicable law. If the reader
>of this message is not the intended recipient, you are hereby notified
>that=20
>any printing, copying, dissemination, distribution, disclosure or
>forwarding of this communication is strictly prohibited. If you have
>received this communication in error, please contact the sender
>immediately=20
>and delete it from your system. Thank You.