Mailing-List: contact dev-help@asterixdb.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <F5B5DC38-73CD-4E6C-951A-D857E5BF384E@apache.org>
References: 
 <CAOQkTyPdihOH_-GHFyr1sXZfZOZh_7MKFRQ5QqO9yVOP04XQpg@mail.gmail.com>
 <CAHWyyRwoDFmDLbA4OxxfMWCHyqTW5Kadc-FTdmFfougaZheD6Q@mail.gmail.com>
 <F5B5DC38-73CD-4E6C-951A-D857E5BF384E@apache.org>
From: Ian Maxon <imaxon@uci.edu>
Date: Thu, 3 Dec 2015 15:28:40 -0800
Message-ID: 
 <CAN_YF5xjUry4GbQEV_eCjy0iTYH=to=M+qeuonsf1fPCrh26FA@mail.gmail.com>
Subject: Re: Undetected failed test cases
To: dev@asterixdb.incubator.apache.org
Content-Type: text/plain; charset=UTF-8

Definite +1.

We also should (separately) start checking the output of the CC/NC
logs or somehow otherwise find a way to detect exceptions that are
uncaught there. Right now if the exception doesn't come back to the
user as an error in issuing a query, we'd have no way to detect it.

On Thu, Dec 3, 2015 at 2:43 PM, Till Westmann <tillw@apache.org> wrote:
> +1 !
>
>
> On 3 Dec 2015, at 14:38, Chris Hillery wrote:
>
>> Yes, please propose the change. I've been looking at overhauling the test
>> framework as well so I will review.
>>
>> For Zorba, I implemented a "known failing" mechanism that allowed you to
>> mark a test that was currently broken (associated with a ticket ID)
>> without
>> disabling it. The framework would continue to execute it and expect it to
>> fail. It would also cause the test run to fail if the test started to
>> succeed (ie, the bug was fixed) which ensured that the "known failing"
>> mark
>> would get removed in a timely fashion. To be clear, this is completely
>> distinct from a negative test case - it was a way to not worry about
>> forgetting tests that had to be disabled due to known bugs, and to ensure
>> that all such known bugs had an associated tracking ticket. It was quite
>> useful there and I was planning to re-introduce it here.
>>
>> Ceej
>> aka Chris Hillery
>>
>> On Thu, Dec 3, 2015 at 2:29 PM, abdullah alamoudi <bamousaa@gmail.com>
>> wrote:
>>
>>> Hi All,
>>> Today, I implemented a fix for a critical issue that we have and wanted
>>> to
>>> add a new kind of test cases where the test case has 3 files:
>>>
>>> 1. Creating the dataset.
>>> 2. Fill it with data that have duplicate keys. This is expected to throw
>>> a
>>> duplicate key exception.
>>> 3. Delete the dataset. This is expected to pass (the bug was here where
>>> it
>>> is not being deleted).
>>>
>>> With the current way we use the test framework, we are unable to test
>>> such
>>> case and so I started to improve the test framework starting with
>>> actually
>>> checking the type of exception thrown and making sure that it matches the
>>> expected error.
>>>
>>> ... and boom. I found that many test cases fail but nobody notices
>>> because
>>> no one checks the type of exception thrown. Moreover, If a test is
>>> expected
>>> to fail and it doesn't, the framework doesn't check for that. In
>>> addition,
>>> sometimes the returned exception is meaningless and that is something we
>>> absolutely must avoid.
>>>
>>> What I propose is that I push to master the improved test framework and
>>> disable the failing test cases, create JIRA issues for them and assign
>>> each
>>> to someone to look at them.
>>>
>>> Thoughts?
>>>
>>> Amoudi, Abdullah.
>>>
>