Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Thu, 14 Jan 2016 21:41:39 +0000 (UTC)
From: "Jason Altekruse (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.12929242.1452534208000.114710.1452807699939@Atlassian.JIRA>
In-Reply-To: <JIRA.12929242.1452534208000@Atlassian.JIRA>
References: <JIRA.12929242.1452534208000@Atlassian.JIRA>
 <JIRA.12929242.1452534208538@arcas>
Subject: [jira] [Comment Edited] (DRILL-4259) Add new functional tests to
 ensure that failures can be detected independent of the testing environment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/DRILL-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094201#comment-15094201 ] 

Jason Altekruse edited comment on DRILL-4259 at 1/14/16 9:41 PM:
-----------------------------------------------------------------

That's what I was thinking, although I'm not very familiar with the setup of the regression tests. I know that some tests set options themselves by just including the alter sessions statements in the test files. I also know that we added an enhancement for the framework to set options globally, so that we could leave the decimal tests turned on [1] (even though we turned of the feature by default due to some lingering pref and correctness issues).

I think what I would be looking to do would be configure a few different combinations of configurations that could be applied over an entire suite, so that we can run a number of tests with a few different configurations. Doing this by manually changing the test files to include near copies with the new options set doesn't seem like the right way to go. Let me know if this seems like a reasonable approach. I can work on the enhancement to the framework to enable it if necessary.

[1] - https://github.com/mapr/drill-test-framework/blob/master/framework/before-run.sql


was (Author: jaltekruse):
That's what I was thinking, although I'm not very familiar with the setup of the regression tests. I know that some tests set options themselves by just including the alter sessions statements in the test files. I also know that we added an enhancement for the framework to set options globally, so that we could leave the decimal tests turned on [1] (even though we turned of the feature by default due to some lingering pref and correctness issues).

I think what I would be looking to do would be configure a few different combinations of configurations that could be applied over an entire suite, so that we can run a number of tests with a few different configurations. Doing this by manually changing the test files to include near copes with the new options set doesn't seem like the right way to go. Let me know if this seems like a reasonable approach. I can work on the enhancement to the framework to enable it if necessary.

[1] - https://github.com/mapr/drill-test-framework/blob/master/framework/before-run.sql

> Add new functional tests to ensure that failures can be detected independent of the testing environment
> -------------------------------------------------------------------------------------------------------
>
>                 Key: DRILL-4259
>                 URL: https://issues.apache.org/jira/browse/DRILL-4259
>             Project: Apache Drill
>          Issue Type: Test
>            Reporter: Jason Altekruse
>
> In DRILL-4243 an out of memory issue was fixed after a change to the memory allocator made memory limits more strict. While the regression tests had been run by the team at Dremio prior to merging the patch, running the tests on a cluster with more cores changed the memory limits on the queries and caused several tests to fail.
> While changes of this magnitude are not going to be common, we should have a test suite that reliably fails independent of the environment it is run (assuming that there are sufficient resources for the tests to run).
> It would be good to at least try to reproduce this failure on a few different setups (cores, nodes in cluster) by adjusting available configuration options and adding tests with those different configurations so that the tests will fail in different environments.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)