Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201708010100.v7110aPS029778@ip-10-146-233-104.ec2.internal>
Date: Tue, 1 Aug 2017 01:00:36 +0000
From: "Henry Robinson (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org
Reply-To: henry@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_=28PREVIEW=29_IMPALA-5684=3A_Optionally_run_be_tests_in_sharded_mode=0A?=
In-Reply-To: <gerrit.1500511058000.I865db25b07728f3886133316ded5122c60490967@gerrit.cloudera.org>
References: <gerrit.1500511058000.I865db25b07728f3886133316ded5122c60490967@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.7
archived-at: Tue, 01 Aug 2017 01:00:54 -0000

Henry Robinson has uploaded a new patch set (#2).

Change subject: (PREVIEW) IMPALA-5684: Optionally run be tests in sharded mode
......................................................................

(PREVIEW) IMPALA-5684: Optionally run be tests in sharded mode

Googletest supports sharded execution, where multiple independent
processes can run a subset of a test suite in parallel on the same
machine or different machines. This patch adds preliminary support for
doing so for some of our backend tests, where the parallelism speedup is
worth the extra overhead.

How we do this: for every backend test 'foo', add a new test target in
CMake, called 'foo_sharded'. That target executes a wrapper script,
bin/run-sharded-test.sh, which runs N copies of the test in parallel,
with the environment correctly set up to enable sharding. You can run a
sharded test like this:

ctest -R "foo_sharded"

Then, in bin/run-backend-tests.sh, keep a whitelist of tests that can be
successfully run in sharded mode (i.e. there are no known conflicts from
running multiple copies at once, and the speedup is worth the
overhead). Use that list to run ctest twice, once in serial mode with no
sharded tests (using -E to exclude those tests) and once in sharded mode
running only the whitelisted sharded tests.

Each shard process logs to its own directory 'foo_test_shard_N' under
the usual logs/be_tests/ root.

On my desktop machine, this improves expr-test runtime from ~10 minutes
to about 5 minutes. More importantly, this opens up more opportunity for
easy latency improvements if we break the large test cases into smaller
ones that are parallelisable.

Change-Id: I865db25b07728f3886133316ded5122c60490967
---
M be/CMakeLists.txt
M bin/run-backend-tests.sh
A bin/run-sharded-test.sh
3 files changed, 98 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/7467/2
-- 
To view, visit http://gerrit.cloudera.org:8080/7467
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I865db25b07728f3886133316ded5122c60490967
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <henry@cloudera.com>