From dev-return-39565-archive-asf-public=cust-asf.ponee.io@subversion.apache.org Mon Oct 14 15:33:56 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3EE6B180648 for ; Mon, 14 Oct 2019 17:33:56 +0200 (CEST) Received: (qmail 85964 invoked by uid 500); 14 Oct 2019 15:33:55 -0000 Mailing-List: contact dev-help@subversion.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@subversion.apache.org Received: (qmail 85953 invoked by uid 99); 14 Oct 2019 15:33:55 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.42) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Oct 2019 15:33:55 +0000 Received: from [10.3.6.38] (unknown [213.143.10.133]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id E5D82598D; Mon, 14 Oct 2019 15:33:54 +0000 (UTC) Subject: Re: Make check with different client and server versions To: Nathan Hartman References: From: Julian Foad Cc: Subversion Developers Message-ID: <5160ac49-0c3d-41e3-8b21-2d0aad0b742c@apache.org> Date: Mon, 14 Oct 2019 16:33:53 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit +1. I'd like to support us doing this. Nathan Hartman wrote: > How would we do this? As a starting point, for each client version to be tested, the new multi-combination test driver should: * Run the regression test suite that is supplied with that client version. (This will be easiest because most variation is associated with client side changes.) * Tell the test suite ("make check") which server version to expect. This is partly done: the Python and C tests take an argument --server-minor-version Set the minor version for the server ('3'..'14') or, for the C tests, docs apparently not updated recently: set the minor version for the server ('3', '4', '5', or '6') I'm not sure exactly how one sets up the server appropriately, before running with that option, for various kinds of server. Maybe 'make check' and/or 'make svnserveautocheck' and/or 'make davautocheck' have ways to specify how to find and run the desired server version. We'll probably need to check and update that. Ideally, later, the tests should also be divided or tagged so we can select sets of tests: - client-server tests - client-only tests - server-only tests because we could then eliminate running redundant sets. Initially, that isn't critical. > (1) Which versions are we interested in cross-testing in this manner? Start with a simple fixed set, such as - (client: trunk, server: 1.10) - (client: 1.10, server: trunk) Review later, once that's working. (I suggested 1.10 there because it's the most recent LTS version, but the important thing is just to start with something.) > (2) How do we handle differences between versions? [...] > Is the test driver program supposed to contain knowledge of these > differences and prevent some of the tests from running under certain > combinations of client and server versions? Annotate the tests according to what server versions they require. This is at least partly done. The test suite already uses conditionals like if svntest.main.options.server_minor_version < 9: I'm not sure if this is already done everywhere it needs to be. I would expect to see some of the Python "decorators" such as @SkipUnless(server_authz_has_aliases) using "server_minor_version" but I don't see any. > (3) How do we handle dependencies? Initially: whatever works. Probably neatest to install the dependencies for each built version of Subversion into a location dedicated to that version of Subversion, so they don't affect each other and don't affect the rest of the system. - Julian