From commits-return-205877-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Wed Jan 31 22:24:12 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id D3BA2180662 for ; Wed, 31 Jan 2018 22:24:12 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id C3505160C56; Wed, 31 Jan 2018 21:24:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E4C5E160C2C for ; Wed, 31 Jan 2018 22:24:11 +0100 (CET) Received: (qmail 55300 invoked by uid 500); 31 Jan 2018 21:24:06 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 55281 invoked by uid 99); 31 Jan 2018 21:24:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Jan 2018 21:24:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 9F307C00CD for ; Wed, 31 Jan 2018 21:24:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.011 X-Spam-Level: X-Spam-Status: No, score=-109.011 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id xRdFTm3C2GL1 for ; Wed, 31 Jan 2018 21:24:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id C88F25F3B9 for ; Wed, 31 Jan 2018 21:24:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id D65C9E0226 for ; Wed, 31 Jan 2018 21:24:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 5AEF924102 for ; Wed, 31 Jan 2018 21:24:00 +0000 (UTC) Date: Wed, 31 Jan 2018 21:24:00 +0000 (UTC) From: "Michael Kjellman (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-14134) Migrate dtests to use pytest and python3 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-14134?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId= =3D16347636#comment-16347636 ]=20 Michael Kjellman commented on CASSANDRA-14134: ---------------------------------------------- [~jay.zhuang]=C2=A0I was planning on doing that as a separate ticket/commit= . got a bit sidelined with some other issues =E2=80=93 hoping to get it don= e by EOD today. I'll ping the new=C2=A0JIRA here as well once it's created. > Migrate dtests to use pytest and python3 > ---------------------------------------- > > Key: CASSANDRA-14134 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1413= 4 > Project: Cassandra > Issue Type: Improvement > Components: Testing > Reporter: Michael Kjellman > Assignee: Michael Kjellman > Priority: Major > Fix For: 4.0 > > > h4. Get the C* dtests running on the pytest framework. > C* DTests currently run using the python test framework nosetest. This fr= amework has been largely abandoned with no releases since 2015 and a genera= l strong consensus in the python community that pytest is the future. > h4. Why should we do this. > Currently (and historically) dtests have always been difficult to run, fl= aky and unpredictable in CI environments, and almost impossible to debug. > On November 28th, 2017, I proposed on the dev@ list that we move the dtes= ts from nosetests to pytests. I got replies from Jon Haddad, Philip Thompso= n, and kurt greaves with really only "+1" like replies to the proposal. > Since then I've been working pretty much non stop to complete the large r= efactor of dtests to pytests. As part of this effort (and due to the migrat= ion tools that exist require it) I also ported the code to python3 (from th= e current python 2.7 based code-base). > h4. High-level summary of key changes, improvements, and new features. > * Migrate dtests from executing using the nosetest framework to pytest > * Port the entire code base from Python 2.7 to Python 3.6 > * Update run_dtests.py to work with pytest > * Add --dtest-print-tests-only option to run_dtests.py to get easily pars= able list of all available collected tests > * Update README.md for executing the dtests with pytest > * Add new debugging tips section to README.md to help with some basics of= debugging python3 and pytest > * Migrate all existing Enviornment Variable usage as a means to control d= test operation modes to argparse command line options with documented help = on each toggles intended usage > * Migration of old unitTest and nose based test structure to modern pytes= t fixture approach > * Automatic detection of physical system resources to automatically deter= mine if @pytest.mark.resource_intensive annotated tests should be collected= and run on the system where they are being executed > * new pytest fixture replacements for @since and @pytest.mark.upgrade_tes= t annotations > * Migration to python logging framework > * Upgrade thrift bindings to latest version with full python3 compatibili= ty > * Remove deprecated cql and pycassa dependencies and migrate any remainin= g tests to fully remove those dependencies > * Fixed dozens of tests that would hang the pytest framework forever when= run in CI enviornments > * Ran code nearly 300 times in CircleCI during the migration and to find,= identify, and fix any tests capable of hanging CI > * Upgrade Tests do not yet run in CI and still need additional migration = work (although all upgrade test classes compile successfully) > I started with the *nose2pytest* [https://github.com/pytest-dev/nose2pyte= st] migration tool. As this required python 3 language support I found myse= lf down the 2to3 python migration path. While painful to do this, the benef= its of python3 over python2.7 are numerous and moving to python3 for the ad= ditional debugging tools now available to use when fixing dtests makes the = effort worth it for that reason alone! > After the automated tools did their thing I began what was a much longer = and tedious manual process than I ever could have expected due to the custo= m many ways we did things in dtests (frequently to work around nosetest lim= itations of missing features that thankfully are now all included with the = pytest framework). I've done nearly 300 test runs of my migration branch wi= th circleci. > The latest CircleCI runs can be found at: > (dtests without vnodes) [https://circleci.com/gh/mkjellman/cassandra/277] > (dtests with vnodes) [https://circleci.com/gh/mkjellman/cassandra/278] > With vnodes, there are currently only 6 remaining dtest test failures. > Without vnodes, there are 12 remaining dtest failures. > It turns out that after the dtests were moved to ASF Jenkins from cassci,= the jobs were misconfigured and we actually haven't been running the dtest= s in the non-vnodes configuration. The current most recent trunk dtest job = to complete on ASF Jenkins (with vnodes) was [https://builds.apache.org/vie= w/A-D/view/Cassandra/job/Cassandra-trunk-dtest/387/]. That test run had 36 = test failures. > There are great performance improvements so far with pytests. With a para= llism factor of 90x in CircleCI the dtests-no-vnodes job completed all test= s in 16 minutes and 24 seconds and the dtests-with-vnodes job completed all= tests in 14 minutes and 22 seconds.. (Compare that to the 8+ hours ASF Jen= kins currently takes!). > A dtest on pytest compatible CircleCI configuration is available at https= ://github.com/mkjellman/cassandra/blob/trunk_circle/.circleci/config.yml > A very small patch is required to ccm for 2 places that were not yet pyth= on3 compatible. (I didn't do a full audit of ccm, but these appear to be th= e only two issues I've found even with a ton of testing when using ccmlib w= ith dtests and running with python 3). > {code} > --- /home/cassandra/env/src/ccm/ccmlib/node.py 2017-12-13 03:11:13.00000= 0000 +0000-8").splitlines())) > +++ node.py 2017-12-19 09:51:49.083186951 +0000 > @@ -57,10 +57,16 @@ > message =3D "Subprocess {} exited with non-zero status; exit sta= tus: {}".format(command, exit_status) > if stdout: > message +=3D "; \nstdout: " > - message +=3D stdout > + if isinstance(stdout, str): > + message +=3D stdout > + else: > + message +=3D stdout.decode("utf-8") > if stderr: > message +=3D "; \nstderr: " > - message +=3D stderr > + if isinstance(stderr, str): > + message +=3D stderr > + else: > + message +=3D stderr.decode("utf-8") > =20 > Exception.__init__(self, message) > =20 > @@ -2009,7 +2015,10 @@ > =20 > out, _, _ =3D handle_external_tool_process(p, ["sstableutil", '-= -type', 'final', ks, table]) > =20 > - return sorted(filter(lambda s: s.endswith('-Data.db'), out.split= lines())) > + if isinstance(out, str): > + return sorted(filter(lambda s: s.endswith('-Data.db'), out.s= plitlines())) > + else: > + return sorted(filter(lambda s: s.endswith('-Data.db'), out.d= ecode("utf-8").splitlines())) > =20 > def _get_load_from_info_output(info): > {code} > For the actual dtest work itself: I've squashed all my work down into a s= ingle commit available at [https://github.com/mkjellman/cassandra-dtest/tre= e/dtests_on_pytest_v2] > And the actual commit available at [https://github.com/mkjellman/cassandr= a-dtest/commit/11201c149d1198c0d6d1a479e08bfb8695f14c9e] > I tried to put a lot of effort into making the end user experience better= for both bootstrapping and running dtests for the first time, but also day= to day. I hope people like the approach! In addition to moving to argparse= with commented command line arguments to control execution, I also updated= the README.md to reflect the new pytest world and also added a new section= with a few tips I learned while I was debugging hundreds of tests during t= his migration process that hopefully are helpful to the rest of the communi= ty going forward! > I'm sure the first commit above isn't 100% perfect (although I really hav= e put a lot of effort into it) but my one request is to hopefully get conse= nsus with this first revision of changes committed to reduce any chance of = further drift with the codebase as changes are committed into master. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org For additional commands, e-mail: commits-help@cassandra.apache.org