airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (Jira)" <>
Subject [jira] [Commented] (AIRFLOW-5373) Super fast pre-commit check for basic python2 compatibility (for cherry-picking)
Date Tue, 03 Sep 2019 16:06:00 GMT


ASF subversion and git services commented on AIRFLOW-5373:

Commit 00b50e9481968ab0df40c5c57173a44c8ab26fa6 in airflow's branch refs/heads/v1-10-test
from Jarek Potiuk
[;h=00b50e9 ]

[AIRFLOW-5373] Super fast pre-commit check for basic python2 compatibility

> Super fast pre-commit check for basic python2 compatibility (for cherry-picking)
> --------------------------------------------------------------------------------
>                 Key: AIRFLOW-5373
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: ci
>    Affects Versions: 1.10.5
>            Reporter: Jarek Potiuk
>            Assignee: Jarek Potiuk
>            Priority: Major
>         Attachments: master_code_with_python2_py_compile.txt, matches_python3.txt
> I thought about a super-fast way of protecting against bad python3 cherry-picks in our
v1-10-test branch. From the experience, it looks like we have two types of problems most often:
>  # super()
>  # python3 type annotations in definition of function parameters/return values
>  # python3 type annotations in variables
> I tried to find some good "proper" solution to check automatically if the code is python2-compliant
but surprisingly could not find anything fast and good (actually there are plenty of python3
compat checkers and auto-converters but I could not find good verification is some python3
constructs are used in python2 code).
> However it came to me that we can likely do a simple grep that should be able to catch
vast majority of those case with very limited (if at all) false positives. It turned out to
be a good idea for 1 and 2 (which is vast majority of cases I think) :
> Such simple and straigthforward regexp does the work beautifully:
> {code:java}
> ".super\\(\\)|^\\s+def\\s*\\S*\\([^):]*:.*\\)|^\\sdef\\s*\\S*\\(.*\\):\\s*\\-\\>\\s*\\S*" 
> {code}
> I managed to find two actual problems (but the code was in comments so no impact ! )-
Python3 incompatibilitites in v1-10-test this way (!!!). In master we have 830 matching lines
(attached) so I think it's rather good.
> I also added another check with py_compile run on all files. It works perfectly for v1-10-test
and for master produces plenty of syntax errors. Attached as well. 

This message was sent by Atlassian Jira

View raw message