beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Davor Bonaci (JIRA)" <>
Subject [jira] [Assigned] (BEAM-1677) Extend automated PR check with PyCharm's code inspector for Python-SDK
Date Mon, 27 Mar 2017 17:20:42 GMT


Davor Bonaci reassigned BEAM-1677:

    Assignee: Ahmet Altay

> Extend automated PR check with PyCharm's code inspector for Python-SDK
> ----------------------------------------------------------------------
>                 Key: BEAM-1677
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: build-system
>            Reporter: Tibor Kiss
>            Assignee: Ahmet Altay
>         Attachments: inspection-results-1d8be97439051ae770f99dcb49c7e8486962b118.tar.gz
> *Proposal:*
> Leverage PyCharm's static code analysis capabilities to detect possible errors if a Python-SDK
related PR is submitted.
> *Problem statement:*
> The Python SDK test goal has linting phase, which utilizes {{pep8}} and {{pylint}} to
validate PEP8 compliance and do static code analysis. These steps are helpful but {{pylint}}'s
capabilities are somewhat limited compared to modern IDE's (like PyCharm). 
> In particular: Unresolved attributes, unbound local variables and invalid calls (due
to argument number mismatch) type of errors  are not detected by pylint, but they do cause
runtime error. See BEAM-1649 for problems not detected by pylint but reported by PyCharm.
> *Technical details:*
> PyCharm provides command line interface [1] to trigger code inspection. The results are
created as entries in XML documents, containing their location, severity, category and textual
description of the problem.
> Since the inspector reports back typos, warnings and errors it is required that the result
XML files are properly parsed and filtered.
> A good signal-to-noise ratio is crucial for this improvement to be useful and accepted,
therefore it is required to make good selection of events to minimize false positives.
> The filtered results shall be posted back to GitHub through Commit Status API [2] to
notify the contributor and reviewer of the quality of the change.
> The static code analysis could be done on either Apache Build Jenkins servers or in Travis
> *Alternatives considered:*
>  - Eclipse's PyDev plugin [3] has similar code inspection capabilities as PyCharm. The
lack of command line interface makes it difficult to integrate it with the current build system.
>  - SonarQUBE's Python plugin [4] was briefly investigated. The number of rules seems
very limited compared to PyCharm's code analysis.
> *Required changes:*
> - PyCharm to be installed in Jenkins Slave or Travis-CI's container
> - PyCharm project & inspection profile created for Python SDK
> - Result parser & poster application/script created
> - Trigger to run the result parser & poster set up
> *Example Code Analysis* run on OS X (given a PyCharm project is available in beam/sdks/python/.idea
> {code}
> beam/sdks/python $ /Applications/ inspect $(pwd) $(pwd)/.idea/inspectionProfiles/Project_Default.xml
$(pwd)/inspection-results -v2 -d $(pwd)/apache_beam
> {code}
> *Example result:*
> {code:title=PyUnresolvedReferencesInspection.xml}
> <problem>
> <file>file://$PROJECT_DIR$/apache_beam/io/</file>
> <line>342</line>
> <module>python</module>
> <entry_point TYPE="file" FQNAME="file://$PROJECT_DIR$/apache_beam/io/"
> <problem_class severity="WARNING" attribute_key="WARNING_ATTRIBUTES">Unresolved
> <description>Unresolved attribute reference '_end_position' for class 'OrderedPositionRangeTracker'</description>
> </problem>
> {code}
> *References:*
> [1] PyCharm Command Line Interface:
> [2] GitHub Commit Status API:
> [3] Eclipse PyDev Plugin:
> [4] SonarQUBE Python plugin:
> CC: [~altay] / [~jbonofre] / [~jasonkuster]

This message was sent by Atlassian JIRA

View raw message