aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Sweeney <kevi...@apache.org>
Subject [PROPOSAL] Changes to the Python BUILD layout
Date Thu, 30 Jul 2015 23:00:14 GMT
I propose a simplification of the Python BUILD layout as follows and a set
of new conventions, as follows:

1) 1 BUILD per 3rd level directory. These are currently

```
% find src/main/python -maxdepth 3 -mindepth 3 -type d |while read dirname;
do echo $dirname | sed 's@src/main/python/\(.*\)/\(.*\)/\(.*\).*@\1.\2.\3@';
done
apache.aurora.client
apache.aurora.common
apache.aurora.tools
apache.aurora.admin
apache.aurora.executor
apache.aurora.config
apache.thermos.monitoring
apache.thermos.common
apache.thermos.cli
apache.thermos.testing
apache.thermos.core
apache.thermos.runner
apache.thermos.observer
apache.thermos.config
```

2) Each BUILD file exports 1 python_library that provides a setup_py
containing each python_binary in the BUILD file, named the same as the
directory it's in so that it can be referenced without a ':' character. The
sources field in the python_binary will always be rglobs('*.py').

3) Other BUILD files may only depend on this single public python_library
target. Any other target is considered a private implementation detail and
should be prefixed with an _.

4) python_binary targets are always named the same as the exported console
script.

5) python_binary targets must have identical dependencies to the
python_library exported by the package and must use entry_point.

The advantage of this change is that a PEX file generated by pants will
contain exactly the same files that will be available on the PYTHONPATH in
the case of pip-installation of the corresponding library target. This will
help our migration off pants in the future.

Annotated example: apache.thermos.runner (renamed thermos/bin ->
thermos/runner)

```
% find src/main/python/apache/thermos/runner
src/main/python/apache/thermos/runner
src/main/python/apache/thermos/runner/__init__.py
src/main/python/apache/thermos/runner/thermos_runner.py
src/main/python/apache/thermos/runner/BUILD
% cat src/main/python/apache/thermos/runner/BUILD
# License boilerplate omitted
import os


# Private target so that a setup_py can exist without a circular
dependency. Only targets within this file should depend on
# this.
python_library(
  name = '_runner',
  # The target covers every python file under this directory and
subdirectories.
  sources = rglobs('*.py'),
  dependencies = [
    '3rdparty/python:twitter.common.app',
    '3rdparty/python:twitter.common.log',
    # Source dependencies are always referenced without a ':'.
    'src/main/python/apache/thermos/common',
    'src/main/python/apache/thermos/config',
    'src/main/python/apache/thermos/core',
  ],
)

# Binary target for thermos_runner.pex. Nothing should depend on this -
it's only used as an argument to ./pants binary.
python_binary(
  name = 'thermos_runner',
  # Use entry_point, not source so the files used here are the same ones
tests see.
  entry_point = 'apache.thermos.bin.thermos_runner',
  dependencies = [
    # Notice that we depend only on the single private target from this
BUILD file here.
    ':_runner',
  ],
)

# The public library that everyone importing the runner symbols uses.
# The test targets and any other dependent source code should depend on
this.
python_library(
  name = 'runner',
  dependencies = [
    # Again, notice that we depend only on the single private target from
this BUILD file here.
    ':_runner',
  ],
  # We always provide a setup_py. This will cause any dependee libraries to
automatically reference this library
  # in their requirements.txt rather than copy the source files into their
sdist.
  provides = setup_py(
    # Conventionally named and versioned.
    name = 'apache.thermos.runner',
    version = open(os.path.join(get_buildroot(),
'.auroraversion')).read().strip().upper(),
  ).with_binaries({
    # Every binary in this file should also be repeated here.
    # Always use the dict-form of .with_binaries so that commands with
dashes in their names are supported.
    # The console script name is always the same as the PEX with .pex
stripped.
    'thermos_runner': ':thermos_runner',
  }),
)
```

Let me know what you think, if y'all agree I'll prepare a patch shortly.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message