aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Smith <yasumo...@gmail.com>
Subject Re: [PROPOSAL] Changes to the Python BUILD layout
Date Fri, 31 Jul 2015 02:30:38 GMT
Cool, thanks for thinking this through! Also a +1



> On Jul 30, 2015, at 5:17 PM, Bill Farner <wfarner@apache.org> wrote:
> 
> +1
> 
> -=Bill
> 
> On Thu, Jul 30, 2015 at 4:24 PM, Kevin Sweeney <ksweeney@twitter.com.invalid
>> wrote:
> 
>> Somewhat complementary work - this change would make the generated packages
>> equivalent to the ultimate goal of replacing these top-level BUILD files
>> directly with setup.py files. Users could begin testing deployment with the
>> pants-generated sdists and seamlessly drop in setup.py-generated
>> replacements. Users wishing to deploy via PEX could use the pex tool on the
>> generated sdists rather than the pants pex plugin.
>> 
>>> On Thu, Jul 30, 2015 at 4:13 PM, Bill Farner <wfarner@apache.org> wrote:
>>> 
>>> We've talked in the past about switching build tools.  Just to keep that
>> in
>>> context - how would you weigh this effort against a tool change?
>>> 
>>> -=Bill
>>> 
>>>> On Thu, Jul 30, 2015 at 4:00 PM, Kevin Sweeney <kevints@apache.org>
>>> wrote:
>>> 
>>>> I propose a simplification of the Python BUILD layout as follows and a
>>> set
>>>> of new conventions, as follows:
>>>> 
>>>> 1) 1 BUILD per 3rd level directory. These are currently
>>>> 
>>>> ```
>>>> % find src/main/python -maxdepth 3 -mindepth 3 -type d |while read
>>> dirname;
>>>> do echo $dirname | sed 's@src
>>> /main/python/\(.*\)/\(.*\)/\(.*\).*@\1.\2.\3@
>>>> ';
>>>> done
>>>> apache.aurora.client
>>>> apache.aurora.common
>>>> apache.aurora.tools
>>>> apache.aurora.admin
>>>> apache.aurora.executor
>>>> apache.aurora.config
>>>> apache.thermos.monitoring
>>>> apache.thermos.common
>>>> apache.thermos.cli
>>>> apache.thermos.testing
>>>> apache.thermos.core
>>>> apache.thermos.runner
>>>> apache.thermos.observer
>>>> apache.thermos.config
>>>> ```
>>>> 
>>>> 2) Each BUILD file exports 1 python_library that provides a setup_py
>>>> containing each python_binary in the BUILD file, named the same as the
>>>> directory it's in so that it can be referenced without a ':' character.
>>> The
>>>> sources field in the python_binary will always be rglobs('*.py').
>>>> 
>>>> 3) Other BUILD files may only depend on this single public
>> python_library
>>>> target. Any other target is considered a private implementation detail
>>> and
>>>> should be prefixed with an _.
>>>> 
>>>> 4) python_binary targets are always named the same as the exported
>>> console
>>>> script.
>>>> 
>>>> 5) python_binary targets must have identical dependencies to the
>>>> python_library exported by the package and must use entry_point.
>>>> 
>>>> The advantage of this change is that a PEX file generated by pants will
>>>> contain exactly the same files that will be available on the PYTHONPATH
>>> in
>>>> the case of pip-installation of the corresponding library target. This
>>> will
>>>> help our migration off pants in the future.
>>>> 
>>>> Annotated example: apache.thermos.runner (renamed thermos/bin ->
>>>> thermos/runner)
>>>> 
>>>> ```
>>>> % find src/main/python/apache/thermos/runner
>>>> src/main/python/apache/thermos/runner
>>>> src/main/python/apache/thermos/runner/__init__.py
>>>> src/main/python/apache/thermos/runner/thermos_runner.py
>>>> src/main/python/apache/thermos/runner/BUILD
>>>> % cat src/main/python/apache/thermos/runner/BUILD
>>>> # License boilerplate omitted
>>>> import os
>>>> 
>>>> 
>>>> # Private target so that a setup_py can exist without a circular
>>>> dependency. Only targets within this file should depend on
>>>> # this.
>>>> python_library(
>>>>  name = '_runner',
>>>>  # The target covers every python file under this directory and
>>>> subdirectories.
>>>>  sources = rglobs('*.py'),
>>>>  dependencies = [
>>>>    '3rdparty/python:twitter.common.app',
>>>>    '3rdparty/python:twitter.common.log',
>>>>    # Source dependencies are always referenced without a ':'.
>>>>    'src/main/python/apache/thermos/common',
>>>>    'src/main/python/apache/thermos/config',
>>>>    'src/main/python/apache/thermos/core',
>>>>  ],
>>>> )
>>>> 
>>>> # Binary target for thermos_runner.pex. Nothing should depend on this -
>>>> it's only used as an argument to ./pants binary.
>>>> python_binary(
>>>>  name = 'thermos_runner',
>>>>  # Use entry_point, not source so the files used here are the same
>> ones
>>>> tests see.
>>>>  entry_point = 'apache.thermos.bin.thermos_runner',
>>>>  dependencies = [
>>>>    # Notice that we depend only on the single private target from this
>>>> BUILD file here.
>>>>    ':_runner',
>>>>  ],
>>>> )
>>>> 
>>>> # The public library that everyone importing the runner symbols uses.
>>>> # The test targets and any other dependent source code should depend on
>>>> this.
>>>> python_library(
>>>>  name = 'runner',
>>>>  dependencies = [
>>>>    # Again, notice that we depend only on the single private target
>> from
>>>> this BUILD file here.
>>>>    ':_runner',
>>>>  ],
>>>>  # We always provide a setup_py. This will cause any dependee
>> libraries
>>> to
>>>> automatically reference this library
>>>>  # in their requirements.txt rather than copy the source files into
>>> their
>>>> sdist.
>>>>  provides = setup_py(
>>>>    # Conventionally named and versioned.
>>>>    name = 'apache.thermos.runner',
>>>>    version = open(os.path.join(get_buildroot(),
>>>> '.auroraversion')).read().strip().upper(),
>>>>  ).with_binaries({
>>>>    # Every binary in this file should also be repeated here.
>>>>    # Always use the dict-form of .with_binaries so that commands with
>>>> dashes in their names are supported.
>>>>    # The console script name is always the same as the PEX with .pex
>>>> stripped.
>>>>    'thermos_runner': ':thermos_runner',
>>>>  }),
>>>> )
>>>> ```
>>>> 
>>>> Let me know what you think, if y'all agree I'll prepare a patch
>> shortly.
>> 
>> 
>> 
>> --
>> Kevin Sweeney
>> @kts
>> 

Mime
View raw message