yetus-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: [DISCUSS] Requiring Python 2.7
Date Tue, 06 Sep 2016 18:00:28 GMT
In my own experience, I’ve had to work under all of the following policies:

1. Write Python code however you want, and any necessary upgrades are fine.
2. You must stick to Python version X, but installing additional third-party libraries is
acceptable.
3. You must stick to “vanilla” Python version X, and additional third-party libraries
cannot be installed.

My suggestion to consider bundling argparse and front-loading it on PYTHONPATH is a workaround
that I have used in scenario 3 in some cases (when the library’s licensing allowed it).
 It’s a trade-off between the convenience of whatever the library in question provides and
the additional build and deployment complexity.  This was considered acceptable under policy,
because use of the library was constrained to my application and not installed globally.

However, as you pointed out, we’re not currently doing this in Yetus.  We currently prompt
to install the additional libraries.  I think this is acceptable.  Policy 3 was an unusual
circumstance for me.  If we get a user asking for vanilla Python 2.6 support, then it’s
something we could explore again.

--Chris Nauroth

On 9/6/16, 10:35 AM, "Andrew Wang" <andrew.wang@cloudera.com> wrote:

    I already updated YETUS-445 for Python 2.6 support, so it's not blocked on
    this conversation.
    
    Chris, in the situation you describe where "IT policies prevented me from
    performing non-essential/non-security related upgrades or installing new
    software", does this policy not also apply to third party libraries
    installed via pip? I honestly don't see the difference, since both conda
    and pip can be used without root, and install things to your local homedir.
    
    This is why I keep bringing up third-party libraries. If the restriction is
    not just Python 2.6 but vanilla Python 2.6, that's also something that
    should be documented, and we should retrofit RDM and YETUS-445 to remove
    those dependencies.
    
    On Tue, Sep 6, 2016 at 10:23 AM, Chris Nauroth <cnauroth@hortonworks.com>
    wrote:
    
    > I’m just catching up on this thread now.
    >
    > I fall in the camp that would prefer not to dictate a requirement of
    > Python 2.7 for anything in Yetus.  My reasoning agrees with what Allen has
    > already described.  I have been in the situation that he describes, needing
    > to maintain build infrastructure on old OSes and old Python versions.
    > Upgrading the OS or installing Python 2.7 via conda or a custom rpm was not
    > an option.  IT policies prevented me from performing
    > non-essential/non-security related upgrades or installing new software.
    > Instead, the solution was to stick to Python code compatible with the old
    > versions.
    >
    > I haven’t reviewed the code for YETUS-445, but the motivation for this
    > topic coming up now seems to be argparse.  I think Ajay’s proposal for
    > handling that sounds fine.  Alternatively, we could explore if it’s
    > feasible to check in a copy of argparse to be front-loaded onto PYTHONPATH.
    >
    > --Chris Nauroth
    >
    > On 8/24/16, 9:45 PM, "Ajay Yadava" <ajayyadavaatos@gmail.com> wrote:
    >
    >     If the only motivation for Python 2.7 for this script is to use
    > argparse
    >     library then it can be installed on Python 2.6 and hence shouldn't be
    > an
    >     issue IMO. We have external dependencies for RDM as well. A simple
    >     try-catch will ensure that 2.6 users know about the external
    > dependency.
    >     Does this work?
    >
    >     On Thu, Aug 25, 2016, 9:22 AM Dima Spivak <dspivak@cloudera.com>
    > wrote:
    >
    >     > What if we compromise by creating some sort of designation for tools
    >     > that require runs to use Docker mode? This way, people could still
    > use
    >     > Andrew's awesome script either on their own or in Apache Infra (even
    >     > precommit!) without unfortunate surprises when someone tries to run
    > it on
    >     > an older OS.
    >     >
    >     > On Wednesday, August 24, 2016, Andrew Wang <andrew.wang@cloudera.com
    > >
    >     > wrote:
    >     >
    >     > > On Wed, Aug 24, 2016 at 4:46 PM, Allen Wittenauer <
    >     > > aw@effectivemachines.com <javascript:;>>
    >     > > wrote:
    >     > >
    >     > > >
    >     > > > > On Aug 24, 2016, at 3:49 PM, Andrew Wang <
    > andrew.wang@cloudera.com
    >     > > <javascript:;>>
    >     > > > wrote:
    >     > > > > My reasoning here has been that this tool will primarily
be
    > run by
    >     > > > release
    >     > > > > managers, and RMs are very likely to have Python 2.7 on their
    >     > machine.
    >     > > >
    >     > > >         Whereas I believe the opposite: there's a good chance
if
    > one is
    >     > > > the RM for enterprise software, the machine you are building on
    >     > probably
    >     > > > doesn't have python 2.7 on it because you aren't doing this work
    > on
    >     > your
    >     > > > desktop!  It's extremely common to have build servers that are
    > several
    >     > OS
    >     > > > revs behind because software tends to be more upwardly
    > compatible....
    >     > > >
    >     > > > > ASF infra and servers in general are not really a target.
    >     > > >
    >     > > >         Umm, yeah, they kind of are. While there are certainly
    > issues
    >     > > with
    >     > > > them, they do serve as sort of a model of what is out in the real
    >     > world.
    >     > > > It's old and crufty and that's how a lot of software is built.
    > Where
    >     > > > people build software is *exactly* our target.
    >     > > >
    >     > > >         Here's a fun anecdote.  Firefox v48 is the first version
    > to
    >     > drop
    >     > > > support for Mac OS X 10.6, 10.7, and 10.8.  This means they were
    > almost
    >     > > > certainly building on 10.6.  So up until earlier this year, they
    > were
    >     > > using
    >     > > > a 7 year old OS that most definitely didn't have python 2.7 on
    > it by
    >     > > > default. Mozilla, by many accounts, would be considered quite
    >     > aggressive
    >     > > > but they can get away with it because they are targeting the
    > desktop.
    >     > > >
    >     > > >
    >     > > I think we're talking past each other here, since this is not a
    > build
    >     > tool.
    >     > > It's an API checking tool. It only needs to be run on a developer's
    >     > > computer, not all boxes at ASF. And, like I've been saying, you
    > can just
    >     > > install Python 2.7 like you would any other dependency. RDM
    > requires `pip
    >     > > install python-dateutils` for instance.
    >     > >
    >     > >
    >     > > > > Maybe one day if
    >     > > > > we want to run this as part of precommit, but then it can
be
    > optional
    >     > > > like
    >     > > > > gradle and docker.
    >     > > >
    >     > > >
    >     > > >         It sends really bizarre mixed messages if one tool has
    >     > different
    >     > > > python requirements than another just because someone wanted to
    > use a
    >     > > > different option parsing library.
    >     > >
    >     > >
    >     > > I don't know about "really bizzare"; this is the same problem as
    > any
    >     > > external dependency (e.g. python-dateutils).
    >     > >
    >     > > The real issue here is the lack of a standardized runtime
    > environment. Is
    >     > > there build/packaging for Yetus? Some of this could be handled by
    >     > wrappers
    >     > > that created virtualenvs or downloaded conda as appropriate.
    >     > >
    >     >
    >     >
    >     > --
    >     > -Dima
    >     >
    >     --
    >     Regards
    >     Ajay Yadava
    >
    >
    >
    

Mime
View raw message