airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bas Harenslak <>
Subject Re: [DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0
Date Sun, 24 Feb 2019 19:25:58 GMT
Following the comments to warn users about dropping Python 2, I went ahead and added the deprecation
library to Airflow in this PR:, and added the
first @deprecated annotation.

I checked a few major packages and all use some self-made annotation for showing deprecation
warnings. I didn’t want to re-invent the wheel and used a tiny package for it. More details
are in the PR itself. If you know a better way to do this, let me know :-)


On 17 Feb 2019, at 09:00, Bas Harenslak <<>>

The PR comments say drop hdfs3 and use pyarrow, however this is not implemented yet.

On 17 Feb 2019, at 08:42, zhong jiajie <<>>

I overlook the PR still use **hdfs3** rather than
**PyArrow**. So which package we choice finilly?

[AIRFLOW-2697] Drop snakebite in favour of hdfs3 by jrderuiter ・ Pull Request #3560 ・
apache/airflow ・ GitHub<>
Commit #52a2ff3 changes the HdfsSensor classes for the new HdfsHook. The biggest change is
that it rewrites the HdfsSensor + HdfsSensorFolder (from contrib) sensors into a HdfsFileSensor
and a HdfsFolderSensor class, which aim to retain the functionality of the previous classes.<>


One blocker is HDFS support. Right now we rely on Snakebite which is
Python2 only, so we need to update this as well. There is an open ticket:

Op vr 15 feb. 2019 om 07:55 schreef zhong jiajie <

Agree with XD's second idea, we should warn Airflow users.


My two cents:

1. Can we discuss on the timeline for 2.0? Another big change is FAB-UI
ONLY, which is already in master branch. Personally I think it’s already
worth considering 2.0, unless there is  other existing expectation for 2.0.

2. Before we completely drop Python2 support, we can start adding
warning/reminder mentioning that it’s stronly recommended to use py3 & py2
drooping can happen “at any time”  for Airflow (we can add these warnings
in 1.10.3?)


On Thu, Feb 14, 2019 at 18:42 Ash Berlin-Taylor <> wrote:

We talked about this in the past <

but I'd like to bring it up again with a more concrete proposal.

At the time Airflow 2.0 wasn't very concrete, but now with a lot of
otherwise breaking changes already on master how about we use that time
also drop support for Py2?

(Reminder: Python 2 is reaching End of Life in January 1, 2020 and will
receive zero updates, even security ones past this date. At least not by
the Python team. I imagine RedHat will still be running Python 2.6 in
;) )


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message