airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bas Harenslak <basharens...@godatadriven.com>
Subject Re: [DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0
Date Sun, 17 Feb 2019 08:00:19 GMT
The PR comments say drop hdfs3 and use pyarrow, however this is not implemented yet.


> On 17 Feb 2019, at 08:42, zhong jiajie <zhongjiajie955@hotmail.com> wrote:
> 
> I overlook the PR https://github.com/apache/airflow/pull/3560 still use **hdfs3** rather
than **PyArrow**. So which package we choice finilly?
> [https://avatars3.githubusercontent.com/u/307739?s=400&v=4]<https://github.com/apache/airflow/pull/3560>
> 
> [AIRFLOW-2697] Drop snakebite in favour of hdfs3 by jrderuiter ・ Pull Request #3560
・ apache/airflow ・ GitHub<https://github.com/apache/airflow/pull/3560>
> Commit #52a2ff3 changes the HdfsSensor classes for the new HdfsHook. The biggest change
is that it rewrites the HdfsSensor + HdfsSensorFolder (from contrib) sensors into a HdfsFileSensor
and a HdfsFolderSensor class, which aim to retain the functionality of the previous classes.
> github.com
> 
> 
> 
> ________________________________
> 
> One blocker is HDFS support. Right now we rely on Snakebite which is
> Python2 only, so we need to update this as well. There is an open ticket:
> https://jira.apache.org/jira/browse/AIRFLOW-2697
> 
> Op vr 15 feb. 2019 om 07:55 schreef zhong jiajie <zhongjiajie955@hotmail.com
>> :
> 
>> Agree with XD's second idea, we should warn Airflow users.
>> 
>> ________________________________
>> 
>> My two cents:
>> 
>> 1. Can we discuss on the timeline for 2.0? Another big change is FAB-UI
>> ONLY, which is already in master branch. Personally I think it’s already
>> worth considering 2.0, unless there is  other existing expectation for 2.0.
>> 
>> 2. Before we completely drop Python2 support, we can start adding
>> warning/reminder mentioning that it’s stronly recommended to use py3 & py2
>> drooping can happen “at any time”  for Airflow (we can add these warnings
>> in 1.10.3?)
>> 
>> 
>> XD
>> 
>> On Thu, Feb 14, 2019 at 18:42 Ash Berlin-Taylor <ash@apache.org> wrote:
>> 
>>> We talked about this in the past <
>>> 
>> http://mail-archives.apache.org/mod_mbox/airflow-dev/201808.mbox/%3CCAK%2Be0oujpY1A0mD0G775CmmRneVDjQMTi99aqK%2BA_C07ZW8ySA%40mail.gmail.com%3E
>>> 
>>> but I'd like to bring it up again with a more concrete proposal.
>>> 
>>> At the time Airflow 2.0 wasn't very concrete, but now with a lot of
>>> otherwise breaking changes already on master how about we use that time
>> to
>>> also drop support for Py2?
>>> 
>>> (Reminder: Python 2 is reaching End of Life in January 1, 2020 and will
>>> receive zero updates, even security ones past this date. At least not by
>>> the Python team. I imagine RedHat will still be running Python 2.6 in
>> 2120
>>> ;) )
>>> 
>>> -ash
>> 

Mime
View raw message