airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <>
Subject WebHdfsSensor doesn't support HDFS HA
Date Wed, 29 Aug 2018 02:22:44 GMT
Hi all,

We've been using WebHdfsSensor happily to sensor the state of upstream
tasks outputting to HDFS except when there is a namenode switch. I've
opened to discuss the
HDFS HA support.

There are two solutions that I can see,

1. use pyarrow.hdfs which has HA support
2. allow user to configure a list of namenodes


Manu Zhang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message