airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <owenzhang1...@gmail.com>
Subject WebHdfsSensor doesn't support HDFS HA
Date Wed, 29 Aug 2018 02:22:44 GMT
Hi all,

We've been using WebHdfsSensor happily to sensor the state of upstream
tasks outputting to HDFS except when there is a namenode switch. I've
opened https://issues.apache.org/jira/browse/AIRFLOW-2901 to discuss the
HDFS HA support.

There are two solutions that I can see,

1. use pyarrow.hdfs which has HA support
2. allow user to configure a list of namenodes

WDYT ?

Thanks,
Manu Zhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message