airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naik Kaxil <>
Subject Re: Multiple hosts for a single connection
Date Tue, 07 Aug 2018 14:33:00 GMT
Agree that we should have the same functionality over the WebUI and CLI. We need multiple
connections for a single conn-id for load balancing, so +1 for adding something like `--allow-multiple`
or just a warning when a connection with the same id is created with cli.

On 07/08/2018, 15:29, "Ash Berlin-Taylor" <> wrote:

    Hmm yes, it appears that the `airflow connections` CLI doesn't let you create multiple
connections of the same conn_id. What the WebUI can do the CLI should be able to do also!
It should allow that in some way (behind a `--allow-multiple` flag perhaps? I can see an argument
for not allowing set multiple by default as it is often not what people want.)
    > On 7 Aug 2018, at 15:17, Deng Xiaodong <> wrote:
    > Hi Ben,
    > If you would like to set multiple connections with the same *conn_id*, you
    > can only do that in Web UI. That is, you need to build multiple entries
    > with different host, but with the same *conn_id*.
    > The method of setting connection in environment variables can only help set
    > one connection for each *conn_id*.
    > (Of course Web UI stores your connections in the metadata database. So if
    > you really need to avoid manual interventions and need multiple connections
    > for one single conn_id, hacking into the metadata database may be an
    > option? even thought not recommended)
    > Other guys please correct me if I'm wrong in any parts. Thanks.
    > Regards,
    > XD
    > On Tue, Aug 7, 2018 at 9:43 PM Ben Laird <> wrote:
    >> Hello -
    >> I'd like to define multiple connections for the `webhdfs_default` conn_id,
    >> as we have a primary and backup host for the hadoop namenode. This is for
    >> use with the WebHDFSHook, which queries for connections and seemingly can
    >> accept a list of them from the database.
    >> ```
    >>        nn_connections = self.get_connections(self.webhdfs_conn_id)
    >>        for nn in nn_connections:
    >> ```
    >> I can manually add 2 entries for `webhdfs_default` in the Web UI, one for
    >> each hostname, however with the CLI and adding the entries to the
    >> airflow.cfg I get errors that a connection already exists.
    >> It seems WebHDFSHook doesn't support reading multiple hosts from one entry.
    >> Is this a bug? Or is there a different way to accomplish without manually
    >> adding via the UI?
    >> Thanks!
    >> Ben Laird

Kaxil Naik 

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK 
phone: +44 (0)20 7730 6000
View raw message