From dev-return-8100-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Fri Apr 12 14:51:36 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id A692B180621 for ; Fri, 12 Apr 2019 16:51:35 +0200 (CEST) Received: (qmail 86774 invoked by uid 500); 12 Apr 2019 14:51:34 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 86585 invoked by uid 99); 12 Apr 2019 14:51:34 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Apr 2019 14:51:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D1ED2CCE58 for ; Fri, 12 Apr 2019 14:51:33 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.8 X-Spam-Level: * X-Spam-Status: No, score=1.8 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=quantopian.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 6yo6l5WU6Pq4 for ; Fri, 12 Apr 2019 14:51:32 +0000 (UTC) Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 055525F66C for ; Fri, 12 Apr 2019 14:51:31 +0000 (UTC) Received: by mail-ot1-f53.google.com with SMTP id j10so8641625otq.0 for ; Fri, 12 Apr 2019 07:51:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=FNApykaq4XUoIacePwZxzDj+ccQoOKE26k31871ZNPk=; b=I3twlP5D7xyA9vjN6A/CzCV2KqIR+QwuljFx8J3h6vusUHZSv6hd6/m9LkSy1QsXwH ikaHPHwu0goVGkFM3j5mbuXpGomGpZUudfGb/e8XL7TzppvI93SFxl2akCo9cQa3DPz6 0JaSDBuID60MFi6ekLY+ieQni8rO4xN4n/0nj8Vd2iA5jbBXsUry+WiuDUzMY4J7R0k9 B7usiYLYbXIvtBWW7w/bMgS6YSZTuz5TgHdnU9+7tbIzMQeG0nKAfRGD0OGbEGx4Xq41 zAK68NWG3o8R5y8ilh455XvctkIHfhmObWmGRvYeuLtKgJmFMzx588X890jnuZGaoiNE JbIA== X-Gm-Message-State: APjAAAVjFCtExrh+PgzDjh1i0LpXTpHJG4VlM8HwuUeWxxadgRbUD+OY Ko5z7Rjr8VsnpfR4nlP4497f6KHYH0V5OmySV+h9bdIN7s7KvxIYNzmv/qU6W8Yv+vFWHOWH9ID 04oFVnDp4l0ABJeMsEjEXjjgcxcem57q6RyioajVEWLniQGiVs08MT4lSIOvRpA1lO0U60hgGuH ljHjm65OBg/fT/LMARyPSuiwO9AvSIcM0zxOKzJmWq X-Google-Smtp-Source: APXvYqzQq03q43LXZv0niT9Fk5medM2UuLpbEEqsih3dd+njUmCJEGk2r3y8Rvm/6ybwQuLuWJQxwzG9f0lAN4Gnaqg= X-Received: by 2002:a9d:7a48:: with SMTP id z8mr6233390otm.69.1555080685167; Fri, 12 Apr 2019 07:51:25 -0700 (PDT) MIME-Version: 1.0 From: James Meickle Date: Fri, 12 Apr 2019 10:51:10 -0400 Message-ID: Subject: [2.0 spring cleaning] Require unique conn_id To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="000000000000d6438b0586566fe1" --000000000000d6438b0586566fe1 Content-Type: text/plain; charset="UTF-8" Airflow fetches connections by name, but doesn't enforce unique names. My team got bit by this, since it's very unexpected behavior for most types of data entry. The reason for this behavior is explained in the docs: "Many connections with the same conn_id can be defined and when that is the case, and when the hooks uses the get_connection method from BaseHook, Airflow will choose one connection randomly, allowing for some basic load balancing and fault tolerance when used in conjunction with retries." I think this is very non-intuitive UX. If we even want to support this feature within Airflow - and I don't think that is a given - it would make much more sense to require a unique (conn_id, conn_type) but allow storing multiple related records. This wouldn't be a huge data modeling change, but would require changing the web UI to appear as a form with subforms. --000000000000d6438b0586566fe1--