Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
References: <CAH053HR=1_Wo0cVSK3FWewoCC70ZzgyihsTF3fGnyMvHeWh4rQ@mail.gmail.com>
 <CALCMnts4Reo7T_BHU5uYeyVRLBhZ-hmLhC1+nX3JZYt931nneQ@mail.gmail.com>
 <CAH053HQ7XUBu6AmD0TFTTLdG8Vo7P3gvAZOcVzEQdmW8=kP9Sg@mail.gmail.com>
 <CALCMntspi1gSyPMTz_gG12v1yKp9=5To8b_FSU1BW7i-7rvDCg@mail.gmail.com> <CAH053HRzb3R8APboJwweqDzHz+am5+bSFKZgSjOy58zQZ7k0SA@mail.gmail.com>
In-Reply-To: <CAH053HRzb3R8APboJwweqDzHz+am5+bSFKZgSjOy58zQZ7k0SA@mail.gmail.com>
From: Alex Van Boxel <alex@vanboxel.be>
Date: Mon, 09 Jan 2017 13:13:37 +0000
Message-ID: <CALCMntv1qMgRRCPKbA4YnkjU6nwjTiGrX8AT3qo0wPVw-SkGxg@mail.gmail.com>
Subject: Re: Refactoring Connection
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a113d729e57bb090545a92352
archived-at: Mon, 09 Jan 2017 13:14:04 -0000

--001a113d729e57bb090545a92352
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I was actually going to propose something different with entry-points, but
your requirement beat me to it (but that's ok :-). Actually I think with
this mechanism people would be able to extend Airflow connection mechanism
(and later other stuff) by doing *pip install airflow-sexy-new-connection*
(for example).

On Mon, Jan 9, 2017 at 1:39 PM Gael Magnan <gaelmagnan@gmail.com> wrote:

> Thank you for the read, I'm gonna look at it, it's probably gonna be bett=
er
> that what I have.
>
> Point taken about the URI, I'll see if i can find something generic enoug=
h
> to handle all those cases.
>
> Le lun. 9 janv. 2017 =C3=A0 13:36, Alex Van Boxel <alex@vanboxel.be> a =
=C3=A9crit :
>
> > Thanks a lot, yes it clarifies a lot and I do agree you really need to
> hack
> > inside Airflow to add a Connection type. While you're working at this
> could
> > you have a look at the standard python *entry-point mechanism* for
> > registering Connection types/components.
> >
> > A good read on this:
> >
> >
> http://docs.pylonsproject.org/projects/pylons-webframework/en/latest/adva=
nced_pylons/entry_points_and_plugins.html
> >
> > My first though would be that just by adding an entry to the factory
> method
> > would be enough to register your Connection + ConnectionType and UI.
> >
> > Also note that not everything works with a URI. The Google Cloud
> Connection
> > doesn't have one, it uses a secret key file stored on disk, so don't
> force
> > every connection type to work with URI's.
> >
> >
> >
> > On Mon, Jan 9, 2017 at 1:15 PM Gael Magnan <gaelmagnan@gmail.com> wrote=
:
> >
> > > Yes sure,
> > >
> > > The question was the following:
> > > "I was looking at the code of the connections, and I realized you can=
't
> > > easily add a connection type without modifying the airflow code
> source. I
> > > wanted to create a mongodb connection type, but I think the best
> approche
> > > would be to refactor connections first. Thoughts anyone?"
> > >
> > > The answer of Bolke de Bruin was: "making it more generic would be
> > > appreciated"
> > >
> > > So basically the way the code is set up actually every types of
> > connection
> > > existing is defined though a list in the Connection class. It
> implements
> > > exactly the same code for parsing uri to get connections info and
> doesn't
> > > allow for a simple way to get back the uri from the connection infos.
> > >
> > > I need to add a mongodb connection and a way to get it back as a uri,
> so
> > i
> > > could use an other type of connection and play around with that or
> juste
> > > add one more hard coded connection type, but I though this might be
> > > something that comes back regularly and having a simple way to plug i=
n
> > new
> > > types of connection would make it easier for anyone to contribute a n=
ew
> > > connection type.
> > >
> > > Hope this clarifies my proposal.
> > >
> > > Le lun. 9 janv. 2017 =C3=A0 12:46, Alex Van Boxel <alex@vanboxel.be> =
a
> =C3=A9crit
> > :
> > >
> > > > Hey Gael,
> > > >
> > > > could you please recap the question here and provide some context.
> Not
> > > > everyone on the mailinglist is actively following Gitter, including
> me.
> > > > With some context it would be easier to give feedback. Thanks.
> > > >
> > > > On Mon, Jan 9, 2017 at 11:15 AM Gael Magnan <gaelmagnan@gmail.com>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > following my question on gitter the other day and the response fr=
om
> > > Bolke
> > > > > de Bruin, I've started working on refactoring the connections in
> > > airflow.
> > > > >
> > > > > Before submitting a PR I wanted to share my proposal with you and
> get
> > > > > feedbacks.
> > > > >
> > > > > The idea is quite simple, I've divided the Connection class in tw=
o,
> > > > > Connection and ConnectionType, connection has the same interface =
it
> > had
> > > > > before plus a few methods, but the class keeps a reference to a
> > > > dictionary
> > > > > of registered ConnectionType. It delegates the work of parsing fr=
om
> > > URI,
> > > > > formatting to URI (added) and getting the hook to the
> ConnectionType
> > > > > associated with the conn_type.
> > > > >
> > > > > I've thought of two ways of registering new ConnectionTypes, the
> > first
> > > is
> > > > > making the BaseConnectionType use a metaclass that registered any
> new
> > > > > ConnectionType with Connection when the class is declared, it wou=
ld
> > > > require
> > > > > the less work to extend the connection module, as just importing
> the
> > > file
> > > > > with the connection would do the trick.
> > > > > The second one is juste to have a function/classmethod that you
> call
> > > > > manually to register your connection. It would be simpler to
> > understand
> > > > but
> > > > > requires more work every time you create a new ConnectionType.
> > > > >
> > > > > Hope this proposal is clear enough, and I'm waiting for feebacks
> and
> > > > > possible improvements.
> > > > >
> > > > > Regards
> > > > > Gael Magnan de Bornier
> > > > >
> > > > --
> > > >   _/
> > > > _/ Alex Van Boxel
> > > >
> > >
> > --
> >   _/
> > _/ Alex Van Boxel
> >
>
--=20
  _/
_/ Alex Van Boxel

--001a113d729e57bb090545a92352--