airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From airflowuser <airflowu...@protonmail.com.INVALID>
Subject Re: Short Airflow user survey
Date Thu, 28 Feb 2019 08:25:27 GMT
Kamil the Jira contains many requests for new operators.

To name a few:

Zeppelin Operator
https://issues.apache.org/jira/browse/AIRFLOW-1032

scrapyd operator
https://issues.apache.org/jira/browse/AIRFLOW-827

PostgresTableSensor
https://issues.apache.org/jira/browse/AIRFLOW-1759

S3 download file operator
https://issues.apache.org/jira/browse/AIRFLOW-2999

GCS rsync operator
https://issues.apache.org/jira/browse/AIRFLOW-2842


There are also many requests for additional functionality and bug fixes in existed operators.

Regarding the user experience there are many pending PRs in github mainly for the UI. Some
are pending for weeks & months. The project doesn't have enough core committers to review
them all. I noticed that many times if there are no comments to PRs the authors neglect them
after a while but there isn't much we can do about it.



‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, February 27, 2019 7:56 PM, Kamil Breguła <kamil.bregula@polidea.com>
wrote:

> Very interesting results. It turns out areas that require more attention.
> The most popular areas are:
>
> 1.  Documentation - 21 comments
> 2.  User Experience: 11 comments
> 3.  More Operators: 11 comments
>
>     Now I would like to think about what we can do about it.
>
> 4.  The documentation has received a number of PR from my side:
>     My main PR is:
>     Automatic generation of API Reference:
>     https://lists.apache.org/thread.html/9099a3b6700f04138cf607fc4c554b14f3691bae05aef52d10489b1f@<dev.airflow.apache.org>
>     Linking to external documentation
>     https://github.com/apache/airflow/pull/4655:
>     Fix warnings:
>     https://github.com/apache/airflow/pull/4585
>     Validation on documentation on CI:
>     https://github.com/apache/airflow/pull/4593
>     I wonder whether if people would base their experience on these changes,
>     the ratings would be better.
>
>     I miss ideas for future improvement. Do you have any other idea for
>     improvement?
>
> 5.  User Experience
>     User Experience is an area that has not received a lot of patches
>     recently. I think that this could be due to the existence of two variants
>     of the interface: www and www_rbac. Another reason is the lack of automated
>     tests. Testing each fragment manually is very problematic. I think this is
>     an area that needs special attention. Now we have one variant of the
>     interface, so introducing changes is much simpler.
>
>     But i have already started work and want to continue working in my spare
>     time. Now, I prepared a PR that extract Jinja from Javascript:
>     https://github.com/apache/airflow/pull/4787 As the next step, I would like
>     to extract the Javascipt code to separate files. Combining HTML and JS code
>     into one file is not a good idea. After these steps are completed, further
>     improvements will be possible including:
>
>     1.  linting for Javascript and HTML.
>     2.  unit tests for javascript logic,
>     3.  visual regression tests for generated graphs. Before or after droping
>         `nvd3` library
>
>         Regarding the first improvement, It will be a small change. The history of
>         commits does not exist after the www_rbac directory has become a www
>         directory. I think it will not be a change that will cause a lot of
>         discussion, unlike AIP-6. Now is the best time to force the formatting of
>         the code and unify the style.
>
>         Testing Javascript logic is problematic in applications where HTML code is
>         generated by template engines. Separation of the visual layer would
>         significantly improve this process, but at the same time introduced other
>         restrictions. Currently, I would not like to introduce it. However, this
>         will be easier when the REST API is completed.
>
>         The nvd3 library is very problematic for us due to licensing reasons. I
>         think you have to think about deleting it. Visual tests allow you to check
>         if graphs are always generated correctly. Making them is relatively easy,
>         but there is a problem of mixing environment. The JS code is generated by
>         the library in Python
>
>         The world of Javascript has developed drastically recently and we have to
>         catch up with it.
>
>         Do you have any other visions for improvement?
>
> 6.  More operators
>     Airflow is constantly evolving in this area. I do not see any significant
>     limitations. Today, I send a PRs that add 12 operators, 1 sensors,
>     However, we must observe whether there are problems that limit development.
>
>     Do you have ideas for new operators that will be useful?
>
>     On Wed, Feb 27, 2019 at 5:45 PM Sid Anand r39132@gmail.com wrote:
>
>
> > Woot!
> > -s
> > On Mon, Feb 25, 2019 at 9:17 PM Kevin Yang yrqls21@gmail.com wrote:
> >
> > > +1, thank you very much.
> > > On Mon, Feb 25, 2019 at 12:02 PM Maxime Beauchemin <
> > > maximebeauchemin@gmail.com> wrote:
> > >
> > > > +1, this is great and we should do it periodically!
> > > > On Mon, Feb 25, 2019 at 10:42 AM Dan Davydov
> > > > <ddavydov@twitter.com.invalid
> > > >
> > > > >
> > > >
> > > > wrote:
> > > >
> > > > > This is very interesting and useful, big thanks for conducting the
> > > > > survey!
> > > > > On Mon, Feb 25, 2019 at 12:24 PM Ash Berlin-Taylor ash@apache.org
> > > > > wrote:
> > > > >
> > > > > > Thanks for all those who answered, there's some useful answers
in
> > > > > > there.
> > > > >
> > > > > > I've done a short write up
> >
> > https://ash.berlintaylor.com/writings/2019/02/airflow-user-survey-2019/
> >
> > > > > > Headline takeways:
> > > > > > Average rating of 8.3, NPS of 35
> > > > > > Many people expecting to use @ApacheAirflow more this year
> > > > > > Most people (65%) are using Celery executor, with Local and
Kube
> > > > > > most
> > >
> > > > of
> > > >
> > > > > > the rest
> > > > > > Lots of call for sensible requests. Now we just need time to
build
> > > > > > them!
> > > > >
> > > > > > Thanks,
> > > > > > -ash
> > > > > >
> > > > > > > On 14 Feb 2019, at 16:27, Ash Berlin-Taylor ash@apache.org
> > > > > > > wrote:
> > > >
> > > > > > > Hi all,
> > > > > > > It occurs to me I don't really know how people use Airflow,
and
> > > > > > > I'd
> > >
> > > > > like
> > > > >
> > > > > > to know a bit more about what sort of size clusters people run.
> > > > > >
> > > > > > > So if you have 5 minutes spare I'd appreciate filling out
this
> > > > > > > short, 7
> > > > >
> > > > > > question survey: https://ashberlintaylor.typeform.com/to/hIO0Ks
> > > > > >
> > > > > > > I'll give it a week or so then summarise the answers back
here :)
> > > > > > > Thanks,
> > > > > > > Ash
>
> --
>
> Kamil Breguła
> Polidea https://www.polidea.com/ | Software Engineer
>
> M: +48 505 458 451 <+48505458451>
> E: kamil.bregula@polidea.com
> [image: Polidea] https://www.polidea.com/
>
> We create human & business stories through technology.
> Check out our projects! https://www.polidea.com/our-work
> [image: Github] https://github.com/Polidea [image: Facebook]
> https://www.facebook.com/Polidea.Software [image: Twitter]
> https://twitter.com/polidea [image: Linkedin]
> https://www.linkedin.com/company/polidea [image: Instagram]
> https://instagram.com/polidea [image: Behance]
> https://www.behance.net/polidea



Mime
View raw message