airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: [PLEASE PARTICIPATE] Brainstorming ideas for Season of Docs
Date Sat, 13 Apr 2019 10:45:47 GMT
One more idea - providing that we will implement AIP-7 Simplified
Development Workflow (
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow)
- maybe it would be worthwhile to take a look at updates in the
"Contributors" document and "Breeze" documentation. That's the new local
development environment which allows to replicate Travis CI tests locally
and have it ready in < 10 minutes from the scratch).

There are two sections/areas there that might need a bit of care and touch
from Technical Writer:

   - I reorganised the "Setting up a development environment" chapter in
   "contributing" document

https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/CONTRIBUTING.md

   - But especially I documented a lot about convenient and fast
   development with Breeze - including screencast, screenshots and a lot hints
   (including some from the recent Ash's Hangout):

https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst

Those are documents I imagine might be super-important for new contributors
to Airflow and it would be great to have them structured and worded in the
way that will be easy to follow. I am super-happy to iterate and cooperate
if we can get this part included!

J.


On Fri, Apr 12, 2019 at 7:37 PM Aizhamal Nurmamat kyzy
<aizhamal@google.com.invalid> wrote:

> Thank you all for sharing the ideas!
>
> We will start moving them to a separate doc and start scoping each idea
> with mentors.
>
> If you have more ideas, keep posting in this thread.
>
> Thank you,
> Aizhamal
>
>
> On Wed, Apr 10, 2019 at 6:34 AM Driesprong, Fokko <fokko@driesprong.frl>
> wrote:
>
> > Thanks Aizhamal for opening the discussion.
> >
> > I would like to see more documentation on the Kubernetes executor, which
> is
> > under heavy development, but the documentation is still lacking. I would
> > love to see some more extended documentation on how to set it up, and how
> > to make the most out of it. I think this the k8s executor has a lot of
> > potential, but isn't fully utilized due to lack of documentation.
> >
> > Cheers, Fokko
> >
> > Op wo 10 apr. 2019 om 11:29 schreef Kaxil Naik <kaxilnaik@gmail.com>:
> >
> > > +1 for using plantuml
> > >
> > > On Wed, Apr 10, 2019, 10:19 Kamil Gałuszka <kamil@flyrlabs.com> wrote:
> > >
> > > > In matter of documentation it would be nice to add PlantUML to
> Sphinx,
> > so
> > > > we can visualise a lot of complexities of Airflow (see here:
> > > > https://pypi.org/project/sphinxcontrib-plantuml/).
> > > >
> > > > Especially that sometimes one diagram can speak much clearly than
> wall
> > of
> > > > text.
> > > >
> > > > Not sure how others see that, but I found PlantUML easy to use and
> > > helping
> > > > a lot dealing with complexity of documenting more advanced concepts.
> I
> > > can
> > > > prepare JIRA ticket and PR if there is interest in that.
> > > >
> > > > Thanks
> > > > Kamil
> > > >
> > > > On Wed, Apr 10, 2019 at 8:04 AM Kamil Breguła <
> > kamil.bregula@polidea.com
> > > >
> > > > wrote:
> > > >
> > > > > - The structure of the documentation requires rethinking.
> > > > > A lot of information is contained on one subpage - "concepts". Some
> > > > > information should not be on this subpage e.x. .airflowignore
> > > > > Other information should be extended, but then they will occupy too
> > > much
> > > > > space on this page e. x. creating relation including via chain
> method
> > > > > https://github.com/apache/airflow/pull/4779
> > > > >
> > > > > - Difference between Kubernetes Executor and Operator
> > > > > The difference is not obvious for beginners.
> > > > >
> > > > > - How to setup Kubernetes Executor.
> > > > > It causes problems.
> > > > >
> > https://apache-airflow.slack.com/archives/CCV3FV9KL/p1554319091033300
> > > > >
> > https://apache-airflow.slack.com/archives/CCV3FV9KL/p1553708569192000
> > > > >
> > > > > - How Airflow distributes tasks.
> > > > > Inspiration:
> > > > >
> > https://blog.sicara.com/using-airflow-with-celery-workers-54cb5212d405
> > > > >
> > > > > - Description of Airflow architecture and description of components
> > > > > Scheduler/webserver/worker/metadata db
> > > > > I would like the part of this document to be a diagram similar to
> > this:
> > > > > https://imgur.com/a/YGpg5Wa
> > > > >
> > > > > - Setup Prometheus and Grafana - simple introduction
> > > > >
> > > > > - Recommendations on how to create new operators.
> > > > > When reviewing new integration code(hook/operators)  very often the
> > > same
> > > > > mistakes are made. I would like to have one document to which I
> could
> > > > > refer.
> > > > >
> > > > > - Xcom and macros are a lot of problems for new users.
> > > > > The documentation is not well described. I am sending other people
> > > > > references to describe the fields in the class when I want to
> explain
> > > > this
> > > > > concept.
> > > > >
> > https://apache-airflow.slack.com/archives/CCR6P6JRL/p1551967217291400
> > > > >
> > > > > I think it's worth reviewing the list of articles that the
> community
> > > > > wrote.  This can be the source of inspiration for new pages.
> > > > > https://github.com/jghoman/awesome-apache-airflow
> > > > >
> > > > > I will try to write down the problems that appear in the community.
> > In
> > > > the
> > > > > next step, we can check whether the solution is in the
> documentation
> > > and
> > > > is
> > > > > clearly explained.
> > > > >
> > > > > I think PMCs should look more closely at the documentation when
> > > > accepting a
> > > > > new piece of code. I look at a large amount of PR and very often
> > > suggest
> > > > to
> > > > > write new documentation. I feel that I am alone in this matter. Any
> > new
> > > > > functionality that has only docstring can be accepted.
> > > > > For example:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/commit/b78193240ad4266a7a22dced3832f51a9dce1897
> > > > > A new function has been added, but the description in doc is
> missing.
> > > > > https://airflow.readthedocs.io/en/latest/concepts.html#variables
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Apr 10, 2019, 3:01 AM Gerardo Curiel <gerardo@gerar.do>
> > wrote:
> > > > >
> > > > > > - Deployment in {AWS, Azure, GCP}
> > > > > > - Monitoring, health checks, etc
> > > > > > - Best practices for {dags, subdags, dag scheduling}
> > > > > >
> > > > > > Gerard Toonstra has some amazing docs here for some of these
> topics
> > > as
> > > > > > well: https://gtoonstra.github.io/etl-with-airflow/
> > > > > >
> > > > > > <https://gtoonstra.github.io/etl-with-airflow/>
> > > > > >
> > > > > > On 10 April 2019 at 5:56:52 am, Ash Berlin-Taylor (
> ash@apache.org)
> > > > > wrote:
> > > > > >
> > > > > > - how to configure different types of connections
> > > > > > - how to rerun a failed task.
> > > > > > - tips for designing dags
> > > > > > - writing custom operators (or tweaking the behaviour of a
> built-in
> > > > one)
> > > > > > without needing plug in.
> > > > > >
> > > > > > Ash
> > > > > >
> > > > > > On 9 April 2019 20:52:02 BST, Aizhamal Nurmamat kyzy
> > > > > > <aizhamal@google.com.INVALID> wrote:
> > > > > > >Hello all,
> > > > > > >
> > > > > > >Let's put together ideas for the Season of Docs. Can you
please
> > > share
> > > > > > >any
> > > > > > >documentation pain points that you are aware of as a user
or as
> a
> > > > > > >contributor? Anything where you think the Airflow docs could
be
> > > > > > >improved
> > > > > > >significantly?
> > > > > > >
> > > > > > >Some ideas that members of the community have shared with
me
> are:
> > > > > > >
> > > > > > >- Document how to unit-test DAGs
> > > > > > >- Document how to set up a local testing environment
> > > > > > >- Go through JIRA documentation requests.
> > > > > > >
> > > > > > >Please share any other ideas that you may have. There are
no bad
> > > ideas
> > > > > > >here. We want to generate as many as possible, and prioritize
> the
> > > most
> > > > > > >important ones later.
> > > > > > >
> > > > > > >Thank you,
> > > > > > >Aizhamal
> > > > > >
> > > > > > --
> > > > > > Gerardo Curiel // https://gerar.do
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
E: jarek.potiuk@polidea.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message