airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Guziel <alex.guz...@airbnb.com.INVALID>
Subject Re: Airflow profiling
Date Tue, 27 Jun 2017 20:40:47 GMT
Yeah, actually we have setup Newrelic for Airflow too at Airbnb, which
gives decent insights into webserver perf. In terms of SQL queries, adding
`echo=True` to the SQLAlchemy engine creation is pretty good for seeing
which sql queries get created. I tried some Python profilers before but
they weren't super helpful.

On Tue, Jun 27, 2017 at 1:27 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> Nice. It would be great if DAG parsing was faster, and some of the
> endpoints on the website have grown really slow as you we've grown the
> number of DAGs, and on the DAGs with large number of tasks.
>
> I had the intuition that DAG parsing could be faster if operators
> late-imported hooks (who themselves import external libs) but I have no
> evidence or test to support it.
>
> I'm sure there's tons of low hanging fruit and this type of tool should
> make it really clear.
>
> We've set up NewRelic (which seems similar as this tooling at first sight)
> for Superset at Airbnb and it gave us great insight.
>
> Max
>
> On Tue, Jun 27, 2017 at 1:01 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:
>
> > Free version also there, maybe more integration testing and benchmarking.
> >
> > https://stackimpact.com/pricing/ <https://stackimpact.com/pricing/>
> >
> > B.
> >
> > > On 27 Jun 2017, at 22:00, Chris Riccomini <criccomini@apache.org>
> wrote:
> > >
> > > Seems you have to pay?
> > >
> > > On Tue, Jun 27, 2017 at 12:56 PM, Bolke de Bruin <bdbruin@gmail.com>
> > wrote:
> > >
> > >> Just saw this tool on hacker news:
> > >>
> > >> https://github.com/stackimpact/stackimpact-python <
> https://github.com/
> > >> stackimpact/stackimpact-python>
> > >>
> > >> Might be interesting for some profiling.
> > >>
> > >> Bolke
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message