airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruslan Dautkhanov <dautkha...@gmail.com>
Subject Re: Airflow profiling
Date Tue, 27 Jun 2017 21:20:17 GMT
Maybe not as powerful as stackImpact, another nice option:
https://www.jetbrains.com/help/pycharm/profiler.html



-- 
Ruslan Dautkhanov

On Tue, Jun 27, 2017 at 2:40 PM, Alex Guziel <alex.guziel@airbnb.com.invalid
> wrote:

> Yeah, actually we have setup Newrelic for Airflow too at Airbnb, which
> gives decent insights into webserver perf. In terms of SQL queries, adding
> `echo=True` to the SQLAlchemy engine creation is pretty good for seeing
> which sql queries get created. I tried some Python profilers before but
> they weren't super helpful.
>
> On Tue, Jun 27, 2017 at 1:27 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > Nice. It would be great if DAG parsing was faster, and some of the
> > endpoints on the website have grown really slow as you we've grown the
> > number of DAGs, and on the DAGs with large number of tasks.
> >
> > I had the intuition that DAG parsing could be faster if operators
> > late-imported hooks (who themselves import external libs) but I have no
> > evidence or test to support it.
> >
> > I'm sure there's tons of low hanging fruit and this type of tool should
> > make it really clear.
> >
> > We've set up NewRelic (which seems similar as this tooling at first
> sight)
> > for Superset at Airbnb and it gave us great insight.
> >
> > Max
> >
> > On Tue, Jun 27, 2017 at 1:01 PM, Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >
> > > Free version also there, maybe more integration testing and
> benchmarking.
> > >
> > > https://stackimpact.com/pricing/ <https://stackimpact.com/pricing/>
> > >
> > > B.
> > >
> > > > On 27 Jun 2017, at 22:00, Chris Riccomini <criccomini@apache.org>
> > wrote:
> > > >
> > > > Seems you have to pay?
> > > >
> > > > On Tue, Jun 27, 2017 at 12:56 PM, Bolke de Bruin <bdbruin@gmail.com>
> > > wrote:
> > > >
> > > >> Just saw this tool on hacker news:
> > > >>
> > > >> https://github.com/stackimpact/stackimpact-python <
> > https://github.com/
> > > >> stackimpact/stackimpact-python>
> > > >>
> > > >> Might be interesting for some profiling.
> > > >>
> > > >> Bolke
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message