airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Maritato <fmarit...@opentable.com.INVALID>
Subject Re: [External] Re: scheduler logging
Date Tue, 23 Apr 2019 20:09:42 GMT
Those two issues don't seem to be present in 1.10.0. I only see one message
for "db connection invalidated" and I don't see anything about harvesting.

I have tried using a custom logger to see if that effects the output to
airflow-scheduler.{log|out} but so far I haven't had any luck. The logs
that go to /var/log/airflow/scheduler are affected, but
AIRFLOW_HOME/airflow-scheduler.log always is printing out INFO or below.

Is there any documentation on how to do this? Or examples somewhere?

On Tue, Apr 23, 2019 at 11:39 AM Daniel Standish <dpstandish@gmail.com>
wrote:

> I noticed two issues in 1.10.2, one of which has been resolved, but I am
> not sure if they were present in 1.10.0, or if they are the messages that
> are bothering you.
>
> 1. "Harvesting DAG parsing results" was printed every 2 seconds or
> something.
> This was resolved in commit [AIRFLOW-3911] Change Harvesting DAG parsing
> results to DEBUG log level (#4729) which I believe is in 1.10.3 now.
> It just changes log level to debug for that message.
>
> 2. Frequent "db connection invalidated" warning
> This one is unsolved.  I created a ticket here:
> https://issues.apache.org/jira/browse/AIRFLOW-4134
> It seems like every 5 seconds the warning "db connection invalidated" is
> logged.
> It happens in a connection reconnect try loop.  It always seems to be able
> to reconnect on first try, so one idea is we could just set it to only warn
> if the first retry fails (debug on first reconnect).  But it would be more
> satisfying to figure out why this connection always seems to be invalidated
> and fix the root cause.  Alas I am not sure how to proceed...
>
>
>
> On Tue, Apr 23, 2019 at 9:54 AM Bolke de Bruin <bdbruin@gmail.com> wrote:
>
> > It's probably better to create a custom logging.conf and use that
> instead.
> >
> > B.
> >
> > Op di 23 apr. 2019 18:13 schreef Frank Maritato
> > <fmaritato@opentable.com.invalid>:
> >
> > > No one else has this issue? Or no one has a solution?
> > >
> > >
> > > On Wed, Apr 17, 2019 at 5:49 PM Frank Maritato <
> fmaritato@opentable.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > We are running airflow 1.10.0 and I'm wondering how I can turn off or
> > > turn
> > > > down the logging for the scheduler? I tried setting
> logging_level=WARN
> > in
> > > > airflow.cfg and restarting the process but I'm still seeing a ton of
> > info
> > > > logging to .out and .log.
> > > >
> > > > Thanks!
> > > > --
> > > > Frank Maritato
> > > >
> > >
> > >
> > > --
> > > Frank Maritato
> > >
> >
>


-- 
Frank Maritato

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message