airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Van Boxel <a...@vanboxel.be>
Subject Re: Airflow Logging
Date Mon, 24 Oct 2016 19:08:08 GMT
My requirement would indeed be that I would be able to add my own logging
handler (I did the same in my Luigi days), I included a python log handler
that logged to Google Cloud Logging.

But I also like the current logging to Cloud storage.

So my ideal logging setup would be:
All of Airflow (scheduler + webserver + worker) -> Log to custom log handler
Output of DAG -> Log to customer log handler + Cloud *storage* for easy
access in Airflow

It's currently a nice-to-have.




On Tue, Oct 18, 2016 at 6:13 PM Maycock, Luke <
luke.maycock@affiliate.oliverwyman.com> wrote:

> Thank you for the response Maxime. We will consider the points you made
> when making the changes and may come back with further thoughts as we work
> on it.
>
>
> I can't say I am too familiar with Kibana, Logstash and Elasticsearch but
> I am assuming Airflow would still produce logs and you would be using
> Logstash to process the logs?
>
>
> Thanks,
> Luke Maycock
> OLIVER WYMAN
> luke.maycock@affiliate.oliverwyman.com<mailto:
> luke.maycock@affiliate.oliverwyman.com>
> www.oliverwyman.com<http://www.oliverwyman.com/>
>
>
>
> ________________________________
> From: Maxime Beauchemin <maximebeauchemin@gmail.com>
> Sent: 14 October 2016 17:02
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow Logging
>
> Another consideration is for configuration as code in `settings.py`
> in-place or in conjunction with `airflow.cfg` to allow more dynamic
> configuration, like passing your own custom log handler[s]. Many folks at
> the meetup (including Airbnb) spoke about their intent to ship the logging
> to Kibana and/or other logging backend.
>
> Side note about `airflow.cfg` vs `settings.py`: it would be great to move
> completely away from the cfgs for Airflow 2.0. It turns out that Python's
> ConfigParser may be one of the worst module in the standard library and the
> cfg not-so-standard standard isn't great. We have many instances where we
> allow for people to pass objects or function as configuration parameters
> anyways, so let's keep the whole configuration in one place!
>
> Max
>
> On Fri, Oct 14, 2016 at 7:43 AM, Maycock, Luke <
> luke.maycock@affiliate.oliverwyman.com> wrote:
>
> > Carrying on from the below. We believe it would be useful to have a
> number
> > of additional configuration options for logs as below. Do you think there
> > are any important options we have missed or do you have any other
> feedback?
> > If so, please let us know.
> >
> >
> > Global
> >
> > CLEAR_INHERITED_LOGGING_SETTINGS (Boolean)
> > Clears any handlers, inherited from the root logger, from the Airflow
> > logger object and its children.
> >
> > LOG_TO_DEBUG_FILE (Boolean)
> >
> > Turn on/off logging of all events to a file.
> >
> > LOG_TO_ERROR_FILE (Boolean)
> > Turn on/off the logging of all errors to a file.
> >
> > LOG_TO_CONSOLE (Boolean)
> > Turn on/off the logging of all events to the console.
> >
> > Per log file
> > The Python logging library, being used for logging in the Airflow code,
> > has the ability to rotate log files based on time:
> >
> >
> > •         TimedRotatingFileHandler - (https://docs.python.org/2.6/
> > library/logging.html#timedrotatingfilehandler) - rotates based on the
> > product of when and interval (see below).
> >
> > The TimedRotatingFileHandler takes the following arguments:
> >
> >  *   Filename – the filename
> >  *   When – used to specify the type of interval
> >  *   Interval – defines the interval
> >  *   BackupCount - If backupCount is nonzero, at most backupCount files
> > will be kept, and if more would be created when rollover occurs, the
> oldest
> > one is deleted. The deletion logic uses the interval to determine which
> > files to delete, so changing the interval may leave old files lying
> around.
> >  *   Encoding - if encoding is not ‘None’, it is used to open the file
> > with that encoding.
> >  *   Delay - if delay is true, then file opening is deferred until the
> > first call to emit() – I don’t think this is suitable to expose as an
> > Airflow configuration option.
> >  *   UTC - if the UTC argument is true, times in UTC will be used;
> > otherwise local time is used.
> > I believe all of the above, except ‘delay’, should be exposed as Airflow
> > configuration.
> >
> >
> > Cheers,
> > Luke Maycock
> > OLIVER WYMAN
> > luke.maycock@affiliate.oliverwyman.com<mailto:luke.
> > maycock@affiliate.oliverwyman.com>
> > www.oliverwyman.com<http://www.oliverwyman.com/>
> >
> >
> >
> > ________________________________
> > From: Maycock, Luke
> > Sent: 13 October 2016 14:52
> > To: dev@airflow.incubator.apache.org
> > Subject: Airflow Logging
> >
> >
> > Hi All,
> >
> > We (owlabs - fork: https://github.com/owlabs/incubator-airflow) have a
> > high level design for how to improve the logging throughout the Airflow
> > code to be more consistent, maintainable and extensible. We'd really
> > appreciate any feedback on the design.
> >
> > Design for Consolidating Logging Setup:
> >
> >  *   Create a Class in the utils\logging.py to instantiate and handle
> > setup of an "airflow" logger object. This allows us to easily find all
> > setup code for further modification and extension in the future.
> >  *   This class would be where any general logging configuration (from
> > Airflow.cfg) would be applied.
> >  *   Instantiate this class in this file, so that importing it allows
> easy
> > control of the logging setup from anywhere else in the application.
> >  *   Move all setup of the logging in general to this class with easy to
> > call control methods to turn forms of logging on and off (e.g.
> > console/debugFile/errorFile).
> >  *   Move any other helper functions related to logging that are still
> > required into the utils\logging.py so that we can easily find them for
> > modification and extension in the future.
> >  *   Ensure that all logging throughout the application is done via the
> > "airflow" logger object, or by a child of this logger object. This allows
> > us to:
> >     *   Leave all settings of the root logger object alone, so that we
> are
> > friendly to any parent or child processes that are not part of the
> > application, allowing them to safely manage their own overriding logging
> > setup.
> >     *   Modify the settings of logging throughout the entire application
> > via a single simple point of control.
> >
> > Cheers,
> > Luke Maycock
> > OLIVER WYMAN
> > luke.maycock@affiliate.oliverwyman.com<mailto:luke.
> > maycock@affiliate.oliverwyman.com>
> > www.oliverwyman.com<http://www.oliverwyman.com/>
> >
> >
> > ________________________________
> > This e-mail and any attachments may be confidential or legally
> privileged.
> > If you received this message in error or are not the intended recipient,
> > you should destroy the e-mail message and any attachments or copies, and
> > you are prohibited from retaining, distributing, disclosing or using any
> > information contained herein. Please inform us of the erroneous delivery
> by
> > return e-mail. Thank you for your cooperation.
> >
>
> ________________________________
> This e-mail and any attachments may be confidential or legally privileged.
> If you received this message in error or are not the intended recipient,
> you should destroy the e-mail message and any attachments or copies, and
> you are prohibited from retaining, distributing, disclosing or using any
> information contained herein. Please inform us of the erroneous delivery by
> return e-mail. Thank you for your cooperation.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message