airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject Re: new logging
Date Mon, 16 Oct 2017 18:00:39 GMT
sorry i was not clear.

In DAG definition file, I would normally import logging and when use with
python operator like below (see print_context).

In 1.8.2 i would see in Airflow log file (for that task) both 'hi from
print' and 'hi from logging'.

now I can only see 'hi from print'.

I installed from master (pip install), and used newer airflow.cfg file.
Only updated it to use local executor and mysql db. Did not change any
other settings.

So my question if it is supposed to work like that and how do I log now
from python operators in a dag.


´╗┐from __future__ import print_function
from builtins import range
import airflow
from airflow.operators.python_operator import PythonOperator
from airflow.models import DAG
import logging
import time
from pprint import pprint

args = {
    'owner': 'airflow',
    'start_date': airflow.utils.dates.days_ago(2)
}

dag = DAG(
    dag_id='python_test_logging', default_args=args,
    schedule_interval=None)

def print_context(ds, **kwargs):
    pprint(kwargs)
    print('hi from print')
    logging.info('hi from loggin info')

run_this = PythonOperator(
    task_id='print_the_context',
    provide_context=True,
    python_callable=print_context,
    dag=dag)


On Mon, Oct 16, 2017 at 1:09 PM, Driesprong, Fokko <fokko@driesprong.frl>
wrote:

> Hi Boris,
>
> Can you please elaborate? Where did the output used to end up? Did you
> provide any custom logging config, or are you using the default config?
> Maybe drop some example code.
>
> Cheers, Fokko
>
> 2017-10-16 19:02 GMT+02:00 Boris <boriskey@gmail.com>:
>
> > Hi guys,
> >
> > I used to do logging.info("somthing") from airflow python operators so i
> > can see output in airflow logs. Worked fine in 1.8.2.
> >
> > Looks like this is no longer the case with master branch.I did look at
> > updating.md but still not clear how to log custom messages from python
> > operators.
> >
> > please advise.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message