airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Machado <pe...@205datalab.com>
Subject Re: Writing logs to s3 in 1.10
Date Mon, 17 Sep 2018 03:03:06 GMT
Bhavani,

I was able to get it to work. I am using a modified version of
https://github.com/puckel/docker-airflow

Here is what I had to do:

Environment variables:

I did not change airflow.cfg, but put the following variables in .env and
modified docker-compose.yml to pass them to the containers:

In .env:

LOGGING_CONFIG_CLASS=log_config.DEFAULT_LOGGING_CONFIG
REMOTE_BASE_LOG_FOLDER=s3://<bucket>/airflow-logs-dev/
REMOTE_LOG_CONN_ID=<my s3 connection>

Then in docker-compose.yml, I passed the following under environment for
all the airflow containers:

        environment:
            - LOAD_EX=n
            - FERNET_KEY=<my key>
            - EXECUTOR=Celery
            - POSTGRES_USER=${POSTGRES_USER}
            - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
            - POSTGRES_DB=${POSTGRES_DB}
            - POSTGRES_HOST=${POSTGRES_HOST}
            - AIRFLOW__CORE__LOGGING_CONFIG_CLASS=${LOGGING_CONFIG_CLASS}
            - AIRFLOW__CORE__REMOTE_LOGGING=True
            -
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER=${REMOTE_BASE_LOG_FOLDER}
            - AIRFLOW__CORE__REMOTE_LOG_CONN_ID=${REMOTE_LOG_CONN_ID}
            - AIRFLOW__SMTP__SMTP_MAIL_FROM=${SMTP_MAIL_FROM}
            - AIRFLOW__WEBSERVER__BASE_URL=${BASE_URL}

I then created the following files:

__init__.py (empty)
log_config.py (used code from
https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py
)

and I set up my Dockerfile to copy them to the  ${AIRFLOW_HOME}/config/
directory:

COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
COPY config/*.py ${AIRFLOW_HOME}/config/

After this, the workers were able to log to s3 successfully.

This StackOverfiow answer helped me but I had to make some tweaks:
https://stackoverflow.com/questions/50222860/airflow-wont-write-logs-to-s3

Let me know if this works for you.

Pedro


On Sun, Sep 16, 2018 at 1:12 AM Bhavani Ramasamy <bhavani.ramasamy@vydia.com>
wrote:

> I am facing the same issue.. Did you find any solution for this?
>
> On 2018/09/08 01:05:26, Pedro Machado <p...@205datalab.com> wrote:
> > I am looking at the documentation here>
> >
> https://airflow.incubator.apache.org/howto/write-logs.html#writing-logs-to-amazon-s3>
>
> > and am wondering if for the s3 configuration it's not necessary to
> create a>
> > log configuration file as it's described under the GCP section>
> >
> https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform>
>
> >
> > I ran a quick test configuring remote_base_log_folder>
> > and remote_log_conn_id through environment variables and it didn't
> work.>
> >
> > Could someone shed some light on this?>
> >
> > Thanks,>
> >
> > Pedro>
> >

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message