airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vsr.bhavani@gmail.com <vsr.bhav...@gmail.com>
Subject Re: Writing logs to s3 in 1.10
Date Mon, 17 Sep 2018 14:30:55 GMT


On 2018/09/17 03:03:06, Pedro Machado <pedro@205datalab.com> wrote: 
> Bhavani,
> 
> I was able to get it to work. I am using a modified version of
> https://github.com/puckel/docker-airflow
> 
> Here is what I had to do:
> 
> Environment variables:
> 
> I did not change airflow.cfg, but put the following variables in .env and
> modified docker-compose.yml to pass them to the containers:
> 
> In .env:
> 
> LOGGING_CONFIG_CLASS=log_config.DEFAULT_LOGGING_CONFIG
> REMOTE_BASE_LOG_FOLDER=s3://<bucket>/airflow-logs-dev/
> REMOTE_LOG_CONN_ID=<my s3 connection>
> 
> Then in docker-compose.yml, I passed the following under environment for
> all the airflow containers:
> 
>         environment:
>             - LOAD_EX=n
>             - FERNET_KEY=<my key>
>             - EXECUTOR=Celery
>             - POSTGRES_USER=${POSTGRES_USER}
>             - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
>             - POSTGRES_DB=${POSTGRES_DB}
>             - POSTGRES_HOST=${POSTGRES_HOST}
>             - AIRFLOW__CORE__LOGGING_CONFIG_CLASS=${LOGGING_CONFIG_CLASS}
>             - AIRFLOW__CORE__REMOTE_LOGGING=True
>             -
> AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER=${REMOTE_BASE_LOG_FOLDER}
>             - AIRFLOW__CORE__REMOTE_LOG_CONN_ID=${REMOTE_LOG_CONN_ID}
>             - AIRFLOW__SMTP__SMTP_MAIL_FROM=${SMTP_MAIL_FROM}
>             - AIRFLOW__WEBSERVER__BASE_URL=${BASE_URL}
> 
> I then created the following files:
> 
> __init__.py (empty)
> log_config.py (used code from
> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py
> )
> 
> and I set up my Dockerfile to copy them to the  ${AIRFLOW_HOME}/config/
> directory:
> 
> COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
> COPY config/*.py ${AIRFLOW_HOME}/config/
> 
> After this, the workers were able to log to s3 successfully.
> 
> This StackOverfiow answer helped me but I had to make some tweaks:
> https://stackoverflow.com/questions/50222860/airflow-wont-write-logs-to-s3
> 
> Let me know if this works for you.
> 
> Pedro
> 
> 
> On Sun, Sep 16, 2018 at 1:12 AM Bhavani Ramasamy <bhavani.ramasamy@vydia.com>
> wrote:
> 
> > I am facing the same issue.. Did you find any solution for this?
> >
> > On 2018/09/08 01:05:26, Pedro Machado <p...@205datalab.com> wrote:
> > > I am looking at the documentation here>
> > >
> > https://airflow.incubator.apache.org/howto/write-logs.html#writing-logs-to-amazon-s3>
> >
> > > and am wondering if for the s3 configuration it's not necessary to
> > create a>
> > > log configuration file as it's described under the GCP section>
> > >
> > https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform>
> >
> > >
> > > I ran a quick test configuring remote_base_log_folder>
> > > and remote_log_conn_id through environment variables and it didn't
> > work.>
> > >
> > > Could someone shed some light on this?>
> > >
> > > Thanks,>
> > >
> > > Pedro>
> > >
> Hello Pedro,

Thanks for quick response. I am using the similar configurations like you mentioned. S3 logs
are working when i am using LocalExecutor. But when I am using CeleryExecutor it is not writing
to S3. I have the celery result backend also connected to the same PG database & broker
as redis queue. Are you able to use CeleryExecutor & run airflow in multiple docker containers
with S3 logs?

Thanks,
Bhavani

Mime
View raw message