airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Machado <pe...@205datalab.com>
Subject Re: Writing logs to s3 in 1.10
Date Tue, 18 Sep 2018 12:54:04 GMT
I haven't done extensive testing as this is a new server but yes, I see the
logs in s3 when using Celery.

One thing that helped me troubleshoot was to start in interactive mode:

docker-compose -f docker-compose up

Without the -d flag. I was able to see some error messages that eventually
led me to the solution.

Did you upgrade from 1.9 or is this a fresh 1.10 implementation?

I think some things changed in the airflow.cfg so in my case, I started
from the 1.10 version.

On Mon, Sep 17, 2018, 9:37 AM vsr.bhavani@gmail.com <vsr.bhavani@gmail.com>
wrote:

>
>
> On 2018/09/17 03:03:06, Pedro Machado <pedro@205datalab.com> wrote:
> > Bhavani,
> >
> > I was able to get it to work. I am using a modified version of
> > https://github.com/puckel/docker-airflow
> >
> > Here is what I had to do:
> >
> > Environment variables:
> >
> > I did not change airflow.cfg, but put the following variables in .env and
> > modified docker-compose.yml to pass them to the containers:
> >
> > In .env:
> >
> > LOGGING_CONFIG_CLASS=log_config.DEFAULT_LOGGING_CONFIG
> > REMOTE_BASE_LOG_FOLDER=s3://<bucket>/airflow-logs-dev/
> > REMOTE_LOG_CONN_ID=<my s3 connection>
> >
> > Then in docker-compose.yml, I passed the following under environment for
> > all the airflow containers:
> >
> >         environment:
> >             - LOAD_EX=n
> >             - FERNET_KEY=<my key>
> >             - EXECUTOR=Celery
> >             - POSTGRES_USER=${POSTGRES_USER}
> >             - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
> >             - POSTGRES_DB=${POSTGRES_DB}
> >             - POSTGRES_HOST=${POSTGRES_HOST}
> >             - AIRFLOW__CORE__LOGGING_CONFIG_CLASS=${LOGGING_CONFIG_CLASS}
> >             - AIRFLOW__CORE__REMOTE_LOGGING=True
> >             -
> > AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER=${REMOTE_BASE_LOG_FOLDER}
> >             - AIRFLOW__CORE__REMOTE_LOG_CONN_ID=${REMOTE_LOG_CONN_ID}
> >             - AIRFLOW__SMTP__SMTP_MAIL_FROM=${SMTP_MAIL_FROM}
> >             - AIRFLOW__WEBSERVER__BASE_URL=${BASE_URL}
> >
> > I then created the following files:
> >
> > __init__.py (empty)
> > log_config.py (used code from
> >
> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py
> > )
> >
> > and I set up my Dockerfile to copy them to the  ${AIRFLOW_HOME}/config/
> > directory:
> >
> > COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
> > COPY config/*.py ${AIRFLOW_HOME}/config/
> >
> > After this, the workers were able to log to s3 successfully.
> >
> > This StackOverfiow answer helped me but I had to make some tweaks:
> >
> https://stackoverflow.com/questions/50222860/airflow-wont-write-logs-to-s3
> >
> > Let me know if this works for you.
> >
> > Pedro
> >
> >
> > On Sun, Sep 16, 2018 at 1:12 AM Bhavani Ramasamy <
> bhavani.ramasamy@vydia.com>
> > wrote:
> >
> > > I am facing the same issue.. Did you find any solution for this?
> > >
> > > On 2018/09/08 01:05:26, Pedro Machado <p...@205datalab.com> wrote:
> > > > I am looking at the documentation here>
> > > >
> > >
> https://airflow.incubator.apache.org/howto/write-logs.html#writing-logs-to-amazon-s3
> >
> > >
> > > > and am wondering if for the s3 configuration it's not necessary to
> > > create a>
> > > > log configuration file as it's described under the GCP section>
> > > >
> > >
> https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform
> >
> > >
> > > >
> > > > I ran a quick test configuring remote_base_log_folder>
> > > > and remote_log_conn_id through environment variables and it didn't
> > > work.>
> > > >
> > > > Could someone shed some light on this?>
> > > >
> > > > Thanks,>
> > > >
> > > > Pedro>
> > > >
> > Hello Pedro,
>
> Thanks for quick response. I am using the similar configurations like you
> mentioned. S3 logs are working when i am using LocalExecutor. But when I am
> using CeleryExecutor it is not writing to S3. I have the celery result
> backend also connected to the same PG database & broker as redis queue. Are
> you able to use CeleryExecutor & run airflow in multiple docker containers
> with S3 logs?
>
> Thanks,
> Bhavani
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message