airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vsr.bhavani@gmail.com <vsr.bhav...@gmail.com>
Subject Re: S3 logging not working with docker
Date Sun, 16 Sep 2018 17:47:08 GMT


On 2018/09/16 05:53:02, Bhavani Ramasamy <vsr.bhavani@gmail.com> wrote: 
> Hello Team,
> I am trying to setup S3 logging with docker & CeleryExecutor. Files are not
> written to S3. I have configured in airflow.cfg like below,
> 
> remote_logging = True
> 
> remote_log_conn_id = s3_connection_mine
> 
> remote_base_log_folder = s3:// mybucket/airflow/logs/
> 
> 
> I have tried with *logging_config_class* as empty as well as custom
> log_config.py using airflow_local_settings.py file. It also doesnt work.
> Can you please help me.
> 
> 
> Thanks,
> 
> Bhavani
> Hello Kyle,
I am not using docker operator. I am trying to run apache-airflow 1.10 inside a docker container.
When i use LocalExecutor, i am able to write logs to S3 since all the airflow webserver &
scheduler resides inside one docker container. But when i try to use CeleryExecutor, the logs
are not getting written to S3. Is there any special configuration needed when using CeleryExecutor
& S3 logs?
Thanks,
Bhavani

Mime
View raw message