airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ikar Pohorsky (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AIRFLOW-1973) Scheduler log dirs permission issue since airflow 1.9.0
Date Wed, 14 Nov 2018 09:53:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686274#comment-16686274
] 

Ikar Pohorsky edited comment on AIRFLOW-1973 at 11/14/18 9:52 AM:
------------------------------------------------------------------

Can still reproduce on 1.10.0 (in the new RBAC web frontend).

The problem is (probably) that the webserver running under root creates the log directory
sooner than the scheduler should.

Suggested fix would be the webserver not to creating directories. Maybe this is too specific
case?


was (Author: porn):
Can still reproduce on 1.10.0.

The problem is (probably) that the webserver running under root creates the log directory
sooner than the scheduler should.

Suggested fix would be the webserver not to creating directories. Maybe this is too specific
case?

> Scheduler log dirs permission issue since airflow 1.9.0
> -------------------------------------------------------
>
>                 Key: AIRFLOW-1973
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1973
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler, webserver
>    Affects Versions: 1.9.0
>            Reporter: Ikar Pohorsky
>            Priority: Major
>
> I'm having a weird (cannot safely reproduce) issue with scheduler log permissions since
version 1.9.0.
> *Problem description:*
> sometimes (not always) it happens that a new log dir for scheduler is created with root
set as owner (and group) which leads to the scheduler being unable to write to the logs and
therefor no task is scheduled and no task is executed.
> Example:
> {code}
> ubuntu@airflow-admin-master:~$ ls -l airflow/logs/scheduler/
> total 24
> drwxr-xr-x 3 ubuntu ubuntu 4096 Jan  3 13:46 2018-01-03
> drwxr-xr-x 3 ubuntu ubuntu 4096 Jan  4 07:16 2018-01-04
> drwxr-xr-x 4 ubuntu ubuntu 4096 Jan  5 12:08 2018-01-05
> drwxr-xr-x 2 root   root   4096 Jan  6 00:00 2018-01-06  # fail
> drwxr-xr-x 4 ubuntu ubuntu 4096 Jan  7 00:00 2018-01-07
> drwxr-xr-x 4 ubuntu ubuntu 4096 Jan  8 00:00 2018-01-08
> lrwxrwxrwx 1 ubuntu ubuntu   46 Jan  8 00:00 latest -> /home/ubuntu/airflow/logs/scheduler/2018-01-08
> {code}
> the other machine:
> {code}
> ubuntu@airflow-client-master:~$ ls -l airflow/logs/scheduler/
> total 24
> drwxr-xr-x 3 ubuntu ubuntu 4096 Jan  3 00:00 2018-01-03
> drwxr-xr-x 3 ubuntu ubuntu 4096 Jan  4 00:00 2018-01-04
> drwxr-xr-x 3 ubuntu ubuntu 4096 Jan  5 12:25 2018-01-05
> drwxr-xr-x 2 ubuntu ubuntu 4096 Jan  6 00:00 2018-01-06
> drwxr-xr-x 2 root   root   4096 Jan  7 00:00 2018-01-07  # fail
> drwxr-xr-x 2 ubuntu ubuntu 4096 Jan  8 00:00 2018-01-08
> lrwxrwxrwx 1 ubuntu ubuntu   46 Jan  8 00:00 latest -> /home/ubuntu/airflow/logs/scheduler/2018-01-08
> {code}
> *My setup:*
> - _scheduler_ runs on EC2 machine under {{ubuntu:ubuntu}} from systemd service
> - _webapp_ runs on the same machine under {{root:root}} in order to be able to serve
on port :80
> - _workers_ run on different machines - actually this is irrelevant info;)
> Also there are two different "environments" - one for "client" and one for "admin". The
admin one is a fresh installation of airflow 1.9.0. The client one is an upgrade from 1.8.2
to 1.9.0. Both encounter the permissions problems.
> I'm not aware of these problems with 1.8.2 version.
> Workaround:
> let's get dirty, add following row into root's crontab:
> {code}
> 1 0 * * * chown ubuntu:ubuntu -Rc /home/airflow/logs/scheduler
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message