airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Silk (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-3171) Flexible task log organization
Date Tue, 09 Oct 2018 01:32:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gabriel Silk updated AIRFLOW-3171:
----------------------------------
    Description: 
Regardless of the backend (eg file system, s3, ...), it would be useful to be able to organize
the task logs in a more flexible manner, rather than defaulting to a flat structure.

 

One use case of this would be to provide a better multi-tenancy experience when deploying
a single airflow cluster to several teams. For example, if the log folders in s3 were organized
like /tasks/[owner]/... then we could provide access to a subset of the logs for each team,
by creating s3 access rules prefixed with the appropriate path.

 

One possible implementation would be to have a configurable, templatized path structure for
logs. We would also need to store the log location for each task instance, so we could easily
change the log folder structure without breaking old log paths.

  was:
Regardless of the backend (eg file system, s3, ...), it would be useful to be able to organize
the logs in a more flexible manner, rather than defaulting to a flat structure.

 

One use case of this would be to provide a better multi-tenancy experience when deploying
a single airflow cluster to several teams. For example, if the log folders in s3 were organized
like /tasks/[owner]/... then we could provide access to a subset of the logs for each team,
by creating s3 access rules prefixed with the appropriate path.

 

One possible implementation would be to have a configurable, templatized path structure for
logs. We would also need to store the log location for each task instance, so we could easily
change the log folder structure without breaking old log paths.


> Flexible task log organization
> ------------------------------
>
>                 Key: AIRFLOW-3171
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: logging
>            Reporter: Gabriel Silk
>            Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be able to
organize the task logs in a more flexible manner, rather than defaulting to a flat structure.
>  
> One use case of this would be to provide a better multi-tenancy experience when deploying
a single airflow cluster to several teams. For example, if the log folders in s3 were organized
like /tasks/[owner]/... then we could provide access to a subset of the logs for each team,
by creating s3 access rules prefixed with the appropriate path.
>  
> One possible implementation would be to have a configurable, templatized path structure
for logs. We would also need to store the log location for each task instance, so we could
easily change the log folder structure without breaking old log paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message