Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 039BD200C45 for ; Tue, 14 Mar 2017 04:14:25 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 022B8160B85; Tue, 14 Mar 2017 03:14:25 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 49A56160B5D for ; Tue, 14 Mar 2017 04:14:24 +0100 (CET) Received: (qmail 8115 invoked by uid 500); 14 Mar 2017 03:14:23 -0000 Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list commits@airflow.incubator.apache.org Received: (qmail 8103 invoked by uid 99); 14 Mar 2017 03:14:23 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Mar 2017 03:14:23 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id F0BBAC036E for ; Tue, 14 Mar 2017 03:14:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.568 X-Spam-Level: X-Spam-Status: No, score=-3.568 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_NEUTRAL=0.652, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id wBWsbobrOCe8 for ; Tue, 14 Mar 2017 03:14:21 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id C20895F5C4 for ; Tue, 14 Mar 2017 03:14:20 +0000 (UTC) Received: (qmail 8063 invoked by uid 99); 14 Mar 2017 03:14:20 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Mar 2017 03:14:20 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id E02E5DFE1E; Tue, 14 Mar 2017 03:14:19 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: bolke@apache.org To: commits@airflow.incubator.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: incubator-airflow git commit: [AIRFLOW-974] Fix mkdirs race condition Date: Tue, 14 Mar 2017 03:14:19 +0000 (UTC) archived-at: Tue, 14 Mar 2017 03:14:25 -0000 Repository: incubator-airflow Updated Branches: refs/heads/master b8164cc69 -> c5cc298cf [AIRFLOW-974] Fix mkdirs race condition mkdirs congtained a race condition for when if the directory is created between the os.path.exists and the os.makedirs calls, the os.makedirs will fail with an OSError. This reworks the function to be non-recursive as well, as permission errors were due to umasks being applied. Closes #2147 from bolkedebruin/AIRFLOW-974 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c5cc298c Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c5cc298c Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c5cc298c Branch: refs/heads/master Commit: c5cc298cf16c9777c90aec1fc8cc24bde62f7b2f Parents: b8164cc Author: Bolke de Bruin Authored: Mon Mar 13 20:14:07 2017 -0700 Committer: Bolke de Bruin Committed: Mon Mar 13 20:14:07 2017 -0700 ---------------------------------------------------------------------- airflow/utils/file.py | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c5cc298c/airflow/utils/file.py ---------------------------------------------------------------------- diff --git a/airflow/utils/file.py b/airflow/utils/file.py index 78ddeaa..352755e 100644 --- a/airflow/utils/file.py +++ b/airflow/utils/file.py @@ -44,16 +44,14 @@ def mkdirs(path, mode): :param path: The directory to create :type path: str - :param mode: The mode to give to the directory e.g. 0o755 + :param mode: The mode to give to the directory e.g. 0o755, ignores umask :type mode: int - :return: A list of directories that were created - :rtype: list[str] """ - if not path or os.path.exists(path): - return [] - (head, _) = os.path.split(path) - res = mkdirs(head, mode) - os.mkdir(path) - os.chmod(path, mode) - res += [path] - return res + try: + o_umask = os.umask(0) + os.makedirs(path, mode) + except OSError: + if not os.path.isdir(path): + raise + finally: + os.umask(o_umask)