airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bo...@apache.org
Subject [1/2] incubator-airflow git commit: [AIRFLOW-443] Make module names unique when importing
Date Wed, 24 Aug 2016 07:06:01 GMT
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 901b60bac -> ee7c6b0e9


[AIRFLOW-443] Make module names unique when importing

Module names are normalized to "unusual_prefix_<filename>" in case
of non-ZIP imports. If there are two modules with the same filename
in a different directory they can overwrite each other. This is
exhibited by the looking at the code from the web interface.

This is fixed by using a hash of the filepath in the normalization.
The hash is required to stay compatible with Python 2.


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/14c868e2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/14c868e2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/14c868e2

Branch: refs/heads/master
Commit: 14c868e2b43d63b8ce79c4337926b6ffb0bc69a0
Parents: 3cba29e
Author: Bolke de Bruin <bolke@xs4all.nl>
Authored: Sat Aug 20 22:16:01 2016 +0200
Committer: Bolke de Bruin <bolke@xs4all.nl>
Committed: Sun Aug 21 20:27:32 2016 +0200

----------------------------------------------------------------------
 airflow/models.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/14c868e2/airflow/models.py
----------------------------------------------------------------------
diff --git a/airflow/models.py b/airflow/models.py
index 9eefaf3..ecb439f 100644
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -42,6 +42,8 @@ import sys
 import textwrap
 import traceback
 import warnings
+import hashlib
+
 from urllib.parse import urlparse
 
 from sqlalchemy import (
@@ -245,7 +247,9 @@ class DagBag(BaseDagBag, LoggingMixin):
 
             self.logger.debug("Importing {}".format(filepath))
             org_mod_name, _ = os.path.splitext(os.path.split(filepath)[-1])
-            mod_name = 'unusual_prefix_' + org_mod_name
+            mod_name = ('unusual_prefix_'
+                        + hashlib.sha1(filepath.encode('utf-8')).hexdigest()
+                        + '_' + org_mod_name)
 
             if mod_name in sys.modules:
                 del sys.modules[mod_name]


Mime
View raw message