airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From san...@apache.org
Subject incubator-airflow git commit: [AIRFLOW-399] - Remove dags/testdruid.py
Date Mon, 08 Aug 2016 16:24:09 GMT
Repository: incubator-airflow
Updated Branches:
  refs/heads/master d69fb31dd -> 62768bc08


[AIRFLOW-399] - Remove dags/testdruid.py

Remove the current example in dags/testdruid.py. It requires the installation of an extraneous
library (Hive) and executes a query against table that no one has. In its place, I am creating
a simple DAG that only depends on DummyOperator and standard Python packages.

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- https://issues.apache.org/jira/browse/AIRFLOW-399

Testing Done:
Manual.

aoen criccomini artwr jlowin bolkedebruin

Closes #1709 from r39132/master


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/62768bc0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/62768bc0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/62768bc0

Branch: refs/heads/master
Commit: 62768bc087e95bf4506a8ec24d4291d7dfa7abaf
Parents: d69fb31
Author: Siddharth Anand <siddharthanand@yahoo.com>
Authored: Mon Aug 8 09:23:49 2016 -0700
Committer: Siddharth Anand <siddharthanand@yahoo.com>
Committed: Mon Aug 8 09:23:49 2016 -0700

----------------------------------------------------------------------
 dags/test_dag.py  | 37 +++++++++++++++++++++++++++++++++++++
 dags/testdruid.py | 33 ---------------------------------
 2 files changed, 37 insertions(+), 33 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/62768bc0/dags/test_dag.py
----------------------------------------------------------------------
diff --git a/dags/test_dag.py b/dags/test_dag.py
new file mode 100644
index 0000000..9c506b2
--- /dev/null
+++ b/dags/test_dag.py
@@ -0,0 +1,37 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from airflow import DAG
+from airflow.operators.dummy_operator import DummyOperator
+from datetime import datetime
+
+now = datetime.now()
+now_to_the_hour = now.replace(hour=now.time().hour-3 , minute=0, second=0, microsecond=0)
+START_DATE = now_to_the_hour 
+DAG_NAME = 'test_dag_v1'
+
+default_args = {
+    'owner': 'airflow',
+    'depends_on_past': True,
+    'start_date': START_DATE,
+}
+dag = DAG(DAG_NAME, schedule_interval='*/10 * * * *', default_args=default_args)
+
+run_this_1 = DummyOperator(task_id='run_this_1', dag=dag)
+run_this_2 = DummyOperator(task_id='run_this_2', dag=dag)
+run_this_2.set_upstream(run_this_1)
+run_this_3 = DummyOperator(task_id='run_this_3', dag=dag)
+run_this_3.set_upstream(run_this_2)
+
+

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/62768bc0/dags/testdruid.py
----------------------------------------------------------------------
diff --git a/dags/testdruid.py b/dags/testdruid.py
deleted file mode 100644
index 2af8f51..0000000
--- a/dags/testdruid.py
+++ /dev/null
@@ -1,33 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from airflow.operators.hive_to_druid import HiveToDruidTransfer
-from airflow import DAG
-from datetime import datetime
-
-args = {
-            'owner': 'qi_wang',
-            'start_date': datetime(2015, 4, 4),
-}
-
-dag = DAG("test_druid", default_args=args)
-
-
-HiveToDruidTransfer(task_id="load_dummy_test",
-                    sql="select * from qi.druid_test_dataset_w_platform_1 \
-                            limit 10;",
-                    druid_datasource="airflow_test",
-                    ts_dim="ds",
-                    dag=dag
-                )


Mime
View raw message