airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Candler <b.cand...@pobox.com>
Subject ImportError: cannot import name HiveOperator (missing dependency on unicodecsv)
Date Tue, 31 May 2016 16:33:12 GMT
Trying to follow the quickstart at 
http://pythonhosted.org/airflow/start.html (inside a fresh virtualenv, 
under ubuntu 14.04)

I get the following error:

(venv)brian@cfprov:~/airflow$ airflow initdb
[2016-05-31 17:02:25,939] {__init__.py:36} INFO - Using executor 
SequentialExecutor
[2016-05-31 17:02:26,049] {driver.py:120} INFO - Generating grammar 
tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-05-31 17:02:26,114] {driver.py:120} INFO - Generating grammar 
tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
DB: sqlite:////home/brian/airflow/airflow.db
[2016-05-31 17:02:26,361] {db.py:222} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, 
current schema
INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 
1507a7289a2f, create is_encrypted
/home/brian/airflow/venv/local/lib/python2.7/site-packages/alembic/util/messaging.py:69: 
UserWarning: Skipping unsupported ALTER for creation of implicit constraint
   warnings.warn(msg)
INFO  [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 
13eb55f81627, maintain history for compatibility with earlier migrations
INFO  [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 
338e90f54d61, More logging into task_isntance
INFO  [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 
52d714495f0, job_id indices
INFO  [alembic.runtime.migration] Running upgrade 52d714495f0 -> 
502898887f84, Adding extra to Log
INFO  [alembic.runtime.migration] Running upgrade 502898887f84 -> 
1b38cef5b76e, add dagrun
INFO  [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 
2e541a1dcfed, task_duration
INFO  [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 
40e67319e3a9, dagrun_config
INFO  [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 
561833c1c74b, add password column to user
INFO  [alembic.runtime.migration] Running upgrade 561833c1c74b -> 
4446e08588, dagrun start end
INFO  [alembic.runtime.migration] Running upgrade 4446e08588 -> 
bbc73705a13e, Add notification_sent column to sla_miss
INFO  [alembic.runtime.migration] Running upgrade bbc73705a13e -> 
bba5a7cfc896, Add a column to track the encryption state of the 'Extra' 
field in connection
INFO  [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 
1968acfc09e3, add is_encrypted column to variable table
INFO  [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 
2e82aab8ef20, rename user table
ERROR [airflow.models.DagBag] Failed to import: 
/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py
Traceback (most recent call last):
   File 
"/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/models.py", 
line 247, in process_file
     m = imp.load_source(mod_name, filepath)
   File 
"/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py",

line 26, in <module>
     from airflow.operators import BashOperator, HiveOperator, 
PythonOperator
ImportError: cannot import name HiveOperator
Done.
(venv)brian@cfprov:~/airflow$

Airflow *does* appear to work (the web UI comes up), but I also get the 
same error logged for the execution of the example_bash_operator code:

[2016-05-31 17:04:16,451] {models.py:154} INFO - Filling up the DagBag from /home/brian/airflow/dags/example_dags/example_bash_operator.py
[2016-05-31 17:04:16,452] {models.py:250} ERROR - Failed to import: /home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py
Traceback (most recent call last):
   File "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/models.py", line
247, in process_file
     m = imp.load_source(mod_name, filepath)
   File "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py",
line 26, in <module>
     from airflow.operators import BashOperator, HiveOperator, PythonOperator
ImportError: cannot import name HiveOperator


If I try it from the command line:

 >>> from airflow.operators.hive_operator import HiveOperator
[2016-05-31 17:24:59,316] {__init__.py:36} INFO - Using executor 
SequentialExecutor
[2016-05-31 17:24:59,427] {driver.py:120} INFO - Generating grammar 
tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-05-31 17:24:59,484] {driver.py:120} INFO - Generating grammar 
tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File 
"/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/operators/hive_operator.py",

line 4, in <module>
     from airflow.hooks import HiveCliHook
ImportError: cannot import name HiveCliHook

And again:

 >>> from airflow.hooks.hive_hooks import HiveCliHook
[2016-05-31 17:27:11,716] {__init__.py:36} INFO - Using executor 
SequentialExecutor
[2016-05-31 17:27:11,824] {driver.py:120} INFO - Generating grammar 
tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-05-31 17:27:11,872] {driver.py:120} INFO - Generating grammar 
tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File 
"/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/hooks/hive_hooks.py",

line 19, in <module>
     import unicodecsv as csv
ImportError: No module named unicodecsv

Aha. This error has been hidden by magic.After "pip install unicodecsv" 
it seems happy:

 >>> from airflow.operators import HiveOperator
[2016-05-31 17:28:15,828] {__init__.py:36} INFO - Using executor 
SequentialExecutor
[2016-05-31 17:28:15,936] {driver.py:120} INFO - Generating grammar 
tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-05-31 17:28:15,984] {driver.py:120} INFO - Generating grammar 
tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
 >>>

So: I think the "airflow" package is missing a dependency on "unicodecsv".

Regards,

Brian Candler.

P.S. Side problem: I wanted to create an issue on JIRA but couldn't see 
how. I have an Apache JIRA account (username "candlerb"). Under the 
"Create" menu it offers me two choices:

* New JIRA Project     # really, I can do this?!

* Create Service Desk Request     # whatever that is

But all I wanted to do was create an issue. Am I missing something 
obvious? Or are only project developers allowed to raise issues?


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message