airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: ImportError: cannot import name HiveOperator (missing dependency on unicodecsv)
Date Tue, 31 May 2016 16:53:39 GMT
Hey Brian,

Try going here to create the JIRA:

https://issues.apache.org/jira/secure/CreateIssue!default.jspa

Cheers,
Chris

On Tue, May 31, 2016 at 9:33 AM, Brian Candler <b.candler@pobox.com> wrote:

> Trying to follow the quickstart at
> http://pythonhosted.org/airflow/start.html (inside a fresh virtualenv,
> under ubuntu 14.04)
>
> I get the following error:
>
> (venv)brian@cfprov:~/airflow$ airflow initdb
> [2016-05-31 17:02:25,939] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> [2016-05-31 17:02:26,049] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-05-31 17:02:26,114] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> DB: sqlite:////home/brian/airflow/airflow.db
> [2016-05-31 17:02:26,361] {db.py:222} INFO - Creating tables
> INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
> INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
> INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current
> schema
> INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 ->
> 1507a7289a2f, create is_encrypted
> /home/brian/airflow/venv/local/lib/python2.7/site-packages/alembic/util/messaging.py:69:
> UserWarning: Skipping unsupported ALTER for creation of implicit constraint
>   warnings.warn(msg)
> INFO  [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO  [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO  [alembic.runtime.migration] Running upgrade 338e90f54d61 ->
> 52d714495f0, job_id indices
> INFO  [alembic.runtime.migration] Running upgrade 52d714495f0 ->
> 502898887f84, Adding extra to Log
> INFO  [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO  [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO  [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO  [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO  [alembic.runtime.migration] Running upgrade 561833c1c74b ->
> 4446e08588, dagrun start end
> INFO  [alembic.runtime.migration] Running upgrade 4446e08588 ->
> bbc73705a13e, Add notification_sent column to sla_miss
> INFO  [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra'
> field in connection
> INFO  [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
> 1968acfc09e3, add is_encrypted column to variable table
> INFO  [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
> 2e82aab8ef20, rename user table
> ERROR [airflow.models.DagBag] Failed to import:
> /home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py
> Traceback (most recent call last):
>   File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/models.py",
> line 247, in process_file
>     m = imp.load_source(mod_name, filepath)
>   File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py",
> line 26, in <module>
>     from airflow.operators import BashOperator, HiveOperator,
> PythonOperator
> ImportError: cannot import name HiveOperator
> Done.
> (venv)brian@cfprov:~/airflow$
>
> Airflow *does* appear to work (the web UI comes up), but I also get the
> same error logged for the execution of the example_bash_operator code:
>
> [2016-05-31 17:04:16,451] {models.py:154} INFO - Filling up the DagBag
> from /home/brian/airflow/dags/example_dags/example_bash_operator.py
> [2016-05-31 17:04:16,452] {models.py:250} ERROR - Failed to import:
> /home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py
> Traceback (most recent call last):
>   File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/models.py",
> line 247, in process_file
>     m = imp.load_source(mod_name, filepath)
>   File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py",
> line 26, in <module>
>     from airflow.operators import BashOperator, HiveOperator,
> PythonOperator
> ImportError: cannot import name HiveOperator
>
>
> If I try it from the command line:
>
> >>> from airflow.operators.hive_operator import HiveOperator
> [2016-05-31 17:24:59,316] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> [2016-05-31 17:24:59,427] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-05-31 17:24:59,484] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/operators/hive_operator.py",
> line 4, in <module>
>     from airflow.hooks import HiveCliHook
> ImportError: cannot import name HiveCliHook
>
> And again:
>
> >>> from airflow.hooks.hive_hooks import HiveCliHook
> [2016-05-31 17:27:11,716] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> [2016-05-31 17:27:11,824] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-05-31 17:27:11,872] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File
> "/home/brian/airflow/venv/local/lib/python2.7/site-packages/airflow/hooks/hive_hooks.py",
> line 19, in <module>
>     import unicodecsv as csv
> ImportError: No module named unicodecsv
>
> Aha. This error has been hidden by magic.After "pip install unicodecsv" it
> seems happy:
>
> >>> from airflow.operators import HiveOperator
> [2016-05-31 17:28:15,828] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> [2016-05-31 17:28:15,936] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-05-31 17:28:15,984] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> >>>
>
> So: I think the "airflow" package is missing a dependency on "unicodecsv".
>
> Regards,
>
> Brian Candler.
>
> P.S. Side problem: I wanted to create an issue on JIRA but couldn't see
> how. I have an Apache JIRA account (username "candlerb"). Under the
> "Create" menu it offers me two choices:
>
> * New JIRA Project     # really, I can do this?!
>
> * Create Service Desk Request     # whatever that is
>
> But all I wanted to do was create an issue. Am I missing something
> obvious? Or are only project developers allowed to raise issues?
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message