airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Ishikawa (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1461) BigQueryOperator has a bug on destination_dataset_table
Date Tue, 25 Jul 2017 22:22:00 GMT
Yu Ishikawa created AIRFLOW-1461:
------------------------------------

             Summary: BigQueryOperator has a bug on destination_dataset_table
                 Key: AIRFLOW-1461
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1461
             Project: Apache Airflow
          Issue Type: Bug
          Components: contrib, operators
    Affects Versions: 1.8.2
            Reporter: Yu Ishikawa


h3. Environment

- Python 2.7
- apache-airflow==1.8.2rc1

h3. Code

{noformat}
dataset_id1 = 'machine_learning_us'
table_id_prefix1 = 'stats_item_view_by_category'
destination_dataset_table1 = "%s:%s.%s_{{ ds_nodash }}" % (project_id, dataset_id1, table_id_prefix1),
destination_dataset_table1 = "%s:%s.%s" % (project_id, dataset_id1, table_id_prefix1),
t1 = BigQueryOperator(
    dag=dag,
    task_id=table_id_prefix1,
    bigquery_conn_id=get_default_google_cloud_connection_id(),
    bql=query1,
    destination_dataset_table=destination_dataset_table1,
    allow_large_results=True,
    use_legacy_sql=False,
)
{noformat}

h3. Log

{noformat}
[2017-07-25 20:28:56,697] {base_task_runner.py:95} INFO - Subtask: [2017-07-25 20:28:56,697]
{models.py:1478} ERROR - Expected destination_dataset_table in the format of <dataset>.<table>.
Got: [u'dummy-project-id:machine_learning_us.stats_item_view_by_category_20170707']
[2017-07-25 20:28:56,697] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent
call last):
[2017-07-25 20:28:56,697] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/bin/airflow",
line 28, in <module>
[2017-07-25 20:28:56,697] {base_task_runner.py:95} INFO - Subtask:     args.func(args)
[2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py",
line 422, in run
[2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:     pool=args.pool,
[2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py",
line 53, in wrapper
[2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:     result = func(*args,
**kwargs)
[2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py",
line 1390, in run
[2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:     result = task_copy.execute(context=context)
[2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/operators/bigquery_operator.py",
line 82, in execute
[2017-07-25 20:28:56,699] {base_task_runner.py:95} INFO - Subtask:     self.allow_large_results,
self.udf_config, self.use_legacy_sql)
[2017-07-25 20:28:56,699] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/bigquery_hook.py",
line 225, in run_query
[2017-07-25 20:28:56,699] {base_task_runner.py:95} INFO - Subtask:     '<dataset>.<table>.
Got: {}').format(destination_dataset_table)
[2017-07-25 20:28:56,700] {base_task_runner.py:95} INFO - Subtask: AssertionError: Expected
destination_dataset_table in the format of <dataset>.<table>. Got: [u'dummy-project-id:machine_learning_us.stats_item_view_by_category_20170707']
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message