Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Thu, 27 Jul 2017 21:15:01 +0000 (UTC)
From: "Yu Ishikawa (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.13090064.1501021306000.30893.1501190101561@Atlassian.JIRA>
In-Reply-To: <JIRA.13090064.1501021306000@Atlassian.JIRA>
References: <JIRA.13090064.1501021306000@Atlassian.JIRA> <JIRA.13090064.1501021306807@jira-lw-us.apache.org>
Subject: [jira] [Closed] (AIRFLOW-1461) BigQueryOperator has a bug on
 destination_dataset_table
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 27 Jul 2017 21:15:12 -0000


     [ https://issues.apache.org/jira/browse/AIRFLOW-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yu Ishikawa closed AIRFLOW-1461.
--------------------------------
    Resolution: Cannot Reproduce

The cause is derived my environment, sorry.

> BigQueryOperator has a bug on destination_dataset_table
> -------------------------------------------------------
>
>                 Key: AIRFLOW-1461
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1461
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib, operators
>    Affects Versions: 1.8.2
>            Reporter: Yu Ishikawa
>
> h3. Environment
> - Python 2.7
> - apache-airflow==1.8.2rc1
> h3. Code
> {noformat}
> dataset_id1 = 'machine_learning_us'
> table_id_prefix1 = 'stats_item_view_by_category'
> destination_dataset_table1 = "%s:%s.%s_{{ ds_nodash }}" % (project_id, dataset_id1, table_id_prefix1),
> destination_dataset_table1 = "%s:%s.%s" % (project_id, dataset_id1, table_id_prefix1),
> t1 = BigQueryOperator(
>     dag=dag,
>     task_id=table_id_prefix1,
>     bigquery_conn_id=get_default_google_cloud_connection_id(),
>     bql=query1,
>     destination_dataset_table=destination_dataset_table1,
>     allow_large_results=True,
>     use_legacy_sql=False,
> )
> {noformat}
> h3. Log
> {noformat}
> [2017-07-25 20:28:56,697] {base_task_runner.py:95} INFO - Subtask: [2017-07-25 20:28:56,697] {models.py:1478} ERROR - Expected destination_dataset_table in the format of <dataset>.<table>. Got: [u'dummy-project-id:machine_learning_us.stats_item_view_by_category_20170707']
> [2017-07-25 20:28:56,697] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
> [2017-07-25 20:28:56,697] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/bin/airflow", line 28, in <module>
> [2017-07-25 20:28:56,697] {base_task_runner.py:95} INFO - Subtask:     args.func(args)
> [2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 422, in run
> [2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:     pool=args.pool,
> [2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 53, in wrapper
> [2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:     result = func(*args, **kwargs)
> [2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1390, in run
> [2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:     result = task_copy.execute(context=context)
> [2017-07-25 20:28:56,698] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/operators/bigquery_operator.py", line 82, in execute
> [2017-07-25 20:28:56,699] {base_task_runner.py:95} INFO - Subtask:     self.allow_large_results, self.udf_config, self.use_legacy_sql)
> [2017-07-25 20:28:56,699] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/bigquery_hook.py", line 225, in run_query
> [2017-07-25 20:28:56,699] {base_task_runner.py:95} INFO - Subtask:     '<dataset>.<table>. Got: {}').format(destination_dataset_table)
> [2017-07-25 20:28:56,700] {base_task_runner.py:95} INFO - Subtask: AssertionError: Expected destination_dataset_table in the format of <dataset>.<table>. Got: [u'dummy-project-id:machine_learning_us.stats_item_view_by_category_20170707']
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)