airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-667) Handle 503 errors on google bigquery_hook
Date Wed, 14 Dec 2016 21:03:59 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15749477#comment-15749477
] 

ASF subversion and git services commented on AIRFLOW-667:
---------------------------------------------------------

Commit cac133001b517dd7d66a31288dd375eb63d8cd26 in incubator-airflow's branch refs/heads/master
from [~kmettu]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=cac1330 ]

[AIRFLOW-667] Handle BigQuery 503 error

Closes #1938 from krmettu/master


> Handle 503 errors on google bigquery_hook 
> ------------------------------------------
>
>                 Key: AIRFLOW-667
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-667
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: gcp
>    Affects Versions: Airflow 2.0, Airflow 1.7.1, Airflow 1.7.0
>            Reporter: Krishnaveni Mettu
>            Assignee: Krishnaveni Mettu
>            Priority: Minor
>
> Getting 503 errors occasionally while querying google bigquery job status
> Here is the error message from logs - 
> [2016-11-30 11:11:46,125] {bigquery_hook.py:454} INFO - Waiting for job to complete:
xxxxxx, job_xxx_xxxx
> [2016-11-30 11:11:51,131] {discovery.py:838} INFO - URL being requested: GET https://www.googleapis.com/bigquery/v2/projects/xxxx/jobs/job_xxxxxx?alt=json
> [2016-11-30 11:12:55,170] {models.py:1321} ERROR - <HttpError 503 when requesting
https://www.googleapis.com/bigquery/v2/projects/xxxx/jobs/job_xxxxxxxx?alt=json returned "Error
encountered during execution. Retrying may solve the problem.">
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1280, in run
>     result = task_copy.execute(context=context)
>   File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/gcs_to_bq.py", line
129, in execute
>     num_retries=self.num_retries)
>   File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/bigquery_hook.py", line
403, in run_load
>     return self.run_with_configuration(configuration)
>   File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/bigquery_hook.py", line
456, in run_with_configuration
>     job = jobs.get(projectId=self.project_id, jobId=job_id).execute()
>   File "/usr/lib/python2.7/site-packages/oauth2client/util.py", line 135, in positional_wrapper
>     return wrapped(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/googleapiclient/http.py", line 838, in execute
>     raise HttpError(resp, content, uri=self.uri)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message