airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1696) GCP Dataproc Operator fails due to '+' in Airflow version
Date Wed, 11 Oct 2017 20:29:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200908#comment-16200908
] 

ASF subversion and git services commented on AIRFLOW-1696:
----------------------------------------------------------

Commit 98b4df945dec5dfe766c8a9a15cf4b3932bf1a5c in incubator-airflow's branch refs/heads/master
from [~TrevorEdwards]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=98b4df9 ]

[AIRFLOW-1696] Fix dataproc version label error

Closes #2676 from TrevorEdwards/airflow-1696


> GCP Dataproc Operator fails due to '+' in Airflow version
> ---------------------------------------------------------
>
>                 Key: AIRFLOW-1696
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1696
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Trevor Edwards
>            Assignee: Trevor Edwards
>             Fix For: 1.10.0
>
>
> Since Dataproc operator attaches the Airflow version as one of its labels, and dataproc
labels cannot include the character '+', dataproc operator currently fails with the following
error:
> {code:none}
> [2017-10-09 19:28:48,035] {base_task_runner.py:115} INFO - Running: ['bash', '-c', u'airflow
run smokey-dataproc start_cluster 2017-10-08T00:00:00 --job_id 6 --raw -sd DAGS_FOLDER/dataa.py']
> [2017-10-09 19:28:49,041] {base_task_runner.py:98} INFO - Subtask: [2017-10-09 19:28:49,041]
{__init__.py:45} INFO - Using executor LocalExecutor
> [2017-10-09 19:28:49,139] {base_task_runner.py:98} INFO - Subtask: [2017-10-09 19:28:49,139]
{models.py:187} INFO - Filling up the DagBag from /home/airflow/dags/dataa.py
> [2017-10-09 19:28:49,258] {base_task_runner.py:98} INFO - Subtask: Cluster name: smoke-cluster-aa05845a-1b60-4543-94d0-f7dfddb90ee0
> [2017-10-09 19:28:49,258] {base_task_runner.py:98} INFO - Subtask: [2017-10-09 19:28:49,257]
{dataproc_operator.py:267} INFO - Creating cluster: smoke-cluster-aa05845a-1b60-4543-94d0-f7dfddb90ee0
> [2017-10-09 19:28:49,265] {base_task_runner.py:98} INFO - Subtask: [2017-10-09 19:28:49,265]
{gcp_api_base_hook.py:82} INFO - Getting connection using a JSON key file.
> [2017-10-09 19:28:59,909] {base_task_runner.py:98} INFO - Subtask: [2017-10-09 19:28:59,906]
{models.py:1564} ERROR - <HttpError 400 when requesting https://dataproc.googleapis.com/v1/projects/cloud-airflow-test/regions/global/clusters?alt=json
returned "Multiple validation errors:
> [2017-10-09 19:28:59,910] {base_task_runner.py:98} INFO - Subtask:  - Not a valid value:
"v1-10-0dev0+incubating". Only lowercase letters, numbers, and dashes are allowed. The value
must start with lowercase letter or number and end with a lowercase letter or number.
> [2017-10-09 19:28:59,910] {base_task_runner.py:98} INFO - Subtask:  - User label value
must conform to '[\p{Ll}\p{Lo}\p{N}_-]{0,63}' pattern">
> [2017-10-09 19:28:59,911] {base_task_runner.py:98} INFO - Subtask: Traceback (most recent
call last):
> [2017-10-09 19:28:59,911] {base_task_runner.py:98} INFO - Subtask:   File "/home/airflow/incubator-airflow/airflow/models.py",
line 1462, in _run_raw_task
> [2017-10-09 19:28:59,911] {base_task_runner.py:98} INFO - Subtask:     result = task_copy.execute(context=context)
> [2017-10-09 19:28:59,912] {base_task_runner.py:98} INFO - Subtask:   File "/home/airflow/incubator-airflow/airflow/contrib/operators/dataproc_operator.py",
line 300, in execute
> [2017-10-09 19:28:59,912] {base_task_runner.py:98} INFO - Subtask:     raise e
> [2017-10-09 19:28:59,913] {base_task_runner.py:98} INFO - Subtask: HttpError: <HttpError
400 when requesting https://dataproc.googleapis.com/v1/projects/cloud-airflow-test/regions/global/clusters?alt=json
returned "Multiple validation errors:
> [2017-10-09 19:28:59,914] {base_task_runner.py:98} INFO - Subtask:  - Not a valid value:
"v1-10-0dev0+incubating". Only lowercase letters, numbers, and dashes are allowed. The value
must start with lowercase letter or number and end with a lowercase letter or number.
> [2017-10-09 19:28:59,914] {base_task_runner.py:98} INFO - Subtask:  - User label value
must conform to '[\p{Ll}\p{Lo}\p{N}_-]{0,63}' pattern">
> [2017-10-09 19:28:59,915] {base_task_runner.py:98} INFO - Subtask: [2017-10-09 19:28:59,909]
{models.py:1585} INFO - Marking task as UP_FOR_RETRY
> [2017-10-09 19:28:59,978] {base_task_runner.py:98} INFO - Subtask: [2017-10-09 19:28:59,977]
{models.py:1613} ERROR - <HttpError 400 when requesting https://dataproc.googleapis.com/v1/projects/cloud-airflow-test/regions/global/clusters?alt=json
returned "Multiple validation errors:
> [2017-10-09 19:28:59,978] {base_task_runner.py:98} INFO - Subtask:  - Not a valid value:
"v1-10-0dev0+incubating". Only lowercase letters, numbers, and dashes are allowed. The value
must start with lowercase letter or number and end with a lowercase letter or number.
> [2017-10-09 19:28:59,979] {base_task_runner.py:98} INFO - Subtask:  - User label value
must conform to '[\p{Ll}\p{Lo}\p{N}_-]{0,63}' pattern">
> [2017-10-09 19:28:59,979] {base_task_runner.py:98} INFO - Subtask: Traceback (most recent
call last):
> [2017-10-09 19:28:59,979] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/bin/airflow",
line 6, in <module>
> [2017-10-09 19:28:59,980] {base_task_runner.py:98} INFO - Subtask:     exec(compile(open(__file__).read(),
__file__, 'exec'))
> [2017-10-09 19:28:59,980] {base_task_runner.py:98} INFO - Subtask:   File "/home/airflow/incubator-airflow/airflow/bin/airflow",
line 27, in <module>
> [2017-10-09 19:28:59,981] {base_task_runner.py:98} INFO - Subtask:     args.func(args)
> [2017-10-09 19:28:59,981] {base_task_runner.py:98} INFO - Subtask:   File "/home/airflow/incubator-airflow/airflow/bin/cli.py",
line 394, in run
> [2017-10-09 19:28:59,982] {base_task_runner.py:98} INFO - Subtask:     pool=args.pool,
> [2017-10-09 19:28:59,982] {base_task_runner.py:98} INFO - Subtask:   File "/home/airflow/incubator-airflow/airflow/utils/db.py",
line 50, in wrapper
> [2017-10-09 19:28:59,982] {base_task_runner.py:98} INFO - Subtask:     result = func(*args,
**kwargs)
> [2017-10-09 19:28:59,983] {base_task_runner.py:98} INFO - Subtask:   File "/home/airflow/incubator-airflow/airflow/models.py",
line 1462, in _run_raw_task
> [2017-10-09 19:28:59,983] {base_task_runner.py:98} INFO - Subtask:     result = task_copy.execute(context=context)
> [2017-10-09 19:28:59,984] {base_task_runner.py:98} INFO - Subtask:   File "/home/airflow/incubator-airflow/airflow/contrib/operators/dataproc_operator.py",
line 300, in execute
> [2017-10-09 19:28:59,984] {base_task_runner.py:98} INFO - Subtask:     raise e
> [2017-10-09 19:28:59,984] {base_task_runner.py:98} INFO - Subtask: googleapiclient.errors.HttpError:
<HttpError 400 when requesting https://dataproc.googleapis.com/v1/projects/cloud-airflow-test/regions/global/clusters?alt=json
returned "Multiple validation errors:
> [2017-10-09 19:28:59,985] {base_task_runner.py:98} INFO - Subtask:  - Not a valid value:
"v1-10-0dev0+incubating". Only lowercase letters, numbers, and dashes are allowed. The value
must start with lowercase letter or number and end with a lowercase letter or number.
> [2017-10-09 19:28:59,985] {base_task_runner.py:98} INFO - Subtask:  - User label value
must conform to '[\p{Ll}\p{Lo}\p{N}_-]{0,63}' pattern">
> {code}
> A fix similar to https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L243
may be applied.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message