airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Crawford (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1477) S3KeySensor throws 400 Error
Date Wed, 27 Sep 2017 15:04:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16182700#comment-16182700
] 

Michael Crawford commented on AIRFLOW-1477:
-------------------------------------------

I have been using the s3keysensor in 1.8.2 with the credentials in the extra field without
issue.

> S3KeySensor throws 400 Error
> ----------------------------
>
>                 Key: AIRFLOW-1477
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1477
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.8.1
>         Environment: Ubuntu Linux 16.10
>            Reporter: bradford stephens
>
> Despite configuring an S3 connection, S3KeySensor throws an error:
> {code:java}
> [2017-07-31 20:06:34,090] {base_task_runner.py:95} INFO - Subtask: [2017-07-31 20:06:34,088]
{models.py:1430} ERROR - S3ResponseError: 400 Bad Request
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent
call last):
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py",
line 1387, in run
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     result = task_copy.execute(context=context)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/operators/sensors.py",
line 76, in execute
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     while not self.poke(context):
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/operators/sensors.py",
line 533, in poke
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     return hook.check_for_key(self.bucket_key,
self.bucket_name)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/S3_hook.py",
line 262, in check_for_key
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     bucket = self.get_bucket(bucket_name)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/S3_hook.py",
line 222, in get_bucket
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     return self.connection.get_bucket(bucket_name)
> [2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/home/bradford/.local/lib/python2.7/site-packages/boto/s3/connection.py",
line 506, in get_bucket
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:     return self.head_bucket(bucket_name,
headers=headers)
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:   File "/home/bradford/.local/lib/python2.7/site-packages/boto/s3/connection.py",
line 553, in head_bucket
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:     response.status,
response.reason, body)
> [2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask: S3ResponseError: S3ResponseError:
400 Bad Request
> {code}
> I have tried both with and without the host parameter in the connection.
> The DAG code includes:
> {code:python}
> inputsensor = S3KeySensor(
>    task_id='check_s3_for_file_in_s3',
>    bucket_key='test-input-temp.json',
>    wildcard_match=False,
>    bucket_name='ventera-staging-data',
>    s3_conn_id='S3Connection',
>    timeout=18*60*60,
>    poke_interval=120,
>    dag=dag)
> {code}
> The "extra" field in S3Connection has been set with the aws keys.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message