airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bradford stephens (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1477) S3KeySensor throws 400 Error
Date Tue, 01 Aug 2017 01:14:00 GMT
bradford stephens created AIRFLOW-1477:
------------------------------------------

             Summary: S3KeySensor throws 400 Error
                 Key: AIRFLOW-1477
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1477
             Project: Apache Airflow
          Issue Type: Bug
          Components: core
    Affects Versions: 1.8.1
         Environment: Ubuntu Linux 16.10
            Reporter: bradford stephens


Despite configuring an S3 connection, S3KeySensor throws an error:

{code:java}
[2017-07-31 20:06:34,090] {base_task_runner.py:95} INFO - Subtask: [2017-07-31 20:06:34,088]
{models.py:1430} ERROR - S3ResponseError: 400 Bad Request
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent
call last):
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py",
line 1387, in run
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     result = task_copy.execute(context=context)
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/operators/sensors.py",
line 76, in execute
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     while not self.poke(context):
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/operators/sensors.py",
line 533, in poke
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     return hook.check_for_key(self.bucket_key,
self.bucket_name)
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/S3_hook.py",
line 262, in check_for_key
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     bucket = self.get_bucket(bucket_name)
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/S3_hook.py",
line 222, in get_bucket
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:     return self.connection.get_bucket(bucket_name)
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask:   File "/home/bradford/.local/lib/python2.7/site-packages/boto/s3/connection.py",
line 506, in get_bucket
[2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:     return self.head_bucket(bucket_name,
headers=headers)
[2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:   File "/home/bradford/.local/lib/python2.7/site-packages/boto/s3/connection.py",
line 553, in head_bucket
[2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:     response.status, response.reason,
body)
[2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask: S3ResponseError: S3ResponseError:
400 Bad Request
{code}

I have tried both with and without the host parameter in the connection.

The DAG code includes:

{code:python}
inputsensor = S3KeySensor(
   task_id='check_s3_for_file_in_s3',
   bucket_key='test-input-temp.json',
   wildcard_match=False,
   bucket_name='ventera-staging-data',
   s3_conn_id='S3Connection',
   timeout=18*60*60,
   poke_interval=120,
   dag=dag)
{code}

The "extra" field in S3Connection has been set with the aws keys.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message