airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Hodgkinson <nik.hodgkin...@collectivehealth.com>
Subject Re: Celery with SQS?
Date Wed, 16 Aug 2017 01:58:39 GMT
Here's the full log if it's helpful for anyone in helping:

 -------------- celery@e6f3c1e29507 v4.0.2 (latentcall)
---- **** -----
--- * ***  * -- Linux-3.13.0-125-generic-x86_64-with-Ubuntu-14.04-trusty
2017-08-16 01:53:32
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:
airflow.executors.celery_executor:0x7fad429d8978
- ** ---------- .> transport:   sqs://localhost//
- ** ---------- .> results:
postgresql://airflow:**@pg-database:5432/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this
worker)
--- ***** -----
 -------------- [queues]
                .> queue-1 exchange=queue-1(direct) key=queue-1

[2017-08-16 01:54:36,481] {__init__.py:57} INFO - Using executor
CeleryExecutor
[2017-08-16 01:54:36,571] {driver.py:120} INFO - Generating grammar tables
from /usr/lib/python3.4/lib2to3/Grammar.txt
[2017-08-16 01:54:36,593] {driver.py:120} INFO - Generating grammar tables
from /usr/lib/python3.4/lib2to3/PatternGrammar.txt
[2017-08-16 01:54:37,393] {__init__.py:57} INFO - Using executor
CeleryExecutor
[2017-08-16 01:54:37,480] {driver.py:120} INFO - Generating grammar tables
from /usr/lib/python3.4/lib2to3/Grammar.txt
[2017-08-16 01:54:37,502] {driver.py:120} INFO - Generating grammar tables
from /usr/lib/python3.4/lib2to3/PatternGrammar.txt
Starting flask
[2017-08-16 01:54:37,744] {_internal.py:87} INFO -  * Running on
http://0.0.0.0:8793/ (Press CTRL+C to quit)
[2017-08-16 01:54:38,582: ERROR/MainProcess] Caught exception reading
instance data
Traceback (most recent call last):
  File
"/usr/local/lib/python3.4/dist-packages/kombu/transport/virtual/base.py",
line 921, in create_channel
    return self._avail_channels.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/urllib/request.py", line 1182, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.4/http/client.py", line 1125, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1163, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.4/http/client.py", line 1121, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.4/http/client.py", line 951, in _send_output
    self.send(msg)
  File "/usr/lib/python3.4/http/client.py", line 886, in send
    self.connect()
  File "/usr/lib/python3.4/http/client.py", line 863, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python3.4/socket.py", line 512, in create_connection
    raise err
  File "/usr/lib/python3.4/socket.py", line 503, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/boto/utils.py", line 210, in
retry_url
    r = opener.open(req, timeout=timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 463, in open
    response = self._open(req, data)
  File "/usr/lib/python3.4/urllib/request.py", line 481, in _open
    '_open', req)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 1210, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.4/urllib/request.py", line 1184, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error timed out>
[2017-08-16 01:54:38,585: ERROR/MainProcess] Unable to read instance data,
giving up
[2017-08-16 01:54:38,585: CRITICAL/MainProcess] Unrecoverable error:
NoAuthHandlerFound("No handler was ready to authenticate. 1 handlers were
checked. ['HmacAuthV4Handler'] Check your credentials",)
Traceback (most recent call last):
  File
"/usr/local/lib/python3.4/dist-packages/kombu/transport/virtual/base.py",
line 921, in create_channel
    return self._avail_channels.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/celery/worker/worker.py",
line 203, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.4/dist-packages/celery/bootsteps.py", line
119, in start
    step.start(parent)
  File "/usr/local/lib/python3.4/dist-packages/celery/bootsteps.py", line
370, in start
    return self.obj.start()
  File
"/usr/local/lib/python3.4/dist-packages/celery/worker/consumer/consumer.py",
line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.4/dist-packages/celery/bootsteps.py", line
119, in start
    step.start(parent)
  File
"/usr/local/lib/python3.4/dist-packages/celery/worker/consumer/connection.py",
line 21, in start
    c.connection = c.connect()
  File
"/usr/local/lib/python3.4/dist-packages/celery/worker/consumer/consumer.py",
line 405, in connect
    conn = self.connection_for_read(heartbeat=self.amqheartbeat)
  File
"/usr/local/lib/python3.4/dist-packages/celery/worker/consumer/consumer.py",
line 412, in connection_for_read
    self.app.connection_for_read(heartbeat=heartbeat))
  File
"/usr/local/lib/python3.4/dist-packages/celery/worker/consumer/consumer.py",
line 436, in ensure_connected
    callback=maybe_shutdown,
  File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line
405, in ensure_connection
    callback)
  File "/usr/local/lib/python3.4/dist-packages/kombu/utils/functional.py",
line 333, in retry_over_time
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line
261, in connect
    return self.connection
  File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line
802, in connection
    self._connection = self._establish_connection()
  File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line
757, in _establish_connection
    conn = self.transport.establish_connection()
  File
"/usr/local/lib/python3.4/dist-packages/kombu/transport/virtual/base.py",
line 941, in establish_connection
    self._avail_channels.append(self.create_channel(self))
  File
"/usr/local/lib/python3.4/dist-packages/kombu/transport/virtual/base.py",
line 923, in create_channel
    channel = self.Channel(connection)
  File "/usr/local/lib/python3.4/dist-packages/kombu/transport/SQS.py",
line 102, in __init__
    self._update_queue_cache(self.queue_name_prefix)
  File "/usr/local/lib/python3.4/dist-packages/kombu/transport/SQS.py",
line 108, in _update_queue_cache
    queues = self.sqs.get_all_queues(prefix=queue_name_prefix)
  File "/usr/local/lib/python3.4/dist-packages/kombu/transport/SQS.py",
line 405, in sqs
    self._sqs = self._aws_connect_to(SQSConnection, regions())
  File "/usr/local/lib/python3.4/dist-packages/kombu/transport/SQS.py",
line 400, in _aws_connect_to
    port=port)
  File "/usr/local/lib/python3.4/dist-packages/boto/sqs/connection.py",
line 62, in __init__
    profile_name=profile_name)
  File "/usr/local/lib/python3.4/dist-packages/boto/connection.py", line
1100, in __init__
    provider=provider)
  File "/usr/local/lib/python3.4/dist-packages/boto/connection.py", line
569, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "/usr/local/lib/python3.4/dist-packages/boto/auth.py", line 1021, in
get_auth_handler
    'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1
handlers were checked. ['HmacAuthV4Handler'] Check your credentials


-N
nik.hodgkinson@collectivehealth.com

On Tue, Aug 15, 2017 at 2:58 PM, Nicholas Hodgkinson <
nik.hodgkinson@collectivehealth.com> wrote:

> I'm wondering if anyone has had any experience setting up Airflow on AWS
> using SQS as the Celery backend. I've got an instance credential set but
> I'm getting this error on the worker:
>
> boto.exception.NoAuthHandlerFound: No handler was ready to authenticate.
> 1 handlers were checked. ['HmacAuthV4Handler'] Check your credentials
>
> Wondering if anyone has any insight.
>
> Thanks,
> -Nik
> nik.hodgkinson@collectivehealth.com
>
>

-- 

         See why we are one of WSJ's Top Tech Companies to Watch 
<https://www.wsj.com/articles/collective-healths-goal-medical-benefits-that-work-1497492300?mg=prod/accounts-wsj>

*This message may contain confidential, proprietary, or protected 
information.  If you are not the intended recipient, you may not review, 
copy, or distribute this message. If you received this message in error, 
please notify the sender by reply email and delete this message.*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message