airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jorge Alpedrinha Ramos <jalpedrinhara...@gmail.com>
Subject Re: Logs And Docker
Date Fri, 24 Mar 2017 10:36:54 GMT
I just read the stacktrace with more attention and celery is not the
culprit, I thought that logs were served by celery but it's the serve_logs
app from airflow that is responsible for this. This sounds like some
configuration on the max buffer size for responses, but no being an expert
in flask this is just a wild guess.

Nicholas is this the behavior you're seing? If this isn't the case you may
have a situation where your webserver container is not able to comunicate
with the worker.

As promised here is the stacktrace:

worker_1         | [2017-03-24 10:29:39,574] {_internal.py:87} INFO -
172.18.0.8 - - [24/Mar/2017 10:29:39] "GET
/log/rates.ticker-to-analytics/parse-syslog/2017-01-12T07:45:00 HTTP/1.1"
200 -
worker_1         | Traceback (most recent call last):
worker_1         |   File "/usr/local/bin/airflow", line 28, in <module>
worker_1         |     args.func(args)
worker_1         |   File
"/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 858, in
serve_logs
worker_1         |     host='0.0.0.0', port=WORKER_LOG_SERVER_PORT)
worker_1         |   File
"/usr/local/lib/python2.7/site-packages/flask/app.py", line 843, in run
worker_1         |     run_simple(host, port, self, **options)
worker_1         |   File
"/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 736, in
run_simple
worker_1         |     inner()
worker_1         |   File
"/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 699, in
inner
worker_1         |     srv.serve_forever()
worker_1         |   File
"/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 536, in
serve_forever
worker_1         |     HTTPServer.serve_forever(self)
worker_1         |   File "/usr/local/lib/python2.7/SocketServer.py", line
233, in serve_forever
worker_1         |     self._handle_request_noblock()
worker_1         |   File "/usr/local/lib/python2.7/SocketServer.py", line
292, in _handle_request_noblock
worker_1         |     self.handle_error(request, client_address)
worker_1         |   File "/usr/local/lib/python2.7/SocketServer.py", line
290, in _handle_request_noblock
worker_1         |     self.process_request(request, client_address)
worker_1         |   File "/usr/local/lib/python2.7/SocketServer.py", line
318, in process_request
worker_1         |     self.finish_request(request, client_address)
worker_1         |   File "/usr/local/lib/python2.7/SocketServer.py", line
331, in finish_request
worker_1         |     self.RequestHandlerClass(request, client_address,
self)
worker_1         |   File "/usr/local/lib/python2.7/SocketServer.py", line
654, in __init__
worker_1         |     self.finish()
worker_1         |   File "/usr/local/lib/python2.7/SocketServer.py", line
713, in finish
worker_1         |     self.wfile.close()
worker_1         |   File "/usr/local/lib/python2.7/socket.py", line 283,
in close
worker_1         |     self.flush()
worker_1         |   File "/usr/local/lib/python2.7/socket.py", line 307,
in flush
worker_1         |
self._sock.sendall(view[write_offset:write_offset+buffer_size])
worker_1         | socket.error: [Errno 32] Broken pipe

On Fri, Mar 24, 2017 at 10:27 AM Jorge Alpedrinha Ramos <
jalpedrinharamos@gmail.com> wrote:

> Hi,
>
> I have run into this on a specific scenario where the log file is huge and
> the worker throws some error when responding to the webserver. You'll
> probably see some issue with writing to buffer if you check the worker
> logs. I'll send a second email with a stacktrace of this, I believe this is
> somehow related to celery, but I'll need to do some more digging.
>
> Another thing that could be a solution would be to make airflow webserver
> to use the remote logs location for fetching the log instead of always
> relying on the worker (which may be decommissioned, after a scaling
> operation for handling a spike, and no longer available).
>
> On Fri, Mar 24, 2017 at 9:37 AM Gael Magnan <gaelmagnan@gmail.com> wrote:
>
> Hi,
>
> we have encountered the same problem on some machines but not all of them.
>
> One of our developer can't access log on his mac since the move to airflow
> 1.8.0, but our production machine on Ubuntu doesn't have the problem.
> For him it seems like the log file name the UI try to access is not the one
> the worker created (IE different date format).
>
> We haven't found out the cause of the problem so if you find something I'm
> interested.
>
> Regards
> Gael
>
> Le ven. 24 mars 2017 à 01:26, Nicholas Hodgkinson <
> nik.hodgkinson@collectivehealth.com> a écrit :
>
> > So I'm running my scheduler and webserver in different Docker container
> on
> > the same host, everything seems to be working fine with the exception of
> > access to logs from the UI. When doing so I get this:
> >
> > *** Log file isn't local.
> > *** Fetching here:
> > http://30751ef88b7a:8793/log/dag-id/task-id/2017-03-23T23:00:00
> > *** Failed to fetch log file from worker.
> >
> > *** Reading remote logs...
> > *** Unsupported remote log location.
> >
> > However both containers have the same log directory mounted as a volume
> > inside the container, which is specified correctly as a environment
> > variable. Resources on this problem are scarce and I'm not sure how to
> > solve it. Thoughts?
> >
> > -Nik
> > nik.hodgkinson@collectivehealth.com
> >
> > --
> >
> >
> > Read our founder's story.
> > <https://collectivehealth.com/blog/started-collective-health/>
> >
> > *This message may contain confidential, proprietary, or protected
> > information.  If you are not the intended recipient, you may not review,
> > copy, or distribute this message. If you received this message in error,
> > please notify the sender by reply email and delete this message.*
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message