airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jorge Alpedrinha Ramos <jalpedrinhara...@gmail.com>
Subject Re: Logs And Docker
Date Fri, 24 Mar 2017 10:27:56 GMT
Hi,

I have run into this on a specific scenario where the log file is huge and
the worker throws some error when responding to the webserver. You'll
probably see some issue with writing to buffer if you check the worker
logs. I'll send a second email with a stacktrace of this, I believe this is
somehow related to celery, but I'll need to do some more digging.

Another thing that could be a solution would be to make airflow webserver
to use the remote logs location for fetching the log instead of always
relying on the worker (which may be decommissioned, after a scaling
operation for handling a spike, and no longer available).

On Fri, Mar 24, 2017 at 9:37 AM Gael Magnan <gaelmagnan@gmail.com> wrote:

> Hi,
>
> we have encountered the same problem on some machines but not all of them.
>
> One of our developer can't access log on his mac since the move to airflow
> 1.8.0, but our production machine on Ubuntu doesn't have the problem.
> For him it seems like the log file name the UI try to access is not the one
> the worker created (IE different date format).
>
> We haven't found out the cause of the problem so if you find something I'm
> interested.
>
> Regards
> Gael
>
> Le ven. 24 mars 2017 à 01:26, Nicholas Hodgkinson <
> nik.hodgkinson@collectivehealth.com> a écrit :
>
> > So I'm running my scheduler and webserver in different Docker container
> on
> > the same host, everything seems to be working fine with the exception of
> > access to logs from the UI. When doing so I get this:
> >
> > *** Log file isn't local.
> > *** Fetching here:
> > http://30751ef88b7a:8793/log/dag-id/task-id/2017-03-23T23:00:00
> > *** Failed to fetch log file from worker.
> >
> > *** Reading remote logs...
> > *** Unsupported remote log location.
> >
> > However both containers have the same log directory mounted as a volume
> > inside the container, which is specified correctly as a environment
> > variable. Resources on this problem are scarce and I'm not sure how to
> > solve it. Thoughts?
> >
> > -Nik
> > nik.hodgkinson@collectivehealth.com
> >
> > --
> >
> >
> > Read our founder's story.
> > <https://collectivehealth.com/blog/started-collective-health/>
> >
> > *This message may contain confidential, proprietary, or protected
> > information.  If you are not the intended recipient, you may not review,
> > copy, or distribute this message. If you received this message in error,
> > please notify the sender by reply email and delete this message.*
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message