airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Joyce <anthony.jo...@omicronmedia.com>
Subject Re: Airflow-Init stuck at initialization
Date Tue, 17 Aug 2021 12:58:13 GMT
Thanks Jarek.

The commands below can grab the IP from the hostname, but nc -zvvn definitely hangs when trying
to connect to my host IP on the Postgres port. This helps me narrow it down a bit. Thank you
for that suggestion!

Anthony

On Aug 16, 2021, at 1:56 PM, Jarek Potiuk <jarek@potiuk.com<mailto:jarek@potiuk.com>>
wrote:

Another thing to try - just run airflow image with `bash` parameter (`docker run -it <image>
bash`) and see if you can connect to your host/port (there is `nc` tool installed in the production
image). I believe when you try it you will hang on trying to connect to your database.

I think there is one reason why it could fail - in case your host name is wrongly parsed from
the URL (because it contains restricted characters) - this could happen.

This is what the connectivity check does (this is where it seems to hang) - copied it with
explanation on what is happening (gethosbyname will also work for IP addresses BTW). You might
try to reproduce this.

    # Checks if it is possible to connect to the host using netcat.
    #
    # We want to avoid misleading messages and perform only forward lookup of the service
IP address.
    # Netcat when run without -n performs both forward and reverse lookup and fails if the
reverse
    # lookup name does not match the original name even if the host is reachable via IP. This
happens
    # randomly with docker-compose in GitHub Actions.
    # Since we are not using reverse lookup elsewhere, we can perform forward lookup in python
    # And use the IP in NC and add '-n' switch to disable any DNS use.
    # Even if this message might be harmless, it might hide the real reason for the problem
    # Which is the long time needed to start some services, seeing this message might be totally
misleading
    # when you try to analyse the problem, that's why it's best to avoid it,
ip=$(python -c "import socket; print(socket.gethostbyname('${host}'))")
nc -zvvn "${ip}" "${port}"


J.


On Mon, Aug 16, 2021 at 7:31 PM Anthony Joyce <anthony.joyce@omicronmedia.com<mailto:anthony.joyce@omicronmedia.com>>
wrote:
Thanks for the suggestions. Will do. And yes, I am using the official apache-airflow image.

Anthony

On Aug 16, 2021, at 1:29 PM, Daniel Standish <dpstandish@gmail.com<mailto:dpstandish@gmail.com>>
wrote:

Let me offer a suggestion.

I think it's helpful to simplify things and isolate components.  To that end what you could
do is scrap docker for a moment.  Create a python virtualenv in your host.  Install airflow
into this virtualenv. Set AIRFLOW__CORE__SQL_ALCHEMY_CONN as appropriate and run airflow in
the virtualenv.  Then you'll be able to confirm the issue is docker and not some other connectivity
issue.

Next move on to docker, perhaps without compose.  One thing you could do is get a terminal
in a running container and try to connect to this server e.g. using telnet or psql (you might
have to install these).  Then you'll figure out what connection string will actually work
and it's just a matter of setting up the env vars correctly for airflow to connect in the
right way.

Also, are you using the official apache-airflow image?  If not, I would recommend that.




On Mon, Aug 16, 2021 at 10:22 AM Anthony Joyce <anthony.joyce@omicronmedia.com<mailto:anthony.joyce@omicronmedia.com>>
wrote:
Hi Daniel,

Thanks for the suggestion! I’m new at Docker so apologies if some of the configs are incorrect.
Still learning.

I removed service Postgres since I am using my local db. Still stuck on airflow-init. There
are no log entries that explain what it is doing or what it is waiting for, but obviously
airflow-init is waiting for something. I’ve since restarted docker service, flushed iptables
chains relating to Docker, restarted again. Still stuck.

Anthony

On Aug 16, 2021, at 1:11 PM, Daniel Standish <dpstandish@gmail.com<mailto:dpstandish@gmail.com>>
wrote:

If you are using an already existing postgres, then why are you running postgres service in
compose?

If that's not the database you are connecting to, then you should remove it from your compose
file. (conversely if you are trying to use compose postgres, then we might have some configuration
suggestions to offer)
---
Separately, on mac compose, for the container to access host i think you might need to add
svc.localhost or something like that... there are SO posts on this topic eg https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mac




--
+48 660 796 129

Mime
View raw message