airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Joyce <anthony.jo...@omicronmedia.com>
Subject Re: Accessing external DAGs from Docker Container
Date Mon, 09 Aug 2021 13:47:18 GMT
Hi Franco,

Adding the Airflow home environmental variable didn’t seem to help, although your suggestion
makes sense. I am not completely sure why it is not being picked up. See below. (I put in
the absolute path with and without single quotes to test it out.)

version: '3'
x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.2}
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgres<postgresql+psycopg2://airflow@>ql+psycopg2://airflow@<postgresql+psycopg2://airflow@>{host}/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+<db+postgresql://airflow@>postgres<db+postgresql://airflow@>ql://airflow@<db+postgresql://airflow@>{host}/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: {key}
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__CORE__DAGS_FOLDER: ‘{HOME}/airflow/dags'
    AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs

Perhaps it has something to do with these WARNINGs I have been receiving from flower?

flower_1             | [2021-08-09 13:28:37,441] {mixins.py:229} INFO - Connected to redis://redis:6379/0
flower_1             | [2021-08-09 13:28:38,654] {inspector.py:42} WARNING - Inspect method
stats failed
flower_1             | [2021-08-09 13:28:38,655] {inspector.py:42} WARNING - Inspect method
conf failed
flower_1             | [2021-08-09 13:28:38,663] {inspector.py:42} WARNING - Inspect method
active_queues failed
flower_1             | [2021-08-09 13:28:38,668] {inspector.py:42} WARNING - Inspect method
scheduled failed
flower_1             | [2021-08-09 13:28:38,669] {inspector.py:42} WARNING - Inspect method
revoked failed
flower_1             | [2021-08-09 13:28:38,670] {inspector.py:42} WARNING - Inspect method
active failed
flower_1             | [2021-08-09 13:28:38,671] {inspector.py:42} WARNING - Inspect method
reserved failed
flower_1             | [2021-08-09 13:28:38,672] {inspector.py:42} WARNING - Inspect method
registered failed


Any other suggestions welcome.

Thanks,

Anthony


On Aug 9, 2021, at 8:47 AM, Franco Peschiera <franco.peschiera@gmail.com<mailto:franco.peschiera@gmail.com>>
wrote:

You don't often get email from franco.peschiera@gmail.com<mailto:franco.peschiera@gmail.com>.
Learn why this is important<http://aka.ms/LearnAboutSenderIdentification>
I don't see you have configured the AIRFLOW_HOME environment variable. Try setting it to the
absolute path so that it can find your dags. Also, there is an environment variable where
you can manually pass the path to the DAGs folder. AIRFLOW__CORE__DAGS_FOLDER

On Mon, Aug 9, 2021, 14:39 Anthony Joyce <anthony.joyce@omicronmedia.com<mailto:anthony.joyce@omicronmedia.com>>
wrote:
Hi all,

This is my first time the email list so I thank you in advance for the help.

Here is my situation:

We are running Airflow 1.10.10..a pretty old version at this point without a container and
built from pip on CentOS. Instead of updating anaconda and dealing with dependency hell, I
decided to download the official apache/airflow docker container and try to configure it to
my already existing meta database and DAGs. It seems the container initialized successfully
picking up our Variables and Connections via our existing Postgres-13 meta database with all
containers healthy at this point. However, I am having a problem connecting our external Airflow
DAGs folder (~/airflow/dags). I have copied the information in our ~/airflow/dags folder in
the ./dags folder but that doesn’t seem to help.

Do you all have any advice/suggestions regarding this issue?

Here is the redacted docker-compose.yaml file below.

Thank you for the help!

Best,

Anthony

version: '3'
x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.2}
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow@{host}/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow@{host}/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: {key}
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    #- /home/etl/airflow/plugins/:/opt/airflow/plugins
  user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-0}"
  depends_on:
    redis:
      condition: service_healthy
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/localdata/pgdata
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5
    restart: always

  redis:
    image: redis:latest
    ports:
      - 6379:6379
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 30s
      retries: 50
    restart: always

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 8080:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    healthcheck:
      test:
        - "CMD-SHELL"
        - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-init:
    <<: *airflow-common
    command: version
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}

  flower:
    <<: *airflow-common
    command: celery flower
    ports:
      - 5555:5555
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

volumes:
  postgres-db-volume:




Mime
View raw message